fix(metrics): stop publishing precision under upstream's mean_precisions key

AGENTS.md

@@ -130,7 +130,22 @@ When migrating or modifying search logic, **always compare precision output** be
 }
 ```
 
-**Rust search result JSON** uses the same field names as Python v0 (`mean_precisions`, `rps`, `p50_time`, etc.). Both versions write to `results/` with filename format: `{engine}-{dataset}-search-{id}-{pid}-{timestamp}.json`.
+**Rust search result JSON** uses the same field names as Python v0 for timing/throughput (`rps`, `p50_time`, …). Both versions write to `results/` with filename format: `{engine}-{dataset}-search-{id}-{pid}-{timestamp}.json`.
+
+#### Quality metric keys: `mean_precisions` is NOT ours (#217)
+
+Python v0 — and upstream `qdrant/vector-db-benchmark`, `engine/base_client/search.py` — computes `len(ids & expected[:top]) / top`, i.e. **recall@top**, and publishes it under the key `mean_precisions`. Our Rust build emits, since **schema version 2**:
+
+| key | formula | notes |
+|---|---|---|
+| `mean_precision_at_returned` | `hits / |results returned|` | was `mean_precisions` before schema v2 — the rename is what closed #217 |
+| `mean_recall` | `hits / |valid ground-truth ids in expected[:top]|` | equals Python/upstream `mean_precisions` **only** when every ground-truth row has >= `top` valid ids |
+| `precisions_at_returned` | per-query array of the above precision | only under `--dump-raw-latencies`; was `precisions` |
+| `precision_at_returned_dist` | digest of that array | was `precision_dist` |
+
+Every result file carries a top-level `metrics_schema` block with these formulas plus a `ground_truth` width profile and a `comparable_to_upstream_mean_precisions` field naming the key (if any) that can be overlaid on upstream numbers. **We never emit a key named `mean_precisions`** — the same name for two formulas is the state that must not ship.
+
+Calibration (`calibration_precision`) targets `mean_precision_at_returned`; when the dataset's ground truth is narrower than `top` the target can be unreachable by construction, which the run now warns about and records in `params.calibration.reached_target`.
 
 ### How to compare precision
 
@@ -141,5 +156,5 @@ When migrating or modifying search logic, **always compare precision output** be
    # Rust
    ./target/release/vector-db-benchmark --engines "redis-m-16-ef-128" --datasets "h-and-m-2048-angular-filters"
    ```
-2. Compare `mean_precisions` (Python) vs `mean_precision` (Rust) — values should match within floating-point tolerance
-3. If precision differs, check: score conversion, neighbor ordering, distance metric, and top-k cutoff logic
+2. Compare `mean_precisions` (Python) vs **`mean_recall`** (Rust) — the matching pair; values should agree within floating-point tolerance on full-width ground truth. `scripts/v0_check.sh` does this mapping for you. Comparing Python `mean_precisions` against Rust `mean_precision_at_returned` compares recall with precision.
+3. If it differs, check: score conversion, neighbor ordering, distance metric, top-k cutoff logic — and whether the dataset's ground-truth rows are shorter than `top` (`metrics_schema.ground_truth.queries_with_fewer_than_top_neighbours` in the Rust output), in which case the two denominators legitimately disagree.

README.md

@@ -340,7 +340,8 @@ Results JSON includes separate metrics for both operation types:
 {
   "results": {
     "rps": 5891.2,
-    "precision": 0.9785,
+    "mean_precision_at_returned": 0.9785,
+    "mean_recall": 0.9785,
     "p50_time": 0.00032,
     "p95_time": 0.00089,
     "p99_time": 0.00142,
@@ -395,8 +396,8 @@ Most datasets are automatically downloaded on first use. The image includes `ran
 | Random-100: Small synthetic dataset                                                                        |        100 |         100 |         9 |         9 | Cosine    |
 | Random-100-Euclidean: Small synthetic dataset                                                              |        100 |         100 |         9 |         9 | L2        |
 | **Filtered Search Datasets**                                                                               |            |             |           |           |           |
-| H&M-2048: Fashion product embeddings (with filters)                                                        |      2,048 |     105,542 |     2,000 |       100 | Cosine    |
-| H&M-2048: Fashion product embeddings (no filters)                                                          |      2,048 |     105,542 |     2,000 |       100 | Cosine    |
+| H&M-2048: Fashion product embeddings (with filters)                                                        |      2,048 |     105,542 |    10,000 |    ≤ 25 † | Cosine    |
+| H&M-2048: Fashion product embeddings (no filters)                                                          |      2,048 |     105,542 |    10,000 |        10 | Cosine    |
 | ArXiv-384: Academic paper embeddings (with filters)                                                        |        384 |   2,205,995 |    10,000 |       100 | Cosine    |
 | ArXiv-384: Academic paper embeddings (no filters)                                                          |        384 |   2,205,995 |    10,000 |       100 | Cosine    |
 | Random Match Keyword-100: Synthetic keyword matching (with filters)                                        |        100 |   1,000,000 |    10,000 |       100 | Cosine    |
@@ -420,6 +421,8 @@ Most datasets are automatically downloaded on first use. The image includes `ran
 | **Multi-Tenancy** (many tenants share one index; every query scoped to one tenant)                          |            |             |           |           |           |
 | Random-768-100-tenants: 100 tenants, per-tenant scoped queries (tenant field `a`)                          |        768 |   1,000,000 |       200 |        25 | Cosine    |
 
+† **The "Neighbors" column is the ground-truth width, and it bounds what recall can mean.** H&M-2048 (with filters) is not uniform: its 10,000 queries carry between **1 and 25** true neighbours (mean 23.4; 931 queries have fewer than 25), because a filtered query only has as many true neighbours as the filter admits. Our `mean_recall` divides by the neighbours that actually exist, so a 1-neighbour query can still score 1.0; upstream `qdrant/vector-db-benchmark` always divides by `top`, so the same query caps at `1/top`. A run's `metrics_schema.ground_truth` block reports the measured profile and the resulting ceiling — at `top: 100` H&M's ceiling is 0.233, so a `calibration_precision` above that is unreachable by construction. Configs that do not set `top` derive it from the ground-truth row width (25 here, 10 for the no-filters variant), which is why this matters mostly when `top` is set explicitly.
+
 ### Generating local datasets
 
 The sparse-vector, hybrid (dense+sparse fusion), and multi-datatype filter code

scripts/v0_check.sh

@@ -62,9 +62,21 @@ failed = False
 print(f\"{'Metric':<20} {'Python v0':>15} {'Rust':>15} {'Status':>20}\")
 print('=' * 72)
 
+# Python v0 (like upstream qdrant/vector-db-benchmark) computes
+# len(ids & expected[:top]) / top and stores it under 'mean_precisions'. That is
+# recall@top, i.e. the Rust 'mean_recall' field — NOT the Rust
+# 'mean_precision_at_returned', whose denominator is the number of results the
+# engine returned (#217). The two agree only on full-width ground truth, which
+# every dataset this script runs has; comparing the wrong pair here is precisely
+# the mistake the rename was meant to make impossible.
+KEY_MAP = {
+    'mean_precisions': 'mean_recall',
+}
+
 for key in ['mean_precisions', 'rps', 'mean_time', 'p50_time', 'p95_time', 'p99_time']:
+    rs_key = KEY_MAP.get(key, key)
     pv = py_r.get(key, 0.0)
-    rv = rs_r.get(key, 0.0)
+    rv = rs_r.get(rs_key, 0.0)
 
     if key == 'mean_precisions':
         if abs(pv - rv) < 0.001:
@@ -81,7 +93,8 @@ for key in ['mean_precisions', 'rps', 'mean_time', 'p50_time', 'p95_time', 'p99_
     else:
         status = 'PASS (Rust <= Py)' if rv <= pv * 1.5 else 'WARN (Rust > Py)'
 
-    print(f'{key:<20} {pv:>15.6f} {rv:>15.6f} {status:>20}')
+    label = key if rs_key == key else f'{key}→{rs_key}'
+    print(f'{label:<20} {pv:>15.6f} {rv:>15.6f} {status:>20}')
 
 print()
 if failed:

src/bin/vector_db_benchmark/engine/mod.rs

@@ -70,7 +70,16 @@ pub struct UpdateSearchRatio {
 pub struct SearchResults {
     pub total_time: f64,
     pub mean_time: f64,
-    pub mean_precision: f64,
+    /// Mean per-query precision, denominator = results the engine actually
+    /// returned: `hits / |deduped results kept|`. Emitted as
+    /// `mean_precision_at_returned`, NOT as `mean_precisions` — upstream
+    /// `qdrant/vector-db-benchmark` publishes recall@top under that key (#217).
+    /// `-1.0` is the filter-only sentinel (no vector search, no quality metric).
+    pub mean_precision_at_returned: f64,
+    /// Mean per-query recall, denominator = the valid, deduped ground-truth ids
+    /// that exist in `expected[:top]` (so a query with 3 true neighbours can
+    /// reach 1.0). Equals upstream's `mean_precisions` only when every
+    /// ground-truth row is at least `top` wide.
     pub mean_recall: f64,
     /// 10th-percentile per-query recall — the "worst 10%" floor. A healthy mean
     /// with a near-zero p10 means a slice of queries return almost nothing (e.g.
@@ -89,7 +98,8 @@ pub struct SearchResults {
     pub p50_time: f64,
     pub p95_time: f64,
     pub p99_time: f64,
-    pub precisions: Vec<f64>,
+    /// Per-query precision-at-returned samples (see `mean_precision_at_returned`).
+    pub precisions_at_returned: Vec<f64>,
     pub latencies: Vec<f64>,
     pub top: usize,
     /// Number of *successful* queries folded into the latency/quality stats.
@@ -331,7 +341,7 @@ pub fn compute_search_stats(
     Ok(SearchResults {
         total_time,
         mean_time,
-        mean_precision: mean(precisions),
+        mean_precision_at_returned: mean(precisions),
         mean_recall: mean(recalls),
         recall_p10,
         mean_mrr: mean(mrrs),
@@ -346,7 +356,7 @@ pub fn compute_search_stats(
         p50_time: pct(0.50),
         p95_time: pct(0.95),
         p99_time: pct(0.99),
-        precisions: precisions.to_vec(),
+        precisions_at_returned: precisions.to_vec(),
         latencies: times.to_vec(),
         top,
         num_queries: times.len(),

src/bin/vector_db_benchmark/engine/mongodb_engine.rs

@@ -303,7 +303,8 @@ impl MongoDBEngine {
         // Route latency stats through the shared percentile path (linear
         // interpolation) so filter-only is measured on the same footing as the
         // main search(). Filter-only has no precision/recall: signal that with the
-        // mean_precision == -1 sentinel, an empty precisions vec, and top == 0.
+        // mean_precision_at_returned == -1 sentinel, an empty precisions vec,
+        // and top == 0.
         let mut results = crate::engine::compute_search_stats(
             &times,
             &[],
@@ -315,7 +316,7 @@ impl MongoDBEngine {
             parallel,
             num_to_run,
         )?;
-        results.mean_precision = -1.0;
+        results.mean_precision_at_returned = -1.0;
         Ok(results)
     }
 

src/bin/vector_db_benchmark/engine/redis.rs

@@ -945,7 +945,8 @@ impl RedisEngine {
         // Route latency stats through the shared percentile path (linear
         // interpolation) so filter-only is measured on the same footing as the
         // main search(). Filter-only has no precision/recall: signal that with the
-        // mean_precision == -1 sentinel, an empty precisions vec, and top == 0.
+        // mean_precision_at_returned == -1 sentinel, an empty precisions vec,
+        // and top == 0.
         let mut results = crate::engine::compute_search_stats(
             &times,
             &[],
@@ -957,7 +958,7 @@ impl RedisEngine {
             parallel,
             num_to_run,
         )?;
-        results.mean_precision = -1.0;
+        results.mean_precision_at_returned = -1.0;
         Ok(results)
     }
 

src/bin/vector_db_benchmark/engine/valkey.rs

@@ -676,7 +676,8 @@ impl ValkeyEngine {
         // Route latency stats through the shared percentile path (linear
         // interpolation) so filter-only is measured on the same footing as the
         // main search(). Filter-only has no precision/recall: signal that with the
-        // mean_precision == -1 sentinel, an empty precisions vec, and top == 0.
+        // mean_precision_at_returned == -1 sentinel, an empty precisions vec,
+        // and top == 0.
         let mut results = crate::engine::compute_search_stats(
             &times,
             &[],
@@ -688,7 +689,7 @@ impl ValkeyEngine {
             parallel,
             num_to_run,
         )?;
-        results.mean_precision = -1.0;
+        results.mean_precision_at_returned = -1.0;
         Ok(results)
     }