fix(metrics): stop publishing precision under upstream's mean_precisions key

[fcostaoliveira]

Closes #217.

The bug

Our results JSON emitted "mean_precisions" carrying results.mean_precision = hits / |results returned|. Upstream qdrant/vector-db-benchmark (engine/base_client/search.py:49-52) emits a key of the same name defined as len(ids & expected[:top]) / top — recall@top, which is our mean_recall.

An engine returning 5 of 10 requested results, all correct, reported mean_precisions = 0.50 upstream and 1.00 here. The shared key name actively invites overlaying the two tools' files on one chart, which silently compared precision against recall.

Rename, not dual-emit — and why

The issue offers "rename" and/or "additionally emit an upstream-compatible recall_at_top". This PR renames, and deliberately does not keep mean_precisions alive:

1. Keeping the name is the one option ruled out. Any variant that keeps emitting mean_precisions — with either definition — leaves a key that means two things across the two tools. Re-pointing it at recall@top would be worse: our own historical files would then hold two different quantities under one name.
2. A true mean_recall_at_top aggregate cannot be added without breaking the query→ground-truth pairing. There appears to be a free derivation: recall_at_top_i = recall_i · t_i / k, reusing the per-query results.recalls we already keep and the per-row truth widths t_i that GroundTruthProfile already loads. It does not work, and the reason is structural rather than cosmetic: every engine's parallel harness pulls query indices off a shared atomic counter and merges per-thread sample buffers in thread-join order (e.g. redis.rs:2416-2424). recalls[j] therefore has no correspondence to ground-truth row j — the pairing is destroyed before aggregation. Recovering it means either re-keying every engine's sample buffers by query index or threading a fifth per-query vector through all 15 harnesses, two of which (elasticsearch.rs, opensearch.rs) are frozen behind PRs fix(opensearch): force-merge to one SEARCHABLE segment, matching elasticsearch (#210) #246 and fix(elasticsearch): disable periodic refresh during bulk upload, matching OpenSearch and upstream (Closes #240) #248.

So instead: the upstream formula gets one canonical, test-pinned implementation (QueryMetrics::recall_at_top in src/metrics.rs), and every result file names the field that IS upstream's number, when one is. comparable_to_upstream_mean_precisions: "mean_recall" is not a hint that the two are similar — it asserts that our mean_recall and upstream's mean_precisions are the same quantity computed the same way for this dataset and top, and may be overlaid directly. It is null exactly when no field in the file can be.

Emitted key changes (schema version 2)

before	after	definition
results.mean_precisions	results.mean_precision_at_returned	hits / |deduped results kept|
results.precisions (only under --dump-raw-latencies)	results.precisions_at_returned	per-query array of the above
results.precision_dist	results.precision_at_returned_dist	digest of that array
—	metrics_schema (new, top-level)	formulas + ground-truth width profile + comparable_to_upstream_mean_precisions
—	params.calibration (new, when calibrating)	param, target, achieved, reached_target, ceiling, note

mean_recall, recall_p10, recall_dist, mrr/ndcg and every timing field are unchanged. Summary files rename the same key inside search_results[] and additionally carry metrics_schema_version: 2, metrics_definitions, and uncalibrated_configs.

Which published numbers are affected, and what stops being comparable

• No measured value changes. mean_precision_at_returned is bit-for-bit the number that used to be called mean_precisions; the filter-only -1.0 sentinel is unchanged. Nothing about how any engine is measured moved.
• Every previously published mean_precisions value was precision, not recall@top. Any chart, table or dashboard that put our mean_precisions on the same axis as upstream's (or Python v0's) was comparing precision to recall. Where the ground truth is full width and the engine returned a full page, the two coincide numerically, so most published dense-benchmark curves are unaffected in value. Where they diverge — short/filtered ground truth, or engines returning fewer than top — the old overlays were wrong and re-reading them against mean_recall is the fix.
• Old result files stay readable but need a mapping: their mean_precisions is today's mean_precision_at_returned. --chart handles this itself (plot.rs reads the new key, falling back to the legacy one, and warns when a summary lacks metrics_schema_version because our pre-v2 files and upstream's files are then indistinguishable). Any external SQL/notebook selecting results.mean_precisions from new files gets NULL and must be pointed at mean_precision_at_returned — or at mean_recall, if what it wanted was the upstream-comparable quantity.
• v0/tools/upload_results_postgres.sh loads .results.mean_precisions into a mean_precisions column; it now falls back to .results.mean_recall for Rust files, which is the quantity that column has always held for Python v0 rows. Rows previously loaded from Rust files into that column held precision and are not comparable with the Python-loaded rows.
• scripts/v0_check.sh compared Python mean_precisions against Rust mean_precisions — i.e. recall against precision. It now maps to Rust mean_recall. It passed only because its datasets have full-width ground truth.

Calibrator (second half of the issue) — and no, the shipped calibrations are not fiction

calibrate() binary-searches on mean_precision_at_returned, whose denominator is results returned. On a dataset whose ground truth is narrower than top, the metric is hard-capped and the target is unreachable no matter how high ef goes — the sweep walked ef to the 1000 bound and printed Calibrated ef=1000 → precision=0.24 as if it had succeeded.

Are any shipped/published calibration numbers actually affected? No — verified, not assumed. Only two shipped configs set calibration_precision (both 0.95 at "top": 100): cohere-calibration.json and dbpedia-calibration.json. Both were profiled directly from the real dataset files:

dataset	queries × neighbours	width	ceiling @ top=100
cohere-768-1M (full 2.9 GB download, decompressed)	10,000 × 100	all rows full	1.000000
dbpedia-openai-1M-{512,1024,1536,2048,3072}-angular (HDF5 neighbors read over HTTP range requests, ~3 MB each)	1,000 × 100	all rows full, all five variants	1.000000

So a 0.95 target is fully reachable on both, and no published cohere-cal-* / dbpedia-cal-* number is a fake calibration. The suspicion that the base DBpedia HDF5 lacked 100 neighbours (raised because a separate dbpedia-openai-1M-1536-angular-100neighbors entry exists) is disproved: the base file has 1000×100 full-width ground truth; the -100neighbors entry is a different packaging (tar), not a repair.

The h-and-m case is therefore a latent trap that no shipped config currently steps in, not evidence of a fake published number. It is still worse than hypothetical, in two ways:

• On the shipped h-and-m-2048-angular/hnm/tests.jsonl, all 10,000 rows are narrower than 100 (931 narrower than 25; min 1, max 25), so at top: 100 the ceiling is 0.2334 — and no shipped config pairs h-and-m with calibration only because none happens to.
• At h-and-m's natural top: 25 (every other shipped config leaves top unset, so it is derived from the ground-truth row width) the ceiling is 0.9337 — still below 0.95. Any 0.95 calibration on h-and-m is unreachable at either top. It only becomes reachable at top: 10, where the ceiling is 0.9592.

What the code now does:

• the ground-truth width profile is computed once per experiment (ground_truth.rs) and the target is bounded against it before the first search, printing the full arithmetic when it is unreachable;
• on termination, a target that was not met prints a second warning naming the achieved value and whether the knob was pinned at the maximum swept value;
• the same warning is repeated at the end of the run (the up-front one is ~10 sweep iterations of scrollback away by then) and recorded in both params.calibration.reached_target per search file and uncalibrated_configs in the summary — so a CI sweep that "calibrated" against an unreachable target no longer exits 0 with a silent artifact.

The ceiling is treated as a warning, not a hard error: a filtered query that legitimately returns fewer than top results can score above it, so failing the run would break valid filtered-dataset calibrations. A --fail-on-uncalibrated gate (mirroring --fail-on-dropped-queries) is a reasonable follow-up but is not in this PR.

Documentation correctness

README.md claimed both H&M-2048 rows have 100 neighbours per query and 2,000 test queries. Measured on the shipped files:

dataset	test queries	ground-truth width
h-and-m-2048-angular-filters	10,000	1 to 25, mean 23.4 (931 queries below 25)
h-and-m-2048-angular-no-filters	10,000	10, uniform

Neither is 100, and neither is 2,000 queries. That false line is exactly the belief that produced this bug, so it is corrected here — with a footnote explaining that the column bounds what recall can mean — rather than left to outlive the fix.

Performance

GroundTruthProfile::load is skipped when nothing consumes it (--skip-search, filter-only runs). Otherwise an upload-only run would pay a full query-file parse per config — on the compound h-and-m dataset that parses every 2048-dim query vector only to drop it (~22 s warm / ~137 s cold per config; a 12-config sweep pays it twelve times). It is not inside any timed window, but it was a real --skip-search regression.

Tests

• src/metrics.rs: the canonical 5-of-10 case (precision 1.00 / recall 0.50 / upstream 0.50), a short-ground-truth row (top=100, one true neighbour: 0.01 / 1.00 / 0.01 — a 100x split on identical data), the full-width case where all three agree, and sentinel padding shrinking only our denominator.
• src/bin/vector_db_benchmark/ground_truth.rs: width profiling, ceiling arithmetic, dedup/sentinel handling, unreachable-target detection (NaN excluded, +inf still flagged).
• src/bin/vector_db_benchmark/experiment.rs: the emitted document must not contain mean_precisions, precisions or precision_dist; must contain the new names, metrics_schema and the calibration block; calibration outcomes reported both ways.
• src/bin/vector_db_benchmark/summary.rs: summaries carry metrics_schema_version: 2, no upstream key name, and list only genuinely unreached calibrations.
• plot.rs: legacy mean_precisions summaries still chart, and the new key wins when both are present.

cargo test --lib --bins (694 + 98 + 2 pass), cargo clippy --all-targets -- -D warnings and cargo fmt --check are clean. elasticsearch.rs and opensearch.rs are untouched.

🤖 Generated with Claude Code

[github-actions]

Docker Build Validation

Platforms tested:

• linux/amd64 (built and tested)
• linux/arm64 (build validated)

Tests performed:

• --help command
• --describe datasets command
• Integration test (random-100 dataset + Redis)
• Multi-platform build validation (amd64 + arm64)

[github-actions]

Docker Build Validation

Platforms tested:

• linux/amd64 (built and tested)
• linux/arm64 (build validated)

Tests performed:

• --help command
• --describe datasets command
• Integration test (random-100 dataset + Redis)
• Multi-platform build validation (amd64 + arm64)