Benchmark reciprocal rank fusion against the current hybrid fusion #1748

Summary

Hybrid search merges its full-text and vector legs with fused = max(f, v) + 0.3 × min(f, v) (FUSION_FORMULA_VERSION = "max+0.3*min/v1", repository/search_reader.py). Evaluate reciprocal rank fusion (RRF) against it on the retrieval benchmarks, and adopt RRF only if it measures better.

This is deliberately kept out of the #1746 search rework. That rework changes cost, not ranking, so its before-and-after benchmarks must match exactly. Changing fusion at the same time would mix a quality change into a performance change, and a recall shift could not be attributed to either.

Why look at it

• The current formula mixes two scales. Full-text scores are divided by the query's best full-text score, so the top lexical hit is always 1.0 however weak it is. Vector scores are raw cosine similarity, about 0.5 to 0.95. A weak keyword match can outrank a strong semantic one, which overlaps with Hybrid search and the similar-notes advisory return weak vector matches #1718 (weak vector matches in hybrid).
• RRF ranks by position only, score = Σ 1 / (k + rank) with k ≈ 60, so the scales stop mattering. It also discards magnitude: a barely related vector hit at rank 1 counts as much as a near-exact one.

Candidates

1. The current formula (baseline).
2. RRF with k = 60, plus a small sweep of k.
3. The current formula with calibrated full-text scores, for example a fixed bm25 / ts_rank normalization in place of the per-query maximum.
4. Any of the above with the Hybrid search and the similar-notes advisory return weak vector matches #1718 weak-match floor applied.

Measure

• test-int/test_search_performance_benchmark.py::test_benchmark_search_quality_recall_by_mode (hit@1, recall@5, MRR@10 for hybrid)
• just test-semantic-report (semantic quality and reranker quality)
• benchmarks/ LoCoMo suite: recall@5/10, MRR, content_hit (just bench-run-long with BM_LOCAL_PATH)

Run with the reranker off and on, since a reranker reorders the fused prefix and can mask fusion differences.

Acceptance

Adopt a new formula only if it does not regress any mode on any benchmark and improves hybrid hit@1 or MRR. Bump FUSION_FORMULA_VERSION when it changes.

Related

• Rework search: rank by id in the database, fetch text only for the returned page #1746 search rework (keeps the current fusion)
• Hybrid search and the similar-notes advisory return weak vector matches #1718 hybrid search returns weak vector matches
• Add a rerank stage to search: ~half of LoCoMo benchmark misses are ranking failures, and there is ~20x latency headroom #950 reranking