All insights
Synthetic benchmarks flatter every retrieval strategy. What changes when the corpus is ten years of scanned PDFs and duplicate policies.
Every retrieval benchmark we have seen is run on clean text: well-formed articles, one topic per document, no duplicates. Every corpus a client has actually handed us is ten years of scanned PDFs, four versions of the same policy with different dates, and a wiki nobody has pruned since 2019.
The strategies that win on the benchmark do not reliably win on the second kind of corpus. Here is what changed when we started evaluating on the real thing.
Ingestion is most of the work
OCR quality, table extraction and header detection decided more of the final accuracy than any choice of embedding model. A page whose columns were merged during extraction cannot be retrieved correctly by anything downstream.
We now budget more time for ingestion than for the retrieval pipeline, and we evaluate ingestion on its own before touching a vector store.
Duplicates are the silent killer
When four near-identical versions of a policy exist, retrieval returns all four, the context window fills with repetition, and the current version is as likely to be dropped as the stale ones. Deduplication with version awareness, so the newest supersedes the rest, gave a larger lift than reranking.
Metadata beats semantics for half the questions
"What is the leave policy for contractors in Ontario?" is mostly a filtering problem: jurisdiction, audience, document type. Hybrid retrieval that filters on structured metadata first and searches semantically second was more accurate and far cheaper than pure embedding search on that class of question.
Evaluate on your corpus or not at all
The honest number comes from a question set drawn from real support tickets, answered against the real documents, scored by the people who would have answered them. It is lower than the benchmark number. It is also the only one that predicts what users will experience.
