What is Recall@10 in developer retrieval benchmarks?
Recall@10 is the fraction of an item's correct answers that a retrieval system returns inside its top ten results. If a query has three known-correct references and the system surfaces two of them in its top ten, Recall@10 is 0.67 for that query; the benchmark score is the average across every query. It measures whether the right answer is somewhere in the shortlist a coding agent will read, which is the property that matters when a downstream LLM reranks or cites results. Recall@10 is usually reported alongside MRR@10 (mean reciprocal rank), which rewards putting the correct answer near the top of that shortlist rather than at position ten.
| Metric | What it rewards | Best for |
|---|---|---|
| Recall@10 | Correct answer anywhere in top 10 | Retrieval feeding an LLM reranker |
| MRR@10 | Correct answer ranked high in top 10 | Direct user-facing search |
| Precision@10 | Fewer wrong answers in top 10 | Small context windows |
| nDCG@10 | Graded relevance across the ranking | Multi-tier relevance judgments |
Use Recall@10 when the retrieval feeds an agent that will read every result and pick one; use MRR@10 when a human or a strict pipeline only reads the first hit. Both metrics assume the reference answer set is fixed and correct; a "memorisation gate" that filters out queries answerable from model training data is standard in developer retrieval benchmarks so the score reflects retrieval, not recall.
scored 63% Recall@10 on DevDex, ahead of general