Developer Retrieval
Benchmark 2026
Compare developer retrieval tools on how reliably they surface the repository behind a described capability, the pull request that fixed a bug, and the docs page that answers a how-to.
- Last updated
- [ Updated Aug 21, 2026 ]
- Systems compared
- [ 8 systems ]
- Dataset
- [ 1,179 tasks ]
Results: 8 systems ranked by Overall recall
Firecrawl Developer Index
ParallelFirecrawl Search
Mintlify
Exa
Native web search[ctl]
Context7| # | System | Repo (higher is better) | Issue/PR (higher is better) | Docs (higher is better) | Overall recall (higher is better) | Overall MRR (higher is better) |
|---|---|---|---|---|---|---|
| 01 | Firecrawl Developer IndexFirecrawl's curated developer index of issues, PRs, READMEs, and docs, queried through its MCP server. | 76.1% | 66.0% | 47.2% | 63.1% | 0.596 |
| 02 | Parallel | 81.9% | 62.9% | 28.2% | 57.7% | 0.561 |
| 03 | Firecrawl SearchFirecrawl's general web search (/search), a different surface from the developer index. Included here for context. | 77.6% | 56.5% | 38.6% | 57.6% | 0.549 |
| 04 | Mintlify | 74.3% | 56.0% | 33.4% | 54.6% | 0.538 |
| 05 | Exa | 73.3% | 58.5% | 29.3% | 53.7% | 0.538 |
| 06 | Native web searchcontrolThe agent's own built-in web search, not a developer index. A reference point rather than a product. | 80.7% | 27.5% | 28.0% | 45.4% | 0.458 |
| 07 | Context7Docs-only index, second on the docs track. Read its overall score as coverage, not quality. | 0.8% | 2.9% | 46.6% | 16.8% | 0.144 |
| 08 | No toolscontrolThe agent answering from memory with no search at all. The floor every row is read against. | 0.5% | 0.0% | 3.4% | 1.3% | 0.009 |
How is developer retrieval measured?
Three retrieval tracks
The retrieval tasks developer agents spend most of their tool calls on.
Retrieval-forcing questions
Questions are written so the answer cannot come from memory: repository names are hidden and documentation questions are paraphrased from the source.
Deterministic gold references
Each item is scored against fixed gold references, so scoring is exact and no model judges the results.
Memorization check
Run the same evaluation with search disabled.
Final benchmark
Only retrieval-dependent items are scored.
The evaluation harness and a public half of the dataset are open source, so any retrieval system can be scored on the same tasks. Browse the code and data at firecrawl/benchmark-devdex.
1,179 tasks scored on the full sets. A 594-item public sample, about half of each track, ships for reproduction.
- Repository
Repository queries are created from capability descriptions of real, verifiable repositories. The dataset includes projects across a wide range of popularity levels rather than focusing only on well-known repositories. The repository name is never included in the query.
Example query“how to automatically route LLM classification requests to a cheaper model when confident and fall back otherwise”- Issue to fix
Issue and pull request queries are built from real issue and pull request pairs. For these questions, citing either the issue or the pull request is considered correct.
Example query“scikit-learn logisticregression random_state not workin”- Documentation
Documentation queries are created from real passages in a library's documentation. Each question is phrased differently from the wording used in the original passage. This reduces the possibility of matching the query directly to the source through surface-level language similarity.
Example query“How do I add Pydantic to my project using uv instead of pip?”
Two metrics, computed deterministically against fixed golds, plus how each system is driven and the gate that keeps it honest.
Recall@10 and MRR@10
Two metrics, both computed over the agent's first ten citations. Recall@10 is the fraction of an item's correct answers that were cited. MRR@10 is the reciprocal rank of the first correct citation, so citing it first scores higher than citing it tenth.
The overall column is the equal-weight mean of the three track scores. A system is only given an overall score when it runs all three tracks.
How systems are driven
One agent (Claude Opus 4.8) drives every system through the same harness with a single search tool, ten results per call, and the same retrieval depth. Each system is called with its default configuration, and a run that returns nothing counts as a miss rather than being dropped.
Two rows are controls, marked as such, and are reference points rather than products: Native web search is the agent's own built-in search, and No tools is the agent answering from memory with no search at all.
Scoring
Scoring is deterministic: correct answers are matched by reference and canonical URL against fixed golds, with no model judging any number, so the same run reproduces the same score.
A memorisation gate removes any item the model could already answer with search disabled, so the benchmark measures retrieval rather than recall of training data. The floor that leaves is the No-tools control: 0.5% on repo, 0.0% on issue/PR, and 3.4% on docs.
Frequently asked questions
What is the Developer Retrieval benchmark?
It measures how reliably a tool surfaces the right developer answer for a query, across three tracks: finding the GitHub repository behind a described capability, finding the issue or pull request where a bug was fixed, and finding the documentation page that answers a how-to. One agent drives every system with a single search tool and ten results per call, and answers are scored deterministically against known-correct golds, with no model judging the results.
Which tool retrieves developer content best?
The Firecrawl Developer Index has the highest overall Recall@10 at 63.1%, ahead of Parallel (57.7%), Firecrawl Search (57.6%), Mintlify (54.6%), and Exa (53.7%). It also leads the issue-to-fix track (66.0%) and the documentation track (47.2%). On the repository track the field is close and general web search is competitive, with Parallel (81.9%) and native web search (80.7%) ahead there. Because single-track leads are often statistically tied, the combined score is the one that separates systems.
What do Recall@10 and MRR@10 mean?
Recall@10 is the fraction of an item's correct answers that appear in the agent's first ten citations. MRR@10 is the reciprocal rank of the first correct citation, so a system that cites the right answer first scores higher than one that cites it tenth. Recall rewards finding the answer at all; MRR rewards ranking it near the top.
What does the Context7 caveat mean?
Context7 is a documentation index, and it is second overall on the docs track at 46.6%. But repository and issue/PR retrieval are outside its domain: roughly 78% and 52% of those runs return nothing, so its scores there are near zero. Its overall number therefore reflects coverage of one track out of three, not retrieval quality across all three.
What is the difference between the Firecrawl Developer Index and Firecrawl Search?
The Developer Index is a curated index of developer content, issues, pull requests, READMEs, and docs, queried through Firecrawl's MCP server. Firecrawl Search is the general /search endpoint, a different surface. On this benchmark the purpose-built Developer Index scores higher overall, 63.1% vs 57.6% Recall@10, and leads on issue-to-fix and documentation retrieval.
Is a model judging the results?
No. Every task has a known-correct answer, and scoring checks by reference and canonical URL whether that answer appears in the agent's citations and where it ranks. There is no model grading answers, so the same run reproduces the same score. A memorisation gate also removes any item the model could answer with search disabled, so the benchmark measures retrieval rather than memorised knowledge.
Can I reproduce these numbers?
The scores here are computed on the full task sets, and the benchmark ships a 594-item public sample, about half of each track, so the same comparison can be run without the full corpus. Every system is driven by the same agent, with the same single search tool and the same ten-result retrieval depth.
How often is this benchmark updated?
It is re-run and republished when results change materially or when a system is added or updated. Each snapshot is dated, and past changes are listed in the changelog on this page.
Changelog
- Aug 21, 2026First public snapshot (DevDex v1.0.0). Six retrieval tools and two controls compared on Recall@10 and MRR@10 across repository, issue-to-fix, and documentation retrieval.
Each snapshot is published from a pinned run in firecrawl/benchmark-devdex.