Legal Information Retrieval
Benchmark 2026
Compare search APIs on how often they return the official document behind a legal research request, a statute, a regulation, a court opinion or agency guidance, and whether they put it first.
- Last updated
- [ Updated Nov 03, 2026 ]
- Systems compared
- [ 5 systems ]
- Dataset
- [ 581 requests ]
How the systems stack up
Which search API finds the official document?
The Firecrawl Government & Legal Index ranks the official page first on 76.2% of requests, 4.8 points more than the runner-up, Claude Code with its own web search. It finds the official page somewhere in its top 10 on 96.6% of requests. A contradicting page makes its top 10 on 9.6% of requests, statistically level with Exa (9.6%) and Parallel (9.3%).
| Use case | Best pick | Measured result |
|---|---|---|
| Putting the official page first |  Firecrawl Government & Legal Index | 76.2% gold@1, 4.8 points ahead of the runner-up, Claude Code with its own web search |
| Finding the official page at all | Firecrawl Government & Legal Index | 96.6% gold@10, 6.1 points ahead of the runner-up, Parallel, a lead that holds at 95% confidence |
| Keeping contradicting pages out of the top 10 | Firecrawl Government & Legal Index Exa Parallel | 9.6%, 9.6% and 9.3% bad@10, statistically tied; the Claude Code setups return one or two addresses, so their near-zero bad@10 is not comparable |
Leaderboard
Share of requests with a 95% confidence interval. Switch the measure to see whether the official page came first, appeared at all, or whether a contradicting page got into the top 10.
Check all metric definitions- best quadrant
- worst quadrant
By cost per request
Cost per request is each engine's list price for one search call: Exa $0.007, Parallel $0.005, and the Government & Legal Index free, with no credits used and no API key required. For the two Claude Code setups it is what the agent spent per request, model tokens plus search fees.
By median latency
Median time per request with all three engines timed in the same run, from one client location; the Claude Code setups are timed from request to answer. The axis is logarithmic so the engines stay apart. The 95th percentile is in the results table.
The official page is the first result. The Government & Legal Index puts it first on 76.2% of requests, Exa on 59.6% and Parallel on 54.2%. The index's lead over each of them holds at 95% confidence. Claude Code with its own web search puts it first on 71.4%, and with no tools, answering from memory, on 47.8%.
By cost per request
Cost per request is each engine's list price for one search call: Exa $0.007, Parallel $0.005, and the Government & Legal Index free, with no credits used and no API key required. For the two Claude Code setups it is what the agent spent per request, model tokens plus search fees.
By median latency
Median time per request with all three engines timed in the same run, from one client location; the Claude Code setups are timed from request to answer. The axis is logarithmic so the engines stay apart. The 95th percentile is in the results table.
The official page is anywhere in the top 10. The Government & Legal Index finds it on 96.6% of requests, Parallel on 90.5% and Exa on 89.5%. The index's lead over each of them holds at 95% confidence. Claude Code returns the one or two addresses it settles on, so its gold@10 stays close to its gold@1: 74.4% with its own web search and 48.2% with no tools.
By cost per request
Cost per request is each engine's list price for one search call: Exa $0.007, Parallel $0.005, and the Government & Legal Index free, with no credits used and no API key required. For the two Claude Code setups it is what the agent spent per request, model tokens plus search fees.
By median latency
Median time per request with all three engines timed in the same run, from one client location; the Claude Code setups are timed from request to answer. The axis is logarithmic so the engines stay apart. The 95th percentile is in the results table.
A page that contradicts the official document is in the top 10, so lower is better. The Government & Legal Index at 9.6%, Exa at 9.6% and Parallel at 9.3% are statistically tied. A contradicting page came first on at most four of the 581 requests for any engine. The Claude Code setups return one or two addresses rather than a top 10, so their 0.2% and 0.0% are not comparable with the engines.
Full results
Every measure for the 3 engines and the 2 Claude Code reference setups, each with its 95% interval. The best value among the engines in each column is highlighted.
| # | System | gold@1 (higher is better) | gold@10 (higher is better) | bad@10 (lower is better) | Median latency (lower is better) | P95 latency (lower is better) | Cost per request (lower is better)list price |
|---|---|---|---|---|---|---|---|
| 01 | Firecrawl Government & Legal IndexFirecrawl's index of US federal, state and local government pages, queried through /search/gov. Searches use no credits. | 76.2%[72.6, 79.5] | 96.6%[94.7, 97.8] | 9.6%[7.5, 12.3] | 478 ms | 632 ms | $0 |
| 02 |  Claude Code + built-in searchagentClaude Code (Sonnet 5.5) with its own web search, no search API, as a reference. It returns the one or two addresses it settles on. Cost is the agent's tokens plus search fees per request. | 71.4%[67.6, 75.0] | 74.4%[70.7, 77.7] | 0.2%[0.0, 1.0] | 11,045 ms | 31,831 ms | $0.093 |
| 03 | Exa | 59.6%[55.5, 63.5] | 89.5%[86.7, 91.7] | 9.6%[7.5, 12.3] | 447 ms | 510 ms | $0.007 |
| 04 | Parallel | 54.2%[50.2, 58.2] | 90.5%[87.9, 92.7] | 9.3%[7.2, 11.9] | 1,148 ms | 3,271 ms | $0.005 |
| 05 | Claude Code, no toolsagentClaude Code (Sonnet 5.5) answering from memory, no tools, as a reference. Cost is the agent's tokens per request. | 47.8%[43.8, 51.9] | 48.2%[44.2, 52.3] | 0.0%[0.0, 0.7] | 4,145 ms | 8,685 ms | $0.016 |
How is legal information retrieval measured?
The data flywheel
A search benchmark only says how an engine did on the questions in it, and a fixed set goes stale: once agents run its questions against live search, the questions, their leaked answers and even the agents' own queries end up on indexed pages that later runs find 12. So this benchmark runs as a loop. Inside Firecrawl it has turned eight times; this release, benchmark-legal-retrieval, is the first turn published, as a way of keeping the work open. The loop is also how the Firecrawl Government & Legal Index gets better.
“How much notice does a landlord have to give before an eviction in Texas?”
- Written fromA customer search with the same jurisdiction, document type and task
- JurisdictionTexas
- DocumentStatute
- TaskFind the notice period
An example of the shape a request takes, not an item from the set.
Start from what customers ask
Each version starts from a sample of the past week's searches on the index, at most one per customer team and never from organizations with zero data retention 3. A model rewrites each search as a twin: a new search with the same jurisdiction, document type, task and phrasing style that asks for a different official document 4. No question is kept, dropped or reworded because of how any engine did on it; the engines under test take no part in building the set 5.
Build the answer key from official sources
Each question has one right answer: the document on its official publisher's site, plus any other official address for it. The top 10 results from seven sources are pooled and labeled by models, copies of the text on other sites marked as copies and pages that contradict the document marked wrong 6. Scoring goes by URL, so the same results always get the same score.
Freeze the version
Once a version is published, its questions and answer key never change. Anything that needs changing goes into the next version, so scores on a version stay comparable for as long as it is in use.
Score every engine on the same questions
The Firecrawl Government & Legal Index, Exa and Parallel answer the same questions, and Claude Code answers them twice as a reference, with its own web search and with no tools. They are compared question by question, with 95% ranges. On this release the index also ran twice, a couple of hours apart, and its top-10 score did not move.
Find and fix the misses
For every question the index misses, the trace says whether the right page is missing from the index, ranked too low, or behind a link that does not load. The search team works through that list, and the next version, built from the following week's searches, is scored before any of its misses reach them, so each version's first score shows how well the fixes carry over. Evaluation drives the fix, and the fix drives the next evaluation 78.
Eight turns, one published
This release has 581 legal and regulatory searches: statutes, regulations, court opinions, agency guidance, tax rulings, bills and local ordinances. Every turn of the data flywheel showed exactly where the index fell short, and after many rounds of improvements it finds the official page in its top 10 on 96.6% of requests, ahead of Parallel (90.5%) and Exa (89.5%) in the same run.
What comes next
The loop keeps turning every week inside Firecrawl: each Friday the past week's real customer searches on the index are sampled, half become a fix list for the search team, and the other half only keep score, so the index is checked week after week on questions nobody tuned for. A new public version of the benchmark ships with each new release of the Government & Legal Index, built from newer searches and scored before any of its misses reach the team. That first score is the number to watch.
The 290 public requests, their answer key and the scoring code are open, so any search API can be scored on the same tasks. Browse the code and data at firecrawl/benchmark-legal-retrieval.
Dataset
581 requests scored on the full set, 290 of them public. Each is a twin of a real request to the Government & Legal Index.
- Legal research requests
581 twins of real requests to the Firecrawl Government & Legal Index, 290 public and 291 private. A twin keeps the kind of document and the jurisdiction of a real request and asks for a different official document. Each twin has one official answer with its official aliases and its known near matches: the same text on a mirror site, or an official page closely tied to it, such as its parent section or the agency's page about it.
Example request shape“How much notice does a landlord have to give before an eviction in Texas?”
Metrics
How each measure is defined, how the engines are driven and scored, and where the requests come from.
gold@1, gold@10 and bad@10
gold@1 is the share of requests where the official page is the first result. gold@10 is the share where it appears anywhere in the top 10. bad@10 is the share where a page labeled as contradicting the official document appears in the top 10, so lower is better; a contradicting page came first on at most four of the 581 requests for any engine.
gold@1 is the headline because for a government index the official page has to come first: an agent or a person acts on the first result, and a copy or a summary in that position is a miss. The release also reports hit@10, which credits near matches such as mirror copies, and recall@10, the share of the official page and its near matches found in the top 10; both are left off this page because the index covers official sources only, so mirror copies sit outside it by design.
How engines are driven
Every engine gets the same 581 requests on the same day and returns its top 10 results with its default settings: the Government & Legal Index, queried through /search/gov, Exa and Parallel. Claude Code (Sonnet 5.5) answers the same requests twice as a reference, once with its own web search and once with no tools, and is scored on the addresses it returns.
Scoring
Every request has one official answer, read and labeled before any engine is scored and then frozen. Scoring checks by canonical URL whether that answer is first, in the top 10, or near-matched; no model judges any score, so the same run always gets the same score.
Each score carries a 95% Wilson interval over the 581 requests. Gaps between engines are tested request by request with a paired bootstrap, and a lead is claimed only when that interval excludes zero.
Where the questions come from
Every week Firecrawl samples requests to the Government & Legal Index, at most one per customer team, leaving out its own teams and any organization with zero data retention. A model writes a twin of each: a new search with the same jurisdiction, document type, task and phrasing style that asks for a different official document. The set is built from the twins, and no question is kept, dropped or reworded because of how any engine did on it.
Each question's answer is the document on its official publisher's site, plus any other official address for it. The top 10 results from seven sources are pooled and labeled by models before any engine is scored: copies of the text on other sites are marked as copies, and pages that contradict the document are marked wrong. The key is then frozen.
Latency and list price
Latency is the wall-clock time per request with all three engines timed in the same run, from one client location, separate from the scores. Cost per request is each engine's list price for one search call: Exa $0.007, Parallel $0.005, and the Government & Legal Index free, with no credits used and no API key required. A full pass over the 581 requests costs $4.07 on Exa, $2.91 on Parallel and nothing on the index. Claude Code ran the same 581 requests as a reference, once with its own web search and once with no tools; for those two rows cost is what the agent spent per request, model tokens plus search fees, and latency runs from request to answer.
Frequently asked questions
Changelog
Dated snapshots of this benchmark and what changed in each run.
- Nov 03, 2026First public release (benchmark-legal-retrieval), the eighth turn of the loop and the first one published. Three search APIs on 581 twins of real legal research requests, scored by exact match against the official document on gold@1, gold@10 and bad@10, from the run of 3 Nov 2026. Latency from the run of 31 Oct 2026; cost per request is each engine's list price.
The public half of each version, with its answer key and the scoring code, lives in firecrawl/benchmark-legal-retrieval.
Related
The product behind this benchmark, and where to read more.
- Firecrawl Government & Legal Indexwww.firecrawl.dev
- Government Index documentationdocs.firecrawl.dev
- benchmark-legal-retrieval on GitHubgithub.com
References
The sources cited by number in the data flywheel section.
- 1Anthropic, Eval awareness in Opus 4.6 on BrowseComp, 2026
- 2Wang et al., Search-Time Contamination in Deep Research Agents: Measuring Performance Inflation in Public Benchmark Evaluation, 2026
- 3Craswell et al., ORCAS: 18 Million Clicked Query-Document Pairs for Analyzing Search, 2020
- 4Carranza et al., Synthetic Query Generation for Privacy-Preserving Deep Retrieval Systems using Differentially Private Language Models, NAACL 2024
- 5Kim et al., AutoBenchmark: benchmark creation and the role of humans, Meta FAIR, 2026
- 6Voorhees and Harman, Overview of TREC 2001, NIST
- 7Shankar, Data Flywheels for LLM Applications
- 8Shukla et al., Adaptive Data Flywheel: Applying MAPE Control Loops to AI Agent Improvement, 2025