Introducing the Legal Retrieval Benchmark

Today we're publishing the Legal Retrieval Benchmark: 581 legal research requests for search APIs. Each request asks for one official document, a statute, a regulation, a court opinion or agency guidance, and every request is grounded in what Firecrawl customers actually search for.

The benchmark is one half of a loop we run every week. Customer requests to the Firecrawl Government & Legal Index tell us where the index falls short, the search team fills those gaps, and the next version measures whether they stayed filled. Inside Firecrawl that loop has already turned eight times. This release, benchmark-legal-retrieval, is the first turn we publish, because a benchmark nobody can inspect is a benchmark nobody can argue with. This post covers the loop, the dataset, each measure, and where the index stands against Exa and Parallel.

• The Legal Retrieval Benchmark measures whether a search API returns the official document behind a legal research request, a statute, a regulation, a court opinion or agency guidance, and whether it puts that document first. The release has 581 requests: 290 are public, with their answer key and the scoring code on GitHub, and 291 are held out.
• The Firecrawl Government & Legal Index puts the official page first on 76.2% of requests, ahead of Exa (59.6%), Parallel (54.2%) and Claude Code with its own web search (71.4%), finds it somewhere in the top 10 on 96.6%, and lets a contradicting page into its top 10 on 9.6%, statistically level with Exa (9.6%) and Parallel (9.3%).
• No model grades the published scores. Every request has one frozen official answer, and scoring is an exact match on the canonical URL, so the same run always gets the same score.
• Government & Legal Index searches are free: no credits, and no API key required. Exa lists $0.007 per request and Parallel $0.005. A full pass over the 581 requests costs $4.07 on Exa, $2.91 on Parallel and nothing on the index.
• The benchmark runs as a data flywheel: every version starts from a sample of recent customer searches, never from organizations with zero data retention, rewritten as twins, scored against an answer key built from official sources and frozen. Every engine answers the same questions, the index's misses go to the search team, and the next version is scored before anyone fixes against it. The loop has turned eight times inside Firecrawl; this is the first version published.

Figure 1: gold@1 with 95% intervals, against cost per request (left: the list price of one search call for the engines, the agent's spend in tokens and fees for the two Claude Code references) and median latency (right, on a log axis). Green marks the best quadrant, cheap or fast and good; red marks the worst. Scores are from the 3 Nov 2026 run; latency is from the 31 Oct 2026 run.

A government-search benchmark needs four things: requests that come from real demand, the official document as the only answer, deterministic grading, and a way to stay fresh. The Legal Retrieval Benchmark is built around those four, and it is rebuilt every week from new requests, which is also how the Firecrawl Government & Legal Index gets better.

People ask government search for one thing above all: the authoritative document. So a useful benchmark should:

1. Come from real demand. Requests should look like what customers send, not puzzles written to be hard.
2. Have the official document as the answer. A copy of a statute on a blog isn't the statute. The score should say whether the engine found the authoritative source, and whether it put that source first.
3. Be graded deterministically. If a model judges the answers, the score depends on the judge. Exact matching against a frozen key doesn't.
4. Stay fresh. Any fixed set goes stale once engines tune for it, ours included. The answer is new versions built from new requests.

A search benchmark only tells you how an engine did on the questions in it, and a fixed set goes stale: once agents run its questions against live search, the questions, their leaked answers and even the agents' own queries end up on indexed pages that later runs find.^[1][2] So we run ours as a loop instead of a one-off: every version starts from recent customer searches, the Firecrawl Government & Legal Index is fixed where the version shows gaps, and the next version is built from newer searches and scored before anyone fixes against it.

Figure 2: one turn of the loop, and one request as the set holds it. The animated version is on the benchmark page.

1. Start from what customers ask. Each version starts from a sample of the past week's searches on the index, at most one per customer team and never from organizations with zero data retention.^[3] A model rewrites each search as a twin: a new search with the same jurisdiction, document type, task and phrasing style that asks for a different official document.^[4] No question is kept, dropped or reworded because of how any engine did on it; the engines under test take no part in building the set.^[5]
2. Build the answer key from official sources. Each question has one right answer: the document on its official publisher's site, along with any other official address for it. We also pool the top 10 results from seven sources and have models label them. Copies on other sites that carry the same text are marked as copies, and pages that contradict the document are marked wrong.^[6] Scoring is automatic and goes by URL, so the same results always get the same score.
3. Freeze it. Once a version is published, its questions and answer key never change. Anything that needs changing goes into the next version, so scores on a version stay comparable for as long as it's in use.
4. Score every engine on the same questions. The Firecrawl Government & Legal Index, Exa and Parallel all answer the same questions, and Claude Code answers them twice as a reference, with its own web search and with no tools. We compare them question by question and report 95% ranges. On this release we also ran the index twice, a couple of hours apart, and its top-10 score didn't move.
5. Find and fix the misses. For every question the index misses, we check whether the right page is missing from the index, ranked too low, or behind a link that doesn't load. The search team works through that list, and the next version, built from the following week's searches, is scored before any of its misses reach them, so each version's first score shows how well the fixes carry over. Evaluation drives the fix, and the fix drives the next evaluation.^[7][8]

Eight turns, one published. This release has 581 legal and regulatory searches: statutes, regulations, court opinions, agency guidance, tax rulings, bills and local ordinances. Every turn of the data flywheel showed exactly where the index fell short, and after many rounds of improvements it finds the official page in its top 10 on 96.6% of requests, ahead of Parallel (90.5%) and Exa (89.5%) in the same run.

What comes next. The loop keeps turning every week inside Firecrawl: each Friday we sample the past week's real customer searches on the index, half become a fix list for the search team, and the other half only keep score, so we can see whether customers get better answers week after week. A new public version of the benchmark ships with each new release of the Government & Legal Index, built from newer searches and scored before any of its misses reach the team. That first score is the number to watch.

benchmark-legal-retrieval is 581 twins of real legal research requests: 290 public and 291 private. Each has one official answer with its official aliases and its known near matches, such as the same text on a mirror site or the agency's page about it. Each engine returns its top 10 for every request.

A near match is either the same text on a mirror such as Cornell LII or Justia, or an official page closely tied to the answer, such as its parent section or the agency's page about the rule. Near matches earn credit on one measure only, hit@10, which the release reports and this post leaves out, so the benchmark can say both whether an engine got the reader to the right text and whether it got them to the official copy of it. The twins, the aliases and the near-match labels all come out of the flywheel above, and they were frozen before any engine was scored.

Three measures, all computed from each engine's top 10 results against the frozen key: gold@1, gold@10 and bad@10, plus latency and list price per request. The headline is gold@1, because for a government index the official page has to come first.

• gold@1. The official page is the engine's first result. An agent or a person usually acts on the first result, so for an official document the official page has to come first.
• gold@10. The official page is anywhere in the top 10. This is the plain question of whether the engine found the document at all.
• bad@10. A page labeled as contradicting the official document is in the top 10, so lower is better. For a government index this is the failure that matters most after a miss; a contradicting page came first on at most four of the 581 requests for any engine.
• hit@10 and recall@10. The release also reports hit@10, which credits a near match such as a mirror copy, and recall@10, the share of the official page and its near matches found in the top 10. Both are left out of the charts: the Government & Legal Index covers official sources only, so mirror copies sit outside it by design and the two measures mostly report coverage of sites the index does not aim at.
• Latency. Wall-clock time per request, median and 95th percentile, with every engine timed in the same run from one client location.
• Cost per request. Each engine's list price for one search call. The Government & Legal Index is free; Parallel lists $0.005 and Exa $0.007. For the two Claude Code reference rows it is what the agent spent per request, model tokens plus search fees.

Every score carries a 95% Wilson interval over the 581 requests. Gaps between engines are tested request by request with a paired bootstrap, and we claim a lead only when that interval excludes zero.

On the run of 3 Nov 2026, the Firecrawl Government & Legal Index leads on gold@1 at 76.2% and on gold@10 at 96.6%. On bad@10, the share of requests with a contradicting page in the top 10, its 9.6% is statistically level with Exa (9.6%) and Parallel (9.3%).

Claude Code (Sonnet 5.5) ran the same requests as a reference, once with its own web search and once with no tools. It returns the one or two addresses it settles on, so its gold@10 sits close to its gold@1 and its bad@10 near zero; those two rows are not compared with the engines' top 10.

What the numbers show:

• gold@1. The index puts the official page first on 76.2% of requests, 16.6 points ahead of Exa and 22.0 ahead of Parallel. Each lead holds at 95% confidence, compared request by request. A second pass of the index 1.4 to 2.3 hours later changed gold@1 on 5 of the 581 requests and gold@10 on none, so run-to-run noise is small next to these gaps.
• gold@10. The index finds the official page in the top 10 on 96.6% of requests, 6.1 points ahead of Parallel and 7.1 ahead of Exa. Each lead holds at 95% confidence.
• bad@10. A contradicting page reached the top 10 on 9.6% of requests for the index, 9.6% for Exa and 9.3% for Parallel, which overlap within their intervals. A contradicting page came first on at most four of the 581 requests for any engine.

The full table with every interval, the interactive chart and the methodology live on the benchmark page. It joins the Developer Retrieval Benchmark on firecrawl.dev/benchmarks.

The cost axis in Figure 1 is list price, where the index is free. An agent still pays for reading results in model tokens, so we also measured the whole bill. In the agent runs, a request answered with the index cost a Claude Code agent $0.040, under Exa and Parallel once their search fees are added, and a Codex agent $0.063.

We gave the same 581 requests to two coding agents, Claude Code with Sonnet 5.5 and Codex with gpt-6.1-sol, each with one search option at a time, the same prompt, at most 20 turns and no page-fetch tool. Each setup ran once, so treat gaps of a few points as unsettled. Cost per request is the model bill plus search fees, at list prices, with Firecrawl credits priced at the Standard plan billed annually.

Model tokens are 71% to 100% of every setup's bill, so what a search puts into the agent's context matters more than the search's price. Index results carry fuller descriptions, a median of 1,289 characters per result, which is where the index's token bill comes from; the agents also searched it about twice per request, 2.04 times on Claude Code and 2.79 on Codex.

Trimming the index's descriptions is one of the next turns of the loop. Codex's built-in search fee assumes $10 per 1,000 searches, which we haven't confirmed.

All three engines were timed in the same run. The Firecrawl Government & Legal Index answers in 478 ms at the median with a 95th percentile of 632 ms. Exa is the fastest at the median, with the tightest spread, and Parallel is the slowest. Claude Code, timed from request to answer, takes 11,045 ms at the median with its own web search, with a 95th percentile of 31,831 ms, and 4,145 ms with no tools, with a 95th percentile of 8,685 ms.

The right panel of Figure 1 plots the same scores against these medians. Latency depends on where the client runs, so treat these numbers as one setup's view. A full pass over the 581 requests costs $4.07 on Exa, $2.91 on Parallel and nothing on the Government & Legal Index.

Searches use no Firecrawl credits and work without an API key; adding one raises the rate limits.

curl -s -X POST https://api.firecrawl.dev/v2/search/gov \
  -H 'Content-Type: application/json' \
  -d '{"query": "Texas notice to vacate before eviction", "k": 10}'

The index is also the firecrawl_gov_search tool in the Firecrawl MCP server, the firecrawl gov command in the CLI, and a source for the Firecrawl plugin in ChatGPT and Claude. The Government Index documentation covers the endpoint and its parameters, and the launch post covers what the index holds.

Is a model judging the published scores? No. Every request has a frozen official answer. Scoring checks by canonical URL whether that answer is in an engine's top 10, and where. The same run always gets the same score.

Why lead with gold@1? Because for a government index the official page has to come first. An agent or a person acts on the first result, and a copy or a summary in that position is a miss. The release also reports hit@10 and recall@10.

Why twins instead of real requests? A twin keeps what the measurement needs, the kind of document and the jurisdiction, and asks for a different document, so the set can be published whole without carrying any customer's search.

Can I rerun it? Yes. The metric definitions and scoring rules are published in full, and the 290 public requests, their answer key and the scoring code are on GitHub at firecrawl/benchmark-legal-retrieval. The 291 held-out requests are scored on every run and kept out of the release.

1. Anthropic. Eval awareness in Opus 4.6 on BrowseComp. 2026.
2. Wang et al. Search-Time Contamination in Deep Research Agents: Measuring Performance Inflation in Public Benchmark Evaluation. 2026.
3. Craswell et al. ORCAS: 18 Million Clicked Query-Document Pairs for Analyzing Search. 2020.
4. Carranza et al. Synthetic Query Generation for Privacy-Preserving Deep Retrieval Systems using Differentially Private Language Models. NAACL 2024.
5. Kim et al. AutoBenchmark: benchmark creation and the role of humans. Meta FAIR, 2026.
6. Voorhees and Harman. Overview of TREC 2001. NIST.
7. Shankar. Data Flywheels for LLM Applications.
8. Shukla et al. Adaptive Data Flywheel: Applying MAPE Control Loops to AI Agent Improvement. 2025.