DeepSearchQA
- Questions
- [ 900 questions ]
- Metric
- [ F1 ]
Multi-step research tasks in 17 fields; about two thirds of the answers are lists.
DeepSearchQA paper
Claude Fable 5.2
Search Lift benchmarks
Web search lifts Claude Fable 5.2 by 37 points. In Claude Code, Firecrawl MCP beats the builtin WebSearch on DeepSearchQA and FSB, ties on HLE, and costs about half as much per question.
- Maker
- [ Anthropic · closed weights ]
- Question sets
- [ 3 question sets ]
- Rows
- [ 3,437 rows ]
- Data as of
- [ Data as of ]
Score with Firecrawl MCP84.7
+37.2 points over closed book050100
- No tools
- With Firecrawl MCP
- Lift over no tools
- +37.2
- points over 47.5 closed book
- Cost per row
- −48%
- $0.28 vs $0.53 with builtin tools
- Time per row
- −26%
- 21s vs 29s with builtin tools
[ 01 / 03 ]
·
Results
//
Results
//
How the setups stack up
3 setups on the same 3,437 questions.
Which setup is best for Claude Fable 5.2?
Firecrawl MCP. It scores 84.7 out of 100, +1.7 points over builtin tools and +37.2 over closed book, at 52% of the cost per row.
| Use case | Best pick | Measured result |
|---|---|---|
| Highest score | Firecrawl MCP | 84.7 across the 3 sets, +1.7 over builtin tools and +37.2 over closed book |
| Cheapest with search | Firecrawl MCP | $0.28 per row against $0.53 with builtin tools, 48% less |
| Fastest with search | Firecrawl MCP | 21s per row against 29s, mean of the per-set medians, 26% less |
| Reasoning questions (HLE) | No toolsFirecrawl MCP | 65.8 each; search adds nothing on HLE |
Score and cost
One dot per setup. Up and to the left is better.
900 rows · F1 · points out of 100, 95% range
- best quadrant
- worst quadrant
By cost per row
By median time per row
Full results
Points out of 100 with 95% intervals, then cost and median time per row. FSB is Firecrawl Search Bench, our own question set.
| Question set | Builtin tools | No tools | Firecrawl MCP |
|---|---|---|---|
| DeepSearchQA900 rows · F1 | 89.6[87.7, 91.4]$1.13 · 46s | 58.6[55.6, 61.6]$0.11 · 18s | 91.6[89.8, 93.1]$0.52 · 35s |
| FSB429 rows · Accuracy | 93.9[92.1, 95.7]$0.25 · 21s | 18.1[14.6, 21.6]$0.02 · 7s | 96.6[95.2, 98.0]$0.15 · 12s |
| HLE2,108 rows · Accuracy | 65.3[63.2, 67.3]$0.21 · 20s | 65.8[63.7, 67.7]$0.19 · 16s | 65.8[63.9, 67.9]$0.16 · 17s |
[ 02 / 03 ]
·
Methodology
//
Methodology
//
How it was run
The question sets
Two public benchmarks and one of our own, run as published.
FSB
Firecrawl
- Questions
- [ 429 questions ]
- Metric
- [ Accuracy ]
Private questions, mostly about pages published just before the set was built. The questions are not published.
HLE
Center for AI Safety and Scale AI
- Questions
- [ 2,108 questions ]
- Metric
- [ Accuracy ]
Expert exam questions in mathematics, science and the humanities, text only. Not comparable to published full-HLE scores.
HLE websiteHow the runs were made
01
Three question sets
Hard research questions with a known answer: DeepSearchQA v4, FSB (Firecrawl Search Bench, our own set) and HLE text v4.
[ DeepSearchQA 900 ][ FSB 429 ][ HLE 2,108 ]
02
Three setups, same questions
Claude Code with its own WebSearch (builtin tools), Claude Code with firecrawl_search and firecrawl_scrape over the Firecrawl MCP server, and the model with no tools on the evals harness.
[ Builtin tools ][ No tools ][ Firecrawl MCP ]
03
One model behind every run
Claude Fable 5.2 through OpenRouter, prompt caching on. Claude Code 2.1.321, 100 turns and a 1,500s timeout per row. When Fable's safety filter blocked a reply, Claude Code reran that turn on Opus 5; those rows count as wrong for Fable.
04
Answer-key domains blocked
The no-tools runs block every domain that hosts a question set. Claude Code can't block domains, so its exposed rows were counted instead; dropping them moves no score by more than 0.3 points.
05
One judge, checked by hand
gpt-5.5 reads each answer against the gold answer. We checked 66 random rows by hand and every verdict matched.
[ gpt-5.5 reader ][ 66 rows checked ]
06
Final score
3,437questions scored. Accuracy on FSB and HLE, F1 on DeepSearchQA, each as points out of 100, with 95% bootstrap intervals over rows.
[ Accuracy ][ F1 ][ 95% CI ]
Metrics
What every number on this page means and how it is counted.
Score[points out of 100]
Accuracy on FSB and HLE, F1 on DeepSearchQA. The overall score is the plain mean of the three sets, equal weight per set.
Lift[points]
Score with a setup minus the score with no tools, on the same questions.
Cost per row[usd]
What one question cost: the model's tokens for that row plus the judge's, averaged over the set.
Time per row[seconds]
The median time from the question to the final answer, per set. Overall figures are the mean of the three medians.
95% range
A bootstrap interval over rows: rerunning the same setup on a resampled set of questions lands inside it 95% of the time.
[ 03 / 03 ]
·
FAQ
//
FAQ
//
Frequently asked questions
Common questions about Claude Fable 5.2 on the Search Lift benchmarks and how to read these results.
What do the Search Lift benchmarks measure?
How much better an AI model answers hard research questions with web search than closed book. Every setup runs the same questions from DeepSearchQA, FSB and HLE, and the score is the mean of the three sets in points out of 100.
What is the difference between Firecrawl MCP and builtin tools?
Both run in Claude Code. Builtin tools is Claude Code's own WebSearch, which returns a written summary of each search. Firecrawl MCP gives Claude Code firecrawl_search and firecrawl_scrape, so it reads the pages themselves.
What is FSB?
Firecrawl Search Bench, Firecrawl's own set of 429 search questions scored by accuracy. Its questions are not published.
Why does HLE barely change with search?
Fable made no tool call on 68% of builtin rows and 59% of Firecrawl MCP rows, so on HLE the score mostly reflects reasoning, not search.
Did Fable see any answers?
On a few DeepSearchQA and HLE rows a search result pointed at a published answer file, because Claude Code cannot block domains. Dropping every exposed row moves no score by more than 0.3 points.
How are cost and time per row counted?
Cost per row is the model's spend on that row plus the judge's, in USD. Time per row is the median from the question to the final answer.
//
Get started
//
Ready to build?
Start getting web context for free and scale seamlessly as your project expands. No credit card needed.
Are you an AI agent? See setup options