OpenAI logo

GPT-6.2 Astra and Sol
Benchmark: Search Lift

With Firecrawl MCP in Codex, GPT-6.2 Astra and Sol both score about 97 out of 100 on Firecrawl Search Bench (FSB), a lift of 78 and 79 points over no tools. Astra vs Sol is a tie on score, and Sol costs about a third as much per question.

Maker
[ OpenAI · closed weights ]
Models
[ 2 models ]
Question sets
[ 1 question set ]
Rows
[ 429 rows ]
Data as of
[ Data as of ]

This chart shows how much web search helps each model answer the 429 questions in Firecrawl Search Bench (FSB). Each bar is a score out of 100. The grey part is the model answering from memory with no tools, and the orange part is what it gains when it can search the web and read pages through Firecrawl MCP. Hover a bar for the numbers.

96.6GPT-6.2 Astra+77.9 points over closed book
96.7GPT-6.2 Sol+78.9 points over closed book
050100
  • No tools
  • With Firecrawl MCP
Points out of 100 on FSB
Search lift with Firecrawl MCP
Astra+77.9Sol+78.9
Points over no tools, about the same for both
Cost per row
Astra$0.224Sol$0.080
With Firecrawl MCP; Sol costs 64% less
Time per row
Astra16.0sSol13.2s
Median with Firecrawl MCP; Sol is 18% faster
[ 01 / 03 ]
·
Results
//
Results
//

How the setups stack up

2 models in 2 setups each on the same 429 questions.

How do GPT-6.2 Astra and Sol compare?

Firecrawl MCP lifts Astra by 77.9 points and Sol by 78.9 over no tools. Astra and Sol tie in both setups, and Sol costs 64% less per row with Firecrawl MCP.

How do GPT-6.2 Astra and Sol compare
Use caseBest pickMeasured result
Highest scoreFirecrawl MCP96.6 on Astra and 96.7 on Sol, +77.9 and +78.9 over no tools
Astra or SolGPT-6.2 AstraGPT-6.2 SolA tie: Astra minus Sol is −0.1 with Firecrawl MCP, 95% range −1.5 to +1.3
Cheapest with searchGPT-6.2 Sol$0.080 per row against $0.224 on Astra, 64% less
Fastest with searchGPT-6.2 Sol13.2s per row against 16.0s on Astra, median

Score and cost

One dot per model and setup. Up and to the left is better.

Astra and Sol on FSB429 rows · Accuracy · points out of 100, 95% range
  • best quadrant
  • worst quadrant

By cost per row

GPT-6.2 Astra and GPT-6.2 Sol: score against cost per row for 4 runs020406080100$0$0.1$0.2$0.3Cost per row, usdAstra · No toolsAstra · Firecrawl MCPSol · No toolsSol · Firecrawl MCPAstra · No tools: 18.7, $0.018Astra · Firecrawl MCP: 96.6, $0.224, Firecrawl credits includedSol · No tools: 17.8, $0.014Sol · Firecrawl MCP: 96.7, $0.080, Firecrawl credits included

By median time per row

GPT-6.2 Astra and GPT-6.2 Sol: score against median time per row for 4 runs02040608010005101520Median time per row, secondsAstra · No toolsAstra · Firecrawl MCPSol · No toolsSol · Firecrawl MCPAstra · No tools: 18.7, 3.7sAstra · Firecrawl MCP: 96.6, 16.0sSol · No tools: 17.8, 3.7sSol · Firecrawl MCP: 96.7, 13.2s

Firecrawl MCP's cost per row includes its Firecrawl credits.

Full results

Points out of 100 on FSB (Firecrawl Search Bench) with 95% intervals, then cost and median time per row.

ModelNo toolsFirecrawl MCP
GPT-6.2 Astra429 rows · Accuracy18.7[15.3, 22.2]$0.018 · 3.7s96.6[95.0, 97.9]$0.221 · 16.0s+ about 4 credits ($0.003)
GPT-6.2 Sol429 rows · Accuracy17.8[14.5, 21.1]$0.014 · 3.7s96.7[95.1, 98.1]$0.075 · 13.2s+ about 6 credits ($0.005)
  • Firecrawl credits are listed separately in this table, at list price; the cost figures above include them.
[ 02 / 03 ]
·
Methodology
//
Methodology
//

How it was run

How the runs were made

01

One question set

FSB (Firecrawl Search Bench), our own set of search questions with known answers.

[ FSB 429 ]
02

Two models, two setups each

Astra and Sol each answered every question with no tools, and in Codex with firecrawl_search and firecrawl_scrape from the Firecrawl MCP server.

[ No tools ][ Firecrawl MCP ]
03

OpenAI's API, medium reasoning

Both models ran on OpenAI's API at medium reasoning effort in both setups. Firecrawl MCP ran in Codex 0.169.0, OpenAI's coding agent.

[ Codex 0.169.0 ][ Medium effort ]
04

One run per question

Each question ran once, on November 17 and 18, 2026, with a 25-minute timeout. None failed or timed out.

[ 25-minute timeout ]
05

One judge

An LLM grader (gpt-5.5-2026-04-23) checks each answer against the answer key.

[ gpt-5.5 grader ]
06

Final score

429questions scored. Accuracy as points out of 100, with 95% bootstrap intervals over rows.
[ Accuracy ][ 95% CI ]

Metrics

What every number on this page means and how it is counted.

Score[points out of 100]

The share of each question's answer-key facts that the answer gets right, averaged over FSB's 429 questions.

Tie

Two models tie when the 95% range of their difference on the same questions includes zero.

Lift[points]

Score with Firecrawl MCP minus the score with no tools, on the same questions.

Cost per row[usd]

What one question cost: the model's tokens for that row plus the judge's, averaged over the set, plus Firecrawl credits at list price.

Time per row[seconds]

The median time from the question to the final answer.

95% range

A bootstrap interval over rows: rerunning the same setup on a resampled set of questions lands inside it 95% of the time.

[ 03 / 03 ]
·
FAQ
//
FAQ
//

Frequently asked questions

Common questions about GPT-6.2 Astra and Sol on the Search Lift benchmark and how to read these results.

What does the GPT-6.2 Astra and Sol benchmark measure?

How much web search improves GPT-6.2 Astra's and Sol's answers on Firecrawl Search Bench (FSB), a private set of 429 questions. Each model answers every question twice, once with no tools and once with Firecrawl MCP in Codex, and the page reports accuracy out of 100, cost per question and median time.

What does Search Lift mean?

The points web search adds to a model's score. Lift is the score with Firecrawl MCP minus the score with no tools on the same FSB questions. Astra goes from 18.7 with no tools to 96.6 with Firecrawl MCP, and Sol from 17.8 to 96.7.

How much does web search add?

78 to 79 points on FSB. Firecrawl MCP adds 77.9 on Astra (95% range 73.9 to 81.7) and 78.9 on Sol (75.3 to 82.4).

Is GPT-6.2 Astra better than GPT-6.2 Sol?

Not on FSB. Astra minus Sol is −0.1 points with Firecrawl MCP (95% range −1.5 to +1.3) and +0.9 with no tools (−0.8 to +2.8): a tie both ways, and Sol costs less.

GPT-6.2 Astra vs Sol: which should I use for web search?

Sol, going by FSB. Both score about 97 and the gap sits inside the 95% range, while Sol costs $0.080 a question against $0.224 for Astra, 64% less, with a median of 13.2 seconds against 16.0.

What does a question cost, and how long does it take?

With Firecrawl MCP, $0.224 and a median of 16.0 seconds on Astra, and $0.080 and 13.2 seconds on Sol. Costs include Firecrawl credits.

How many searches does Firecrawl MCP make?

On Astra, 1.7 searches and 0.9 scrapes a question. On Sol, 2.3 and 1.2.

What is FSB?

Firecrawl Search Bench, Firecrawl's own set of 429 search questions scored by accuracy. Its questions are not published.

Could the models see the answers?

No. FSB's questions and answer key are private, and each answer comes from a public page that search is meant to find.

//
Get started
//
Ready to build?
Start getting web context for free and scale seamlessly as your project expands. No credit card needed.
Are you an AI agent? See setup options