GPT-6.2 Astra and Sol
Benchmark: Search Lift
With Firecrawl MCP in Codex, GPT-6.2 Astra and Sol both score about 97 out of 100 on Firecrawl Search Bench (FSB), a lift of 78 and 79 points over no tools. Astra vs Sol is a tie on score, and Sol costs about a third as much per question.
- Maker
- [ OpenAI · closed weights ]
- Models
- [ 2 models ]
- Question sets
- [ 1 question set ]
- Rows
- [ 429 rows ]
- Data as of
- [ Data as of ]
This chart shows how much web search helps each model answer the 429 questions in Firecrawl Search Bench (FSB). Each bar is a score out of 100. The grey part is the model answering from memory with no tools, and the orange part is what it gains when it can search the web and read pages through Firecrawl MCP. Hover a bar for the numbers.
- No tools
- With Firecrawl MCP
- Search lift with Firecrawl MCP
- Astra+77.9Sol+78.9
- Points over no tools, about the same for both
- Cost per row
- Astra$0.224Sol$0.080
- With Firecrawl MCP; Sol costs 64% less
- Time per row
- Astra16.0sSol13.2s
- Median with Firecrawl MCP; Sol is 18% faster
How the setups stack up
How do GPT-6.2 Astra and Sol compare?
Firecrawl MCP lifts Astra by 77.9 points and Sol by 78.9 over no tools. Astra and Sol tie in both setups, and Sol costs 64% less per row with Firecrawl MCP.
| Use case | Best pick | Measured result |
|---|---|---|
| Highest score | Firecrawl MCP | 96.6 on Astra and 96.7 on Sol, +77.9 and +78.9 over no tools |
| Astra or Sol | GPT-6.2 AstraGPT-6.2 Sol | A tie: Astra minus Sol is −0.1 with Firecrawl MCP, 95% range −1.5 to +1.3 |
| Cheapest with search | GPT-6.2 Sol | $0.080 per row against $0.224 on Astra, 64% less |
| Fastest with search | GPT-6.2 Sol | 13.2s per row against 16.0s on Astra, median |
Score and cost
One dot per model and setup. Up and to the left is better.
- best quadrant
- worst quadrant
By cost per row
By median time per row
Firecrawl MCP's cost per row includes its Firecrawl credits.
Full results
Points out of 100 on FSB (Firecrawl Search Bench) with 95% intervals, then cost and median time per row.
| Model | No tools | Firecrawl MCP |
|---|---|---|
| GPT-6.2 Astra429 rows · Accuracy | 18.7[15.3, 22.2]$0.018 · 3.7s | 96.6[95.0, 97.9]$0.221 · 16.0s+ about 4 credits ($0.003) |
| GPT-6.2 Sol429 rows · Accuracy | 17.8[14.5, 21.1]$0.014 · 3.7s | 96.7[95.1, 98.1]$0.075 · 13.2s+ about 6 credits ($0.005) |
- Firecrawl credits are listed separately in this table, at list price; the cost figures above include them.
How it was run
How the runs were made
One question set
FSB (Firecrawl Search Bench), our own set of search questions with known answers.
Two models, two setups each
Astra and Sol each answered every question with no tools, and in Codex with firecrawl_search and firecrawl_scrape from the Firecrawl MCP server.
OpenAI's API, medium reasoning
Both models ran on OpenAI's API at medium reasoning effort in both setups. Firecrawl MCP ran in Codex 0.169.0, OpenAI's coding agent.
One run per question
Each question ran once, on November 17 and 18, 2026, with a 25-minute timeout. None failed or timed out.
One judge
An LLM grader (gpt-5.5-2026-04-23) checks each answer against the answer key.
Final score
Metrics
What every number on this page means and how it is counted.
Score[points out of 100]
The share of each question's answer-key facts that the answer gets right, averaged over FSB's 429 questions.
Tie
Two models tie when the 95% range of their difference on the same questions includes zero.
Lift[points]
Score with Firecrawl MCP minus the score with no tools, on the same questions.
Cost per row[usd]
What one question cost: the model's tokens for that row plus the judge's, averaged over the set, plus Firecrawl credits at list price.
Time per row[seconds]
The median time from the question to the final answer.
95% range
A bootstrap interval over rows: rerunning the same setup on a resampled set of questions lands inside it 95% of the time.
Frequently asked questions
What does the GPT-6.2 Astra and Sol benchmark measure?
How much web search improves GPT-6.2 Astra's and Sol's answers on Firecrawl Search Bench (FSB), a private set of 429 questions. Each model answers every question twice, once with no tools and once with Firecrawl MCP in Codex, and the page reports accuracy out of 100, cost per question and median time.
What does Search Lift mean?
The points web search adds to a model's score. Lift is the score with Firecrawl MCP minus the score with no tools on the same FSB questions. Astra goes from 18.7 with no tools to 96.6 with Firecrawl MCP, and Sol from 17.8 to 96.7.
How much does web search add?
78 to 79 points on FSB. Firecrawl MCP adds 77.9 on Astra (95% range 73.9 to 81.7) and 78.9 on Sol (75.3 to 82.4).
Is GPT-6.2 Astra better than GPT-6.2 Sol?
Not on FSB. Astra minus Sol is −0.1 points with Firecrawl MCP (95% range −1.5 to +1.3) and +0.9 with no tools (−0.8 to +2.8): a tie both ways, and Sol costs less.
GPT-6.2 Astra vs Sol: which should I use for web search?
Sol, going by FSB. Both score about 97 and the gap sits inside the 95% range, while Sol costs $0.080 a question against $0.224 for Astra, 64% less, with a median of 13.2 seconds against 16.0.
What does a question cost, and how long does it take?
With Firecrawl MCP, $0.224 and a median of 16.0 seconds on Astra, and $0.080 and 13.2 seconds on Sol. Costs include Firecrawl credits.
How many searches does Firecrawl MCP make?
On Astra, 1.7 searches and 0.9 scrapes a question. On Sol, 2.3 and 1.2.
What is FSB?
Firecrawl Search Bench, Firecrawl's own set of 429 search questions scored by accuracy. Its questions are not published.
Could the models see the answers?
No. FSB's questions and answer key are private, and each answer comes from a public page that search is meant to find.