What is the SimpleQA benchmark and why does it matter for search APIs?
SimpleQA is a factuality benchmark from OpenAI that tests whether a language model answers short, fact-seeking questions correctly. OpenAI released it on October 30, 2024 with 4,326 questions, each with one answer that doesn't change over time, and a grader that marks every response correct, incorrect, or not attempted. For a web search API, SimpleQA has become the standard quick test. Teams give an AI agent the API as its search tool, run the questions, and count the correct answers.
SimpleQA matters for search APIs because its questions were written to make models without search hallucinate (OpenAI reported GPT-4o scoring under 40%). In Firecrawl's July 2026 eval, a GPT-5.4 agent scored 43.8% with no search tools and 94.7% with Firecrawl Search and Firecrawl Scrape as its tools, graded by the same judge, so the difference comes from grounding with web search.
| Factor | SimpleQA | SimpleQA Verified | BrowseComp |
|---|---|---|---|
| Publisher | OpenAI, Oct 2024 | Google DeepMind and Google Research, Sep 2025 | OpenAI, Apr 2025 |
| Questions | 4,326 | 1,000 | 1,266 |
| What it tests | One isolated fact per question | The same task with corrected labels and balanced topics | Facts that take many pages to find |
| Grading | Correct, incorrect, not attempted | Same grades, revised grader prompt | One short answer per problem |
| Status | Near the ceiling for browsing agents | Fixes noisy labels, topic bias and duplicates | Built to stay hard for browsing agents |
Use SimpleQA to check that a search API returns the right fact for direct lookups, and read the methodology before comparing scores: the agent model, the tool-call limit, whether a page-fetch tool was allowed, the judge, and whether the best of several runs was kept. OpenAI described SimpleQA as already saturated by models with fast browsing tools in April 2025, so check the harness before reading anything into a one or two point gap between providers. For multi-step research, BrowseComp and FRAMES are better tests; see search API accuracy benchmarks.
Firecrawl Search returns the excerpts that answer the query from each result by default, using about 10x fewer tokens than full pages. The July 2026 eval behind the 94.7% score gave a GPT-5.4 agent up to 20 tool calls with Firecrawl Search plus Firecrawl Scrape for page fetches, ran the full SimpleQA set with the official grader prompt, and reported the better of two sessions.