How do you measure search quality for an AI agent?
Search quality for an AI agent is measured end to end. Run a fixed question set with known answers through the same agent, swap only the search API, and score how often the final answer is correct. Retrieval metrics such as Recall@10 then explain why an answer failed, and cost, latency and token counts show what each correct answer took. A search tool that ranks the right page first but returns noisy text can still lower answer accuracy.
| Metric | What it measures | How to score it |
|---|---|---|
| Answer accuracy | Share of questions the agent answers correctly | Exact match or an LLM judge against reference answers |
| Recall@k | Fraction of labeled relevant sources found in the top k results | Relevant URLs retrieved divided by relevant URLs labeled |
| Freshness | Whether results reflect the current state of a fact | Questions with changing answers; check publish dates |
| Latency | Time per search call and per answered question | p50 and p95 timings |
| Tokens per answer | How much context the agent had to read | Input tokens per question |
| Cost per correct answer | Search and model spend for each right answer | Total spend divided by correct answers |
Build the question set from real agent traffic, and add a public set such as SimpleQA as a sanity check. Hold the model, prompt, tool-call limit and judge constant, run each configuration more than once because agent runs vary, and count not-attempted answers separately so a tool that pushes the agent to guess doesn't look better. Public benchmarks saturate as models improve, so your own questions decide; see search API accuracy benchmarks for what each public set covers and how fresh search API results are for freshness checks.
Firecrawl's SimpleQA eval follows this pattern: a GPT-5.4 agent with up to 20 tool calls, the official grader prompt, two sessions per provider, and a no-search baseline of 43.8% to show the lift to 94.7% with Firecrawl Search and Firecrawl Scrape as its tools. Firecrawl Search returns query-relevant highlights by default, about 10x fewer tokens than processing full pages, which keeps tokens per answer low.