Fantasy Data
Benchmark 2026
Can AI agents get fantasy data right? 272 real fantasy questions from agents on the 2026 NFL season, graded against nflverse stats and saved projections from 7 sites.
- Last updated
- [ Updated Oct 15, 2026 ]
- Systems compared
- [ 3 setups ]
- Dataset
- [ 272 questions ]
How the setups stack up
What is the best fantasy data tool for AI agents?
Firecrawl with Alexandria is the most accurate setup for fantasy data in this benchmark. It answers 80.5% of 272 questions fully right, 8.8 points ahead of native web search and 9.2 ahead of Flaim, and leads or ties on every question type. The gap is on Week 6 projections; on past-game stats the setups are close.
| Use case | Best pick | Measured result |
|---|---|---|
| Answering fantasy questions overall |  Firecrawl Alexandria | 80.5% for Firecrawl Alexandria on all 272 questions, 8.8 points ahead of native web search (p = 0.005) |
| Weekly projections and rankings | Firecrawl Alexandria | 77.9% for Firecrawl Alexandria on Week 6 projections, against 61.5% for Flaim and 60.7% for native web search |
| Injury reports and practice status | Firecrawl Alexandria | 78.6% for Firecrawl Alexandria, against 73.8% for Flaim and 71.4% for native web search (42 questions) |
| Past-game stats and usage | Firecrawl Alexandria Native web search | 84.3% for both Firecrawl Alexandria and Claude web search and web fetch; Flaim 81.5% |
| Numbers you can trust | Firecrawl Alexandria | 98.8% of checked values right for Firecrawl Alexandria, against 92.6% for Flaim and 85.0% for native web search |
| Fastest answers | Firecrawl Alexandria | 47 s median per answer for Firecrawl Alexandria, against 62 s for Flaim and 65 s for native web search |
Leaderboard
Success rate, the share of questions answered fully right, with a 95% confidence interval. Switch tracks to see how the ranking changes.
Check all metric definitionsFirecrawl AlexandriaNative web search
FlaimFirecrawl AlexandriaNative web searchFlaimFirecrawl AlexandriaFlaimNative web searchFull results
Every metric for the 3 setups. The best value in each column is highlighted.
| # | Setup | Past games (higher is better) | Live (higher is better) | Success rate (higher is better) | Accuracy (higher is better) | Median time (lower is better) |
|---|---|---|---|---|---|---|
| 01 | Firecrawl AlexandriaFirecrawl MCP (search, scrape and Alexandria data providers), plus built-in web search. | 82.7% | 77.9% | 80.5% | 98.8% | 46.8s |
| 02 | Native web searchBuilt-in web search and fetch only. | 80.7% | 60.7% | 71.7% | 85.0% | 65.0s |
| 03 | FlaimFlaim MCP (fantasy league tools), plus built-in web search. | 79.3% | 61.5% | 71.3% | 92.6% | 61.9s |
How is fantasy data measured?
Real questions
Fantasy searches made by Firecrawl users' agents, with live questions set to Week 6.
Answer keys saved first
Past games from nflverse, checked against Sleeper and ESPN. Live projections saved directly from 7 sites before and after the answers.
Same agent, different tools
One agent answers every question with the same prompt; only the tools differ.
Graded by code
A model copies values out of each answer; a frozen program decides right or wrong.
Final benchmark
The grader, the answer keys and a public half of the questions with every answer and grade are open source. Browse the code and data at firecrawl/benchmark-fantasy.
Dataset
272 questions on the full set. A 137-question public sample, about half of each question type, ships for reproduction.
- Past games
150 questions on Weeks 1 to 5 of the 2026 season: player stats, usage, injury designations and scores. Answer key built from nflverse and cross-checked against Sleeper (416 of 417 values agree) and ESPN (79 of 79 scores).
Example question“Chris Bell Dolphins Week 5 2026 fantasy usage targets”- Live projections
122 questions on Week 6 projections and rankings. Answer key: pages from CBS, Draft Sharks, StartWho, FantasyPros, Fantasy Football Calculator, FantasyData and FFToday, saved directly right before and right after the answers were collected.
Example question“FantasyData weekly fantasy football rankings Week 6 2026 half PPR”
Metrics
How questions are scored, how every setup is driven, and how answers are checked.
Success rate and accuracy
A past-game question is right when at least 90% of its checked values match the answer key. A live question naming players is right when no checked player is wrong or missing; a live table needs at least 3 checked values, 90% of them right. A live value counts if it matches the saved page within 0.3 points, in the scoring format the question asks for.
Success rate is the share of questions answered fully right. Accuracy is the share of checked values that are correct, a player's projection or rank, a table value or a past-game stat, counted across all answers.
How setups are driven
One agent (claude-opus-5-5) answers every question in a fresh session with the same prompt; only the tools differ. Each setup gets one try per question.
Scoring
A model copies the values out of each answer and never judges them; a frozen program compares them with the answer key, so the same answers always get the same score. Claims that can't be checked against the saved sources are left out rather than counted wrong, and only questions every setup's answer could be checked on are scored.
Frequently asked questions
Changelog
Dated snapshots of this benchmark and what changed in each run.
- Oct 16, 2026First published run. 3 setups on 272 questions (150 past games, 122 Week 6 projections), one model, one try per question.
Each snapshot is published from a pinned run in firecrawl/benchmark-fantasy.