Web data for
model training
Build pre-training corpora, give RL agents a search tool, and ground evals in cited sources.
Crawl, search, parse and refresh web data on one API.
companies of all sizes
Perfect for
Pre-training teams
Crawl approved domains, docs portals and PDFs into clean Markdown corpora, with the source URL kept on every document so you can audit what went in.
Post-training and RL teams
Give agents a web search tool during rollouts. Firecrawl Search placed first of 14 configurations on tokens per task search-only, and third with fetch, in an independent benchmark.
Eval and benchmark teams
Build eval sets from real docs, sites and papers with a URL or paper ID on every item, and limit search to a date range so pages published after your cutoff stay out.
Fine-tuning and SFT builders
Extract headings, sections and fields into JSON with Scrape, then turn them into instruction pairs and Q&A sets in code.
Scientific model teams
Pull query-ranked passages from 3M+ arXiv papers and 41M+ life sciences papers in the Firecrawl Research Index, with paper IDs attached.
Data governance
Scope every job by domain and path, keep a URL and timestamp on every document, and run on SOC 2 Type II infrastructure with Zero Data Retention on enterprise plans.
How it works
Crawl approved sources into a pre-training corpus
Point Crawl at target domains and docs portals. Firecrawl walks every page, renders JavaScript and returns clean Markdown or JSON with the source URL on each document.
Parse PDFs, papers and long reports
Send arXiv PDFs, filings, whitepapers and manuals to Parse. Firecrawl returns layout-aware Markdown, so document-heavy corpora load the same way as web pages.
Put web search inside RL rollouts
Call Firecrawl Search as the agent's tool during post-training. Search-only results keep each rollout's context short, and scrapeOptions adds full-page Markdown when the policy needs the whole page.
Ground evals in cited papers and pages
Query the Firecrawl Research Index for arXiv and life sciences passages with paper IDs, and pass a custom date range to Search so eval answers come from pages published before your cutoff.
Extract structure for SFT and preference data
Request JSON from Scrape with a schema to pull headings, sections and fields, then build instruction pairs, Q&A sets and preference prompts in code.
Refresh datasets when sources change
Point Monitor at the pages behind a training or eval set. An AI judge fires a webhook only when a change is meaningful, so you refresh from real signal instead of blind reruns.
Fewer LLM tokens per retrieval task,
ranked by an independent benchmark.
Search only
Firecrawl Search7,456
Parallel (fast)12,460Parallel (turbo)14,130
Tavily16,299
Exa (fast)22,344Exa (instant)22,423Independent benchmark by Openbenchmarks, published as most token-efficient web search API for coding agents. 100 hard documentation retrieval tasks run through each search API with the same coding agent, model and prompts. The metric is the median number of LLM tokens the agent spent per task, and a pass requires a grounded source URL from that run. The tasks are coding-agent lookups, so read the figures as a guide to how much context search adds to an RL rollout. Runner and scoring are public at openbenchmarks-labs/web-search-for-coding-agents.
· Figures from the run of Sep 15, 2026
People love
building with Firecrawl
Firecrawl is an open-source framework that takes a URL, crawls it, and conver..."
Upload a CSV of emails and..."
Firecrawl is an open-source framework that takes a URL, crawls it, and conver..."
Upload a CSV of emails and..."
How Firecrawl compares to alternatives
| Feature | Firecrawl | Common Crawl dumps | In-house scrapers | SERP APIs |
|---|---|---|---|---|
| Web search tool for agents and RL rollouts | Yes | No | No | Yes |
| Crawl chosen domains on demand | Yes | No | Yes | No |
| Full-page content as clean Markdown | Yes | No | No | No |
| PDF and document parsing built in | Yes | No | No | No |
| Cited academic passages (Research Index) | Yes | No | No | No |
| JavaScript rendering | Yes | No | Yes | No |
| Change monitoring for dataset refresh | Yes | No | No | No |
| Source URL kept on every document | Yes | Yes | Yes | Yes |
| No proxies or browser fleet to maintain | Yes | Yes | No | Yes |
Tutorials & guides
Web Search and Deep Research for AI Agents: What It Is and How to Integrate It into Your Agentic Stack
Learn what web search and deep research mean for AI agents, why 2026 made both production-grade, and how to wire them into your stack with Firecrawl.
Read tutorial →
Reduce LLM & Agent Hallucinations With Real-Time Web Search
LLM agents hallucinate from stale data and knowledge gaps. Why hallucinations persist despite smarter models, and how real-time web search reduces them.
Read tutorial →
LLM Token Optimization: 9 Techniques in the Order to Apply Them
How a 10,000-task agent's LLM bill drops from $11,648 to $425 a month: leaner web context, prompt caching, output caps, and batching, with current pricing.
Read tutorial →Frequently
asked questions
Flexible pricing
Free Plan
Hobby
StandardMost popular
Growth
Scale Plans
High-volume plans for teams that need more power and dedicated support. Get access to higher rate limits, more concurrent browsers, and priority support. Scale checks out instantly, no sales call needed.
Need more? Contact usAlso on AWS Marketplace and Google Cloud Marketplace.