Web Crawling API - Part of the Firecrawl web data API

Firecrawl Crawl turns a domain into clean markdown your agent can read. Pass a URL and Crawl walks the subpages, scrapes each one in a real browser, and returns the same clean markdown for every page. One job, one webhook, one consistent output format across thousands of pages.

Teams building knowledge bases, RAG pipelines, AI agents, content migrations, and onboarding flows that turn a customer's URL into something the product can use. Over 1.25M developers and 150,000+ companies build with Firecrawl, including teams at Zapier, Botpress, Vercel, and Coinbase, and we've served more than 5 billion requests to date.

An in-house crawler is a few weeks of work and a long tail of maintenance: sitemap parsing, link discovery, JS rendering, rate limits, retries, robots.txt, and the HTML-to-markdown layer that LLMs actually need. Crawl ships all of that as one endpoint. The output is the same clean markdown across every page, ready to drop into an LLM.

Use Scrape when the agent already has the URL. Use Map when you want to know what URLs exist on a site before paying to scrape them. Use Crawl when the input is a domain and the output is every page on the site, for knowledge bases, full-site classification, or content migrations. Search pairs well with Crawl when the agent has to discover the URL first.

Install the Firecrawl plugin in ChatGPT and sign in to your Firecrawl account. Then ask ChatGPT to crawl the site, or one section of it, with a page limit and to summarize each page it gets back. Crawl gives ChatGPT the Markdown of every page, so each summary comes from the full page text. Each crawled page costs 1 Firecrawl credit.

Yes. Every page in the crawl renders in a real Chromium instance, so React, Vue, Next.js, and other SPA frameworks come back the same way they would in a browser tab. You get the full DOM, including content fetched after page load.

By default, Crawl reads the site's sitemap and follows links recursively from the root URL. You can switch sitemap handling to skip or only, follow links to subdomains or external sites, and ignore query parameters so the same path isn't re-scraped with different query strings.

Yes. Pass includePaths or excludePaths as regex patterns to control which paths are crawled, set maxDiscoveryDepth to cap how deep the crawler follows links, and use crawlEntireDomain or allowSubdomains when you want sibling paths or subdomains included.

Set up Firecrawl Monitor with crawl targets and a weekly schedule. One monitor holds up to 50 targets. Every check labels each discovered page as new, changed, removed, or same, and reports by webhook, email, or Slack. Spread thousands of sites across several monitors. A check costs 1 Firecrawl credit per discovered page.

Yes. Crawl follows each site's robots.txt rules by default. URLs that robots.txt blocks appear in the robotsBlocked list of the crawl errors endpoint, next to pages that failed for other reasons, such as timeouts.

The API keeps a crawl's results for 24 hours after the job finishes. After that, the job and its results are still in the activity logs of your Firecrawl dashboard. Responses over 10MB arrive in pages, and the SDKs collect every page for you.

Yes. Subscribe to webhook events (crawl.started, crawl.page, crawl.completed, crawl.failed) or a WebSocket to process pages the moment they're ready instead of waiting for the whole job. The SDKs also include a poll-and-wait method for simpler workflows.

Markdown by default. Pass scrapeOptions to request structured JSON via a schema, raw or clean HTML, full-page screenshots, links, and metadata for every page in the crawl. Every option that works on Scrape works on Crawl.

Yes. Production customers run Crawl at hundreds of concurrent browsers and millions of URLs per day. Jobs above your plan's concurrent browser limit wait in a queue. The Scale plan includes 100 concurrent browsers, and Enterprise plans set custom limits and SLAs.

1 Firecrawl credit per page crawled. JSON mode adds 4 Firecrawl credits per page, and PDF parsing costs 1 Firecrawl credit per PDF page. The default crawl limit is 10,000 pages, so set a lower limit to match the crawl you want.

Yes. Free accounts include 1,000 Firecrawl credits per month, enough to crawl small docs sites and try every scrapeOption. No credit card required.

Yes. Firecrawl is open source with over 200K GitHub stars and 2.5M+ weekly downloads on npm and PyPI, and the self-hosted API shares the same SDKs and CLI. The open-source stack covers scrape, crawl, map, and search. It does not include Fire-engine, the managed browser and request layer, so you run and tune your own fetching setup, and screenshots, page actions, Agent, Browser, and Interact are hosted-only. Self-hosting fits simple sites when you already run infrastructure; the hosted API fits complex sites and zero ops. See docs.firecrawl.dev/contributing/open-source-or-cloud for the full comparison.