Question 1

What is Firecrawl Crawl?

Accepted Answer

Firecrawl Crawl turns a domain into clean markdown your agent can read. Pass a URL and Crawl walks the subpages, scrapes each one in a real browser, and returns the same clean markdown for every page. One job, one webhook, one consistent output format across thousands of pages.

Question 2

Who uses Crawl?

Accepted Answer

Teams building knowledge bases, RAG pipelines, AI agents, content migrations, and onboarding flows that turn a customer's URL into something the product can use. Over 1.25M developers and 150,000+ companies build with Firecrawl, including teams at Zapier, Botpress, Vercel, and Coinbase, and we've served more than 5 billion requests to date.

Question 3

How is Crawl different from rolling my own crawler?

Accepted Answer

An in-house crawler is a few weeks of work and a long tail of maintenance: sitemap parsing, link discovery, JS rendering, rate limits, retries, proxies, robots.txt, and the HTML-to-markdown layer that LLMs actually need. Crawl ships all of that as one endpoint. The output is the same clean markdown across every page, ready to drop into an LLM. Customers consolidating from Zyte, Bright Data, and in-house tooling have done the math.

Question 4

When should I use Crawl vs. Scrape or Map?

Accepted Answer

Use Scrape when the agent already has the URL. Use Map when you want to know what URLs exist on a site before paying to scrape them. Use Crawl when the input is a domain and the output is every page on the site — knowledge bases, full-site classification, content migrations. Search pairs well with Crawl when the agent has to discover the URL first.

Question 5

Does Crawl work on JavaScript-heavy sites?

Accepted Answer

Yes. Every page in the crawl renders in a real Chromium instance, so React, Vue, Next.js, and other SPA frameworks come back the same way they would in a browser tab. You get the full DOM, including content fetched after page load.

Question 6

How does Crawl discover pages?

Accepted Answer

By default, Crawl reads the site's sitemap and follows links recursively from the root URL. You can switch sitemap handling to skip or only, follow links to subdomains or external sites, and ignore query parameters so the same path isn't re-scraped with different query strings.

Question 7

Can I crawl only part of a site?

Accepted Answer

Yes. Pass includePaths or excludePaths as regex patterns to control which paths are crawled, set maxDiscoveryDepth to cap how deep the crawler follows links, and use crawlEntireDomain or allowSubdomains when you want sibling paths or subdomains included.

Question 8

Can I process pages as they're crawled?

Accepted Answer

Yes. Subscribe to webhook events (crawl.started, crawl.page, crawl.completed, crawl.failed) or a WebSocket to process pages the moment they're ready instead of waiting for the whole job. The SDKs also include a poll-and-wait method for simpler workflows.

Question 9

What formats can I get back per page?

Accepted Answer

Markdown by default. Pass scrapeOptions to request structured JSON via a schema, raw or clean HTML, full-page screenshots, links, and metadata for every page in the crawl. Every option that works on Scrape works on Crawl.

Question 10

Is Crawl built for production?

Accepted Answer

Yes. Production customers run Crawl at hundreds of concurrent browsers and millions of URLs per day. Jobs queue gracefully if you hit your limit, and Scale and Enterprise plans support 150 to 600+ browsers with dedicated rate limits and SLAs.

Question 11

How much does Crawl cost?

Accepted Answer

1 credit per page crawled. JSON mode adds 4 credits per page, enhanced proxy adds 4 credits per page, and PDF parsing costs 1 credit per PDF page. The default crawl limit is 10,000 pages — set a lower limit to match your intended crawl size.

Question 12

Is there a free tier?

Accepted Answer

Yes. Free accounts include 1,000 credits per month, enough to crawl small docs sites and try every scrapeOption. No credit card required.

Question 13

Can I self-host Crawl?

Accepted Answer

Yes. Firecrawl is open source with over 130K GitHub stars and 2.5M+ weekly downloads on npm and PyPI. Run Crawl on your own infrastructure for full control, or use the hosted API for zero ops. Both share the same SDKs and CLI.

Web Crawling API - Part of the all-in-one web toolkit

One URL.
Every page.

Crawling that finishes
the whole job.

From site to knowledge base.

Clean output, every page.

Built for the long crawl.

JavaScript on every page.

Stream pages as they're ready.

Scope controls when you want them.

Built into the AI tools
you already use

Frequently asked questions