Introducing Browser Sandbox - Give your agents a secure, fully managed browser environment Read more →
How do I crawl an entire website and get content for every page?
Point a crawl API at a starting URL and it follows every internal link automatically, returning the content of each page as clean markdown. No need to know the site structure or list URLs in advance.
Firecrawl's crawl endpoint handles discovery, JavaScript rendering, rate limiting, and robots.txt automatically:
from firecrawl import Firecrawl
from firecrawl.types import ScrapeOptions
firecrawl = Firecrawl(api_key="fc-YOUR_API_KEY")
result = firecrawl.crawl(
"https://example.com",
limit=100,
scrape_options=ScrapeOptions(formats=["markdown"], only_main_content=True)
)Use limit to cap pages, max_depth to control link depth, and include_paths / exclude_paths to scope the crawl to specific sections. For a quick one-off crawl from the terminal, the Firecrawl CLI also works: firecrawl crawl https://example.com --wait.
Last updated: Feb 23, 2026
FOOTER
The easiest way to extract
data from the web
data from the web
. .
.. ..+
.:.
.. .. .::
+.. ..: :.
.:..::. .. ..
.--:::. .. ... .:. ..
.. .:+=-::.:. . ...-.::. ..
::.... .:--+::..: ......:+....:. :.. ..
....... ::-=:::: ..:-:-...: .--..:: .........
.. . . . ..::-:-.. .-+-:::.. ...::::. .: ...::.:..
. -... ....: . . .--=+-::. :-=-:.... . .:..:: .:---:::::-::....
..::........::=..... ...:-.. .:-=--+=-:. ..--:..=::.... . .:.. ..:---::::---=:::..:...
..........::::.:::::::-::.-.. ...::--==:. ..-::-+==-:... .-::....... ..--:. ..:=+==.---=-+-:::::::-..
. .....::......:: ::::-::.---=+-:..::-+==++X=-:. ..:-::-=-== ---.. .:.--::.. .:-==::=--X==-----====--::+:::+...
..-....-:..::-::=-=-:-::--===++=-==-----== X+=-:.::-==----+==+XX+=-::.:+--==--::. .:-+X=----+X=-=------===--::-:...:. ....
....::::...:-:-==+++=++==+++XX++==++--+-+==++++=-===+=---:-==+X:XXX+=-:-=-==++=-:. .:-=+=- -=X+X+===+---==--==--:..::...+....+
..:::---.::.---=+==XXXXXXXX+XX++==++===--+===:+X+====+=--::--=+XXXXXXX+==++==+XX+=: ::::--=+++X++X+XXXX+=----==++.+=--::+::::+. ::.=...
.:::-==-------=X+++XXXXXXXXXXX++==++.==-==-:-==+X++==+=-=--=++++X++:X:X+++X+-+X X+=---=-==+=+++XXXXX+XX=+=--=X++XXX==---::-+-::::.:..-..