What is the best way to crawl documentation sites at scale?

TL;DR

Firecrawl's crawl endpoint recursively extracts entire documentation sites in one call. It handles JavaScript-rendered doc frameworks, preserves code blocks and formatting, and returns clean markdown ready for knowledge bases, RAG systems, or developer tools.

Recursive crawling

Point Firecrawl at a docs root and it discovers all pages automatically:

result = app.crawl("https://docs.example.com", {
    "limit": 500,
    "maxDiscoveryDepth": 10,
    "scrapeOptions": {
        "formats": ["markdown"],
        "onlyMainContent": True,
    },
})

Why Firecrawl for docs

Challenge	How Firecrawl handles it
JavaScript frameworks	Automatic rendering (Docusaurus, GitBook, etc.)
Code blocks	Preserved with language tags
Deep nesting	Configurable depth, follows all nav links
Version filtering	Path include/exclude patterns

Filtering by path

Control scope with path filters:

result = app.crawl("https://docs.example.com", {
    "includePaths": ["/api/*", "/guides/*"],
    "excludePaths": ["/changelog/*", "/blog/*"],
})

Key Takeaways

Firecrawl handles documentation crawling at scale—recursive discovery, JavaScript rendering, and code block preservation in one API. Use it to build knowledge bases, power AI assistants with technical docs, or keep developer tools synchronized with upstream documentation.