Web Crawling APIs Glossary from Firecrawl

Point a crawl API at a root URL and it discovers every page automatically, returning the content of each as clean markdown.

Ingesting a docs site into RAG without broken HTML requires a crawler that renders JavaScript documentation frameworks (Docusaurus, GitBook, Mintlify) and returns clean markdown per page, not raw HTML fragments or empty shells.

Web crawlers systematically browse websites by following links, downloading content, and respecting rules to build search engine indexes.

Scraping 10,000 pages to clean markdown requires a crawl API that handles URL discovery, parallel fetching, rate limiting, and boilerplate removal. Running this yourself means managing job state, retry logic, and JavaScript rendering for modern pages.

Use a crawler that follows internal links from a root URL to discover subpages automatically.

AI web crawlers are bots that collect web content at scale to train large language models or to power live retrieval in AI assistants, distinct from search engine crawlers that index pages for search results.

A seed URL is the starting point for a web crawler to begin discovering and indexing web content.

Sitemaps guide web crawlers to discover and index pages efficiently by providing a structured list of URLs with metadata about priority and updates.

A URL frontier is the queue system that manages which URLs a web crawler visits next, balancing speed with server politeness.

A web crawling API automates the process of systematically discovering and extracting content across websites while handling proxies, JavaScript, and rate limiting.

An agentic web crawler uses AI to click buttons, fill forms, and navigate dynamic content that static crawlers cannot reach.

Breadth-first crawling explores all pages at the current depth before moving deeper, while depth-first crawling follows one path to its end before backtracking.

Crawl budget is the number of pages a search engine will crawl on your website within a specific timeframe, determined by server capacity and content demand.

Crawl delay is an unofficial robots.txt directive that tells web crawlers how many seconds to wait between page requests to prevent server overload.

Crawl depth limit caps how far a crawler will venture from its starting URL, measured either by the number of link hops followed or by discovery order, depending on the crawler's implementation.

Crawl scope defines the boundaries of what a web crawler is permitted to visit through allowed domains, include and exclude path patterns, and depth limits, preventing the crawler from drifting into irrelevant content.

Deep research crawls hundreds of sources and synthesizes findings into comprehensive, cited reports.

Distributed web crawling spreads crawling tasks across multiple machines working in parallel to achieve massive scale and throughput.

Focused crawling targets a specific topic, domain set, or content type rather than following all reachable links, producing a smaller and higher-relevance dataset than general web crawling.

Incremental crawling fetches only pages that are new or have changed since the last crawl run, using stored checksums, ETags, or last-modified timestamps to skip unchanged content.

Javascript-enabled crawling uses headless browsers to execute JavaScript and access dynamically rendered content that traditional HTTP crawlers cannot see.

Link extraction is the step in the crawl loop where a crawler parses a downloaded page, collects all URLs it should visit next, and filters them before adding to the crawl queue.

Polite crawling respects website resources by limiting request rates, following robots.txt rules, and monitoring server responses to avoid overload.

Redirect handling determines how web crawlers follow HTTP redirects like 301 and 302 to reach content at different URLs.

Scrape large websites using crawl APIs with automatic URL discovery, sitemap navigation, rate limiting, and incremental processing.

Use Firecrawl's crawl endpoint to recursively extract entire documentation sites into clean markdown, preserving code blocks and structure for knowledge bases and AI applications.

Deduplicating crawl pages for RAG requires URL normalization to catch URL-variant duplicates before fetching, content hashing to catch identical pages at different URLs post-fetch, and optional near-duplicate filtering to avoid redundant chunks in the vector index.

The robots.txt protocol is a standard file that tells web crawlers which pages they can access and which to avoid on a website.

URL normalization converts different representations of the same URL into a single canonical form before adding it to the crawl queue, preventing duplicate pages from being fetched and processed multiple times.

Crawl company content with Firecrawl, chunk the markdown output, embed in a vector database, and connect to an LLM for RAG-powered conversational search.

Web crawlers and web spiders are the same thing—both terms describe automated programs that systematically browse websites by following links and extracting content for indexing or data collection.