Firecrawl Map: The Cheat Code for Web Scraping

Mapping a site first means you know exactly which URLs exist before spending crawl credits. Firecrawl's /map endpoint takes a single URL and returns every URL on that site, extremely fast, in one API call. It's the quickest way to see a site's structure, or to find exactly which pages you want before you scrape them.

map needs an API key, it isn't one of the keyless endpoints. Sign up for a free account and grab a key (starts with fc-) from the dashboard, you'll get 1,000 free credits and a solid rate limit to start with.

This walkthrough uses the Node SDK, but Python and the CLI work the same way.

import { Firecrawl } from "firecrawl";
 
const firecrawl = new Firecrawl({ apiKey: "fc-YOUR-API-KEY" });
 
const res = await firecrawl.map("https://docs.firecrawl.dev");
 
console.log(res.links);

firecrawl.map discovers links from three sources merged together: the site's sitemap, search engine results, and pages Firecrawl has crawled before, for the widest possible coverage. res.links comes back as an array, each entry with a url, title, and description pulled from the page's metadata, though title and description aren't always present, it depends on what the site exposes.

All of this happens in a fraction of the time a full /crawl would take. Because map only discovers links instead of scraping full pages, each call costs a single credit, no matter how many URLs come back, even at a limit as high as 100,000.

By default map returns up to 5,000 URLs, which is a lot to sift through. The search parameter narrows that down to just the URLs matching a query, ordered most relevant first based on keyword frequency:

const res = await firecrawl.map("https://docs.firecrawl.dev", {
  search: "scrape",
});
 
console.log(res.links);

Use limit to cap how many links come back, useful alongside search or on its own:

const res = await firecrawl.map("https://firecrawl.dev", {
  limit: 100,
});

Map covers subdomains by default. Turn that off with includeSubdomains: false if you only want the main site:

const res = await firecrawl.map("https://firecrawl.dev", {
  includeSubdomains: false,
});

The sitemap parameter controls how the site's sitemap gets used. include is the default, blending sitemap URLs with search and crawl history. skip ignores the sitemap entirely, and only restricts results to just what's in the sitemap:

const res = await firecrawl.map("https://firecrawl.dev", {
  sitemap: "only",
});

Set location to get results tuned to a specific country and language, the same idea as the scrape endpoint's location option:

const res = await firecrawl.map("https://example.com", {
  location: { country: "US", languages: ["en"] },
});

country defaults to US if you don't set it.

Map trades thoroughness for speed. It's built on the sitemap plus supplementary sources, so it may not capture every link on a site. If you need a more thorough, up-to-date list of every URL, reach for /crawl instead, map is for quickly seeing what's there before deciding what to scrape.

Check out the Firecrawl docs to go deeper.