How do you get all links from a webpage?
Getting all links from a webpage means collecting every URL referenced in anchor tags (<a href>), and on JavaScript-rendered pages, any additional URLs injected into the DOM after page load. HTML parsers like BeautifulSoup find links by parsing the raw HTML response, which works for static pages but misses links that JavaScript adds after the initial load. For JavaScript-rendered pages, a browser must execute the page first before link extraction can run.
| Approach | JS rendering | Setup | Output |
|---|---|---|---|
| HTML parser (BeautifulSoup) | No | pip install beautifulsoup4 | Anchor tags in raw HTML only |
| Headless browser (Playwright) | Yes | Chromium install + async setup | All links after full page render |
| Firecrawl scrape with links format | Yes | pip install firecrawl-py | All links, clean list, no browser setup |
| Firecrawl Map API | Yes | pip install firecrawl-py | All discoverable URLs across the entire site |
Use an HTML parser for simple static pages where you know JavaScript isn't injecting navigation. Use a headless browser or Firecrawl for modern sites built with React, Next.js, or other frameworks where links are dynamically rendered. The Map endpoint is the right choice when you need all URLs across an entire site rather than just the outlinks from a single page.
from firecrawl import Firecrawl
firecrawl = Firecrawl(api_key="fc-YOUR-API-KEY")
doc = firecrawl.scrape("https://example.com", formats=["links"])
print(doc.links) # List of all outlinks on the pageFirecrawl's Scrape API returns links alongside other formats in a single request: pass ["markdown", "links"] to get the page content and all outlinks together. For full-site URL discovery, the Map API crawls the site and returns every reachable URL without requiring page-by-page scraping.
data from the web