How to build an AI training dataset from the web, step by step

TL;DR

• An AI training dataset is a set of examples in one fixed shape that a model learns from, such as question and answer pairs, labeled text or plain documents. OpenAI accepts a fine-tuning file with as few as 10 examples and suggests starting with 50 well-written ones.
• Training data comes from five places: open datasets, web crawl archives, the live web, licensed providers and synthetic examples. Common Crawl's October 2026 archive alone holds 2.16 billion pages, but a domain fine-tuning set needs a few thousand pages chosen on purpose.
• Building one from the web takes seven steps: scope, source, collect, clean and dedupe, label, format as JSONL and validate. Most broken datasets come from a skipped step, not a bad source.
• Deduplication and grounding decide the quality. The CuraWeb corpus combined n-gram and semantic deduplication to reach 2 trillion tokens, and a 3B model trained on it scored 1.8% higher on average than the baselines it was compared with. Check every model-written label against a sentence on its source page.
• Split by source website, not by row. Near-duplicate pages from one site otherwise land in both the training and validation files and inflate the evaluation score.
• Licensed data is a growing source. The Media & the Machine tracker counted 91 publicly announced AI content licensing deals by June 2026, and Firecrawl Alexandria brings licensed and curated provider data through the same API key as Firecrawl /search and Firecrawl /scrape.

An AI training dataset is a collection of examples, all in the same structure, that a model learns patterns from. A chat model learns from prompt and response pairs, a classifier learns from text with labels, and a base model learns from large amounts of plain documents. Each example should be accurate, on topic and traceable to where it came from.

The shape of the dataset follows from what you want the model to do:

Fine-tuning sets can be small. OpenAI's supervised fine-tuning guide sets the minimum at 10 examples, sees improvements from 50 to 100, and recommends starting with 50 well-crafted demonstrations. The Unsloth datasets guide suggests at least 100 rows and prefers more than 1,000. Pretraining corpora are far bigger, and CuraWeb, a 2026 English corpus built from Common Crawl, holds 2 trillion tokens.

If you are deciding whether a model needs training at all, our comparison of training data, retrieved data and live web data covers when fine-tuning helps and when retrieval is the better fix.

AI training data comes from five places: open datasets that someone already published, web crawl archives such as Common Crawl, pages you collect from the live web with a tool like Firecrawl, licensed data from providers, and synthetic examples a model writes. Most teams combine two or three, because each source covers a gap the others leave.

Common Crawl's October 2026 archive, published October 21, holds 2.16 billion pages (371 TiB of uncompressed content) from 34.3 million registered domains, a scale that suits pretraining a base model. A fine-tuning set about one domain needs a few thousand pages chosen on purpose, which is what search and scraping give you.

Licensed data is a growing source as well. The Media & the Machine tracker counted 91 publicly announced AI content licensing deals by June 2026, and it projects that deals for ongoing access to content will reach 34 in 2026, up from 18 in 2025.

Building an AI training dataset from the web takes seven steps: scope the task, find sources, collect the pages, clean and deduplicate the text, label each example, write it in a training format such as JSONL, and validate the result. Firecrawl handles the source and collect steps, and returns structured records that make the rest short.

Scoping means writing down three things before collecting anything: the task the model should learn, what one record looks like, and how many you need. "Answer homeowner questions about heat pumps in two sentences, from public guidance" is a scope. "Collect heat pump data" is not, because it gives you no way to say which pages belong.

Write a sample record by hand first. If you can't write five good examples yourself, a model can't learn the task from five thousand scraped ones.

Cleaning removes everything on a page that isn't the content, such as menus, cookie banners, footers, ads and related-post lists. Firecrawl's only_main_content option does most of this at collection time. After that, drop records that are too short, off topic or in the wrong language.

Web data needs deduplication because the same paragraph shows up on many sites. Run an exact pass that hashes normalized text, then a near-duplicate pass that compares word shingles or embeddings. The CuraWeb paper (arXiv, June 29, 2026) combined n-gram and semantic deduplication with quality filtering to build a 2 trillion token English corpus from Common Crawl, and a 3B model trained on it scored 1.8% higher on average than the baselines it was compared with.

Labeling attaches the target to each example, whether that is the ideal answer, a class or a score. For question and answer data, a model can draft the pairs from the page itself, which is what Firecrawl /scrape does when you give it a JSON schema. Ask for a supporting quote with every label, then check that the quote appears on the page. That check is cheap and it catches answers that drift away from the source.

Do that check in code rather than trusting the prompt. In Lenny's Newsletter, user researcher Caitlin Sullivan reports that, over 2,000 hours spent testing AI analysis workflows, she saw every major model still produce invented quotes, plus "Frankenstein" quotes stitched together from several sources. She also found that length caps such as "12 words or fewer" almost always produce those mash-ups, so ask for a full sentence instead. Her explanation of why it happens applies to model-written training labels too:

LLMs don't retrieve quotes like a search engine; they generate text that's statistically likely given the context.

Caitlin Sullivan, Lenny's Newsletter, February 2026

Labels that need judgment, such as tone or the right answer to a hard question, still need a person. Our guide to custom instruction datasets covers writing instruction pairs by hand and with a model.

JSONL, short for JSON Lines, is the standard format for fine-tuning data. Each line holds one complete JSON object, so a file can be streamed, appended to and split without parsing the whole thing. OpenAI's fine-tuning API requires JSONL with at least 10 lines, and the Hugging Face datasets library loads it directly. Chat data goes in a messages list with user and assistant turns.

Keep a metadata object on each line with the source URL, the fetch time and anything else you might filter on later. Training code ignores it, and it is how you trace a bad example back to the page it came from.

Validation checks that every line parses, has the right roles and non-empty content, and still matches its source. It also checks the split. Splitting by source website keeps near-identical examples from one site out of both files at once, which would otherwise inflate your evaluation score.

Firecrawl is the context API for AI agents, and for dataset work it covers three jobs. Firecrawl /search finds sources across the web and Firecrawl's own indexes. Firecrawl /scrape, Firecrawl /crawl and PDF parsing turn those sources into clean Markdown or structured JSON. Firecrawl Alexandria adds licensed and curated provider data through the same API key.

The same pieces serve larger training work too. Firecrawl's AI training use case page covers web data for pre-training, search inside reinforcement learning rollouts, and evals checked against real pages.

Minimal, an AI customer support platform for e-commerce brands, processes nearly six million storefront pages a month through Firecrawl, turning live product and policy content into data its AI support agents answer from, as described in Minimal's customer story.

Storefronts come to us in every shape imaginable, and no matter the complexity, Firecrawl turns each one into clean, structured data our AI support agents can answer from.

Niek Hogenboom, Co-Founder, Minimal

The sections below build a small heat pump question and answer dataset with each part, using the Python SDK. Install it and set your key:

pip install firecrawl-py
export FIRECRAWL_API_KEY=fc-YOUR-KEY

Firecrawl Alexandria gives AI agents and scripts structured data from official and licensed sources through the same Firecrawl API key, MCP server and CLI they use for search and scraping. Providers are paid under agreement when agents pull their data, and records arrive with their source fields, so a training example can carry where it came from and under what license.

Finding a tool is free. Each tool shows its price in Firecrawl credits before you call it, and you pay with Firecrawl credits at that price with no per-vendor minimum. Firecrawl's own indexes sit in the same catalog:

• Research Index: about 43 million paper abstracts across PubMed, bioRxiv, medRxiv and arXiv, with tools to read passages and find related papers.
• Developer Index: issues, merged pull requests and READMEs from public code repositories, plus curated documentation sites.
• Government Index: US federal, state and local statutes, regulations, codes and court opinions.

Firecrawl Alexandria has already supplied a training set. Our walkthrough on fine-tuning Laya on a Mac built 2,980 query and result pairs from the Developer Index for about 600 Firecrawl credits.

These are the dataset jobs we ran through Firecrawl Alexandria on October 27, 2026:

This script runs the first three. It checks the Hugging Face Hub for existing heat pump datasets, adds the current Wikipedia article with its license, and adds paper abstracts from the Research Index, skipping abstracts that come back shortened:

import json, os
from firecrawl import Firecrawl
 
client = Firecrawl(api_key=os.environ["FIRECRAWL_API_KEY"])
 
def call(provider, capability, **options):
    result = client.scrape(
        alexandria={"provider": provider, "capability": capability, "options": options}
    ).alexandria[0]
    if result.error:
        raise RuntimeError(result.error)
    return result.data, result.credits_cost
 
# Check what already exists before building anything
hub, hub_cost = call("huggingface-co", "hub/datasets", search="heat-pump", limit=10)
print(f"Hugging Face datasets matching 'heat-pump': {hub['count']}")
 
docs = []
 
# Licensed reference text: the current Wikipedia revision, with its license attached
articles, wiki_cost = call(
    "wikipedia", "articles/structured-contents", name="Heat pump",
    fields=["name", "abstract", "url", "license", "version.identifier", "date_modified"],
)
for article in articles:
    docs.append({
        "text": article["abstract"],
        "source": article["url"],
        "license": article["license"][0]["identifier"],
        "revision": article["version"]["identifier"],
    })
 
# Paper abstracts from the Firecrawl Research Index
papers, research_cost = call(
    "firecrawl-research-index", "search",
    query="air source heat pump performance in cold climates", k=10, **{"from": "2023-01-01"},
)
truncated = 0
for paper in papers["results"]:
    if paper["abstract"].endswith("…"):
        truncated += 1
        continue
    docs.append({
        "text": f"{paper['title']}\n\n{paper['abstract']}",
        "source": paper["primaryId"],
        "license": None,
        "revision": None,
    })
 
with open("heat_pump_docs.jsonl", "w") as f:
    for doc in docs:
        f.write(json.dumps(doc, ensure_ascii=False) + "\n")
 
print(json.dumps({
    "documents": len(docs),
    "skipped_truncated_abstracts": truncated,
    "licenses": sorted({d["license"] for d in docs if d["license"]}),
    "credits": {"hugging_face": hub_cost, "wikipedia": wiki_cost, "research_index": research_cost},
}))

Output from the run:

Hugging Face datasets matching 'heat-pump': 0
{"documents": 5, "skipped_truncated_abstracts": 6, "licenses": ["CC-BY-SA-4.0"], "credits": {"hugging_face": 5, "wikipedia": 1, "research_index": 0}}

No dataset ID on the Hub matched "heat-pump", which is the signal to build one. The Wikipedia record carries the license CC-BY-SA-4.0 and a revision ID, and Firecrawl Alexandria labels Wikipedia results "Retrieved through Wikimedia Enterprise." Six of the ten paper abstracts came back shortened with an ellipsis, so the script left them out instead of training on cut-off text.

Firecrawl Alexandria groups its providers into categories. Each one can supply a different kind of training data:

For the wider picture, see Firecrawl Alexandria's launch post and the Firecrawl Alexandria page.

Firecrawl /search takes a query and returns ranked web results, and it can scrape each result in the same call. For dataset work, its category filters narrow results to one source type, such as PDFs, the Firecrawl Developer Index, the Firecrawl Government Index or academic sites.

Firecrawl /search sets those filters with the categories parameter: pdf, developer, gov and research. A few other options matter when you collect training data:

• scrape_options returns each result's Markdown or JSON with the search, so one call both finds and collects a page.
• tbs limits results by date (qdr:m for the past month), which keeps a dataset current.
• site: inside the query restricts results to one domain, such as an official documentation site.
• sources: ["news"] returns news articles instead of web pages.

A search costs 2 Firecrawl credits per 10 results, and each scraped result adds the cost of a scrape.

Firecrawl /scrape turns a URL into Markdown, HTML or structured JSON. With a JSON schema and a prompt, it returns exactly the fields you define, so a page comes back as training records instead of text you still have to parse.

The script below runs Firecrawl /scrape through the search call's scrape_options and covers pipeline steps 2 to 7. It searches three heat pump queries, asks for up to five question and answer pairs per page with a supporting sentence quoted from the page, keeps only pairs whose quote appears in the page text, removes exact and near-duplicate questions, and writes chat-format JSONL split by website:

import hashlib, json, os, re
from datetime import datetime, timezone
from urllib.parse import urlsplit
from firecrawl import Firecrawl
from firecrawl.v2.types import ScrapeOptions
 
client = Firecrawl(api_key=os.environ["FIRECRAWL_API_KEY"])
 
QUERIES = [
    "how do air source heat pumps work in cold weather",
    "heat pump sizing and installation guide for homes",
    "heat pump vs furnace running costs explained",
]
QA_SCHEMA = {
    "type": "object",
    "properties": {
        "qa_pairs": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "question": {"type": "string"},
                    "answer": {"type": "string"},
                    "evidence": {"type": "string"},
                    "difficulty": {"type": "string", "enum": ["beginner", "intermediate", "advanced"]},
                },
                "required": ["question", "answer", "evidence", "difficulty"],
            },
        }
    },
    "required": ["qa_pairs"],
}
PROMPT = (
    "Write up to 5 questions a homeowner might ask that this page answers, each with a short answer. "
    "In evidence, copy one sentence from the page, word for word, that supports the answer."
)
 
def norm(text):
    text = re.sub(r"\]\([^)]*\)|https?://\S+", " ", text.lower())
    return " ".join(re.sub(r"[^a-z0-9%$]+", " ", text).split())
 
def shingles(text, n=3):
    words = norm(text).split()
    return {" ".join(words[i:i + n]) for i in range(max(1, len(words) - n + 1))}
 
# 1. Source and 2. collect: one search call per query returns pages already extracted with the schema
pages, seen_urls = [], set()
for query in QUERIES:
    results = client.search(
        query,
        limit=5,
        scrape_options=ScrapeOptions(
            formats=["markdown", {"type": "json", "schema": QA_SCHEMA, "prompt": PROMPT}],
            only_main_content=True,
        ),
    )
    for doc in results.web or []:
        if not getattr(doc, "json", None) or not getattr(doc, "markdown", None):
            continue
        key = doc.metadata.url.split("#")[0].rstrip("/")
        if key in seen_urls:
            continue
        seen_urls.add(key)
        pages.append(doc)
 
# 3. Clean, 4. dedupe and 5. label
fetched_at = datetime.now(timezone.utc).isoformat(timespec="seconds")
records, seen_questions, kept_shingles = [], set(), []
stats = {"pages": len(pages), "extracted": 0, "ungrounded": 0, "duplicates": 0}
for doc in pages:
    page_text = norm(doc.markdown)
    for qa in doc.json.get("qa_pairs", []):
        stats["extracted"] += 1
        question, answer, evidence = (qa.get(k, "").strip() for k in ("question", "answer", "evidence"))
        if not question or len(answer) < 20 or len(evidence) < 20 or norm(evidence) not in page_text:
            stats["ungrounded"] += 1
            continue
        qhash = hashlib.sha1(norm(question).encode()).hexdigest()
        qsh = shingles(question)
        if qhash in seen_questions or any(len(qsh & s) / len(qsh | s) > 0.6 for s in kept_shingles):
            stats["duplicates"] += 1
            continue
        seen_questions.add(qhash)
        kept_shingles.append(qsh)
        records.append({
            "messages": [
                {"role": "user", "content": question},
                {"role": "assistant", "content": answer},
            ],
            "metadata": {
                "source_url": doc.metadata.url,
                "source_title": " ".join((doc.metadata.title or "").split()),
                "evidence": evidence,
                "difficulty": qa.get("difficulty"),
                "fetched_at": fetched_at,
            },
        })
 
# 6. Format as JSONL, split by source domain so one site never lands in both files
def split_for(url):
    domain = urlsplit(url).netloc
    return "val" if int(hashlib.md5(domain.encode()).hexdigest(), 16) % 5 == 0 else "train"
 
files = {name: open(f"heat_pump_{name}.jsonl", "w") for name in ("train", "val")}
for record in records:
    files[split_for(record["metadata"]["source_url"])].write(json.dumps(record) + "\n")
for f in files.values():
    f.close()
 
# 7. Validate what landed on disk
counts = {}
for name in files:
    with open(f"heat_pump_{name}.jsonl") as f:
        rows = [json.loads(line) for line in f]
    assert all([m["role"] for m in r["messages"]] == ["user", "assistant"] for r in rows)
    assert all(r["messages"][1]["content"] and r["metadata"]["source_url"] for r in rows)
    counts[name] = len(rows)
stats.update(kept=len(records), **counts, domains=len({urlsplit(r["metadata"]["source_url"]).netloc for r in records}))
print(json.dumps(stats))

We ran it on October 27, 2026. It finished in 16 seconds and the account balance dropped by 132 Firecrawl credits:

{"pages": 13, "extracted": 65, "ungrounded": 4, "duplicates": 0, "kept": 61, "train": 33, "val": 28, "domains": 10}

Each line of heat_pump_train.jsonl is ready for a fine-tuning job, with the source attached:

{"messages": [{"role": "user", "content": "Do heat pumps work in cold climates?"}, {"role": "assistant", "content": "Yes, there are heat pumps available today that are specifically designed to work in cold climates."}], "metadata": {"source_url": "https://cleanheat.ny.gov/heat-pumps-cold-climates-do-they-work", "source_title": "Heat Pumps for Cold Climates - Do They Work? : NYS Clean Heat", "evidence": "There are heat pumps available today that are specifically designed to work in cold climates and are capable of meeting up to 100% of a New York home\u2019s heating needs.", "difficulty": "beginner", "fetched_at": "2026-10-27T23:54:57+00:00"}}

A few things to know about the run:

• PDF results cost more. Twelve web pages came to 5 Firecrawl credits each (1 for the scrape, 4 for JSON), the three searches to 6, and one result was a PDF sizing guide billed per page for 66 Firecrawl credits.
• Normalize before you compare. Page Markdown keeps link URLs and symbols such as ® inside sentences, so the norm function strips links and punctuation from both the quote and the page before matching.
• The split is uneven at this size. With 10 websites, hashing domains put 4 sites in validation and 6 in training. At a few hundred domains the split settles close to 80/20.
• Pairs vary between runs. The model writing the pairs words them a little differently each time, so earlier runs of the same script kept 56, 58 and 60 records. Keep the queries fixed and save the output you validated.
• Scale with Firecrawl batch scrape. When you already have a list of thousands of URLs, client.batch_scrape(urls, formats=[...]) runs the same JSON extraction across all of them as one job.

A lot of the world's important information is buried in PDFs, including research papers, government reports, standards, manuals and spec sheets, and none of it is in a page's HTML. Firecrawl reads PDFs through the same Firecrawl /scrape call, and Firecrawl /parse takes a local file or a document that isn't online.

The engine underneath is Fire-PDF, built in Rust and one of the fastest PDF parsers in the world. It runs on pdf-inspector, Firecrawl's open-source Rust PDF library and a developer favorite with more than 19,000 GitHub stars. In pdf-inspector's published benchmark on 200 PDFs, refreshed July 31, 2026, it parsed the full set in 0.47 seconds and had the highest overall, reading order and table scores of the five local engines tested.

This script finds a report with categories=["pdf"], reads its first 25 pages, and turns each section into a record while dropping the front matter:

import json, os, re
from firecrawl import Firecrawl
 
client = Firecrawl(api_key=os.environ["FIRECRAWL_API_KEY"])
 
# Find reports published as PDFs
hits = client.search("DOE cold climate heat pump challenge field validation results", categories=["pdf"], limit=3)
pdf_url = next(hit.url for hit in hits.web if hit.url.lower().endswith(".pdf"))
 
# Read the first 25 pages as Markdown
doc = client.scrape(pdf_url, formats=["markdown"], parsers=[{"type": "pdf", "mode": "auto", "max_pages": 25}])
 
# Split on headings so each record is one section, and drop front matter
FRONT_MATTER = re.compile(r"disclaimer|operated by|contents|acknowledg|acronyms|printed in|^figures$|^tables$", re.I)
records = []
for section in re.split(r"\n(?=#{1,6} )", doc.markdown):
    heading = re.sub(r"<[^>]+>|^#+\s*", "", section.strip().splitlines()[0])
    if len(section.split()) < 60 or FRONT_MATTER.search(heading):
        continue
    records.append({"text": section.strip(), "source": pdf_url, "section": heading})
with open("heat_pump_pdf_sections.jsonl", "w") as f:
    for record in records:
        f.write(json.dumps(record) + "\n")
 
print(json.dumps({
    "pdf": pdf_url,
    "pages": doc.metadata.num_pages,
    "credits": doc.metadata.credits_used,
    "sections_kept": len(records),
    "first_sections": [r["section"] for r in records[:4]],
}))

Output from the run:

{"pdf": "https://www.pnnl.gov/main/publications/external/technical_reports/PNNL-37127.pdf", "pages": 25, "credits": 25, "sections_kept": 10, "first_sections": ["Performance Results from DOE Cold Climate Heat Pump Challenge Field Validation", "Executive Summary", "1.0 Introduction", "1.1 Challenge Participants"]}

The PDF came to 25 Firecrawl credits, 1 per page, plus 2 Firecrawl credits for the search. The executive summary alone came back as 2,632 words, so split long sections into chunks that fit your model's context length before training. For a PDF on your own disk, client.parse("report.pdf") returns the same Markdown through Firecrawl /parse.

Firecrawl /crawl follows links from a starting URL and scrapes every page it reaches, up to a limit you set. It suits sources where one site holds the whole topic, such as a documentation site, a help center or a public program's guidance pages, and path filters keep it to the part of the site you want.

The NYS Clean Heat site publishes heat pump guidance in English and Chinese. This Firecrawl /crawl job keeps the English pages and skips PDFs, which get their own section in this post:

import os
from firecrawl import Firecrawl
 
client = Firecrawl(api_key=os.environ["FIRECRAWL_API_KEY"])
 
job = client.crawl(
    "https://cleanheat.ny.gov",
    limit=15,
    exclude_paths=["^/zh-CN/.*", ".*\\.pdf$"],
    scrape_options={"formats": ["markdown"], "only_main_content": True},
)
print(job.status, f"{job.completed} pages, {job.credits_used} credits")
for page in job.data[:5]:
    print(page.metadata.url, len(page.markdown.split()), "words")

Output from the run:

completed 15 pages, 15 credits
https://cleanheat.ny.gov/how-they-work/ 436 words
https://cleanheat.ny.gov/compare-your-options/ 191 words
https://cleanheat.ny.gov/financing-options/ 250 words
https://cleanheat.ny.gov/find-available-rebates/ 360 words
https://cleanheat.ny.gov/what-to-expect/ 349 words

Crawled pages take the same scrape_options as a single scrape, so passing the JSON schema from the Firecrawl /scrape pipeline turns a whole site into question and answer records.

The pipeline in this post uses Python, but every call in it also runs from Firecrawl's other official SDKs, the REST API, the Firecrawl CLI and the Firecrawl MCP server. AI assistants, agent frameworks and no-code tools can call the same Firecrawl /search, Firecrawl /scrape, Firecrawl /crawl and Firecrawl Alexandria tools.

Keyless mode runs Firecrawl /search and Firecrawl /scrape from the CLI, and adds Firecrawl /parse on the keyless MCP server, with no API key. The Research, Developer and Government indexes work without a key too. Firecrawl /crawl and Firecrawl Alexandria provider calls need a Firecrawl API key.

The mistakes that break AI training datasets built from the web are usually skipped steps: no written scope, boilerplate left in, duplicates, unchecked labels, row-level splits, missing source URLs and truncated text. Each one is cheap to catch before a training run and expensive to find after it.

What each one looks like:

1. No written scope. Without a sample record and a target count, the dataset drifts toward whatever the search returned first.
2. Boilerplate left in. Cookie banners, menus and "related posts" lists become training text unless you collect only the main content.
3. Duplicates across sites. Syndicated articles and copied documentation repeat the same passage many times, which skews what the model learns.
4. Labels nobody checked. Model-drafted answers need a grounding check against the source and a human read of a sample.
5. Row-level splits. Splitting by row instead of by source puts near-identical examples in both training and validation files.
6. Missing provenance. Without a source URL and fetch time on each record, you can't trace or remove a bad example later.
7. Truncated text. Shortened abstracts, cut-off pages and over-long sections teach the model incomplete patterns. Filter and chunk them.

For preparing data you already hold, our AI data preparation guide covers transformation and quality checks in more depth, and the guide to data acquisition tools compares ways to collect it.

The quickest way to try Firecrawl for dataset work is keyless mode, which needs no account or API key. Search a topic and scrape a page into JSON from the Firecrawl CLI or the keyless MCP server, then add a free account when you need crawls or Firecrawl Alexandria calls.

Run npx firecrawl-cli@latest or connect the keyless MCP server at https://mcp.firecrawl.dev/v2/mcp to start. Keyless use has a daily limit per IP address. A free account adds 1,000 Firecrawl credits a month, which covers the crawl and Firecrawl Alexandria calls in this post.

67% of Fortune 100 companies use Firecrawl. Teams with procurement requirements can review the enterprise page and security documentation.