AI Tokens: What They Are, How They're Counted, and Why It Costs You

The short version: a token is not a unit of length. It is a unit of whatever the provider's tokenizer decided, and providers decided differently. Compare models on measured cost for your own content, and fix your input format before you shop for a cheaper model, because format is worth more than the model switch.

You benchmarked two models, picked the one with the lower price per million tokens, and the invoice came in higher anyway. Nothing was miscounted. The two providers simply disagree about how many tokens your text contains, and the one with the better sticker price turned your content into more of them.

That disagreement is LLM tokenization, and it is the layer under every token bill you have ever paid. Once you start comparing AI tokens across models rather than assuming they match, two decisions change: which model you buy, and what you feed it.

A token in AI is the smallest unit a language model actually reads, prices, and counts against your context window. AI tokens are not words and not characters: they are whatever fragments the provider's tokenizer produced when it learned its vocabulary, and that vocabulary is different for every model.

Tokenization is the step that turns your text into those discrete units. A tokenizer holds a fixed vocabulary, typically 100,000 to 250,000 entries, learned from a training corpus by repeatedly merging the most frequent character pairs.

Common words survive as single tokens. Rare ones get chopped into fragments. Because each provider trains its vocabulary on its own corpus with its own algorithm, the same string lands on different boundaries, and you get a different count.

That count is the billing unit and the context-window unit. It is also the unit context engineering budgets in, and the unit chunking splits on, which is the first practical consequence: a chunk size tuned on one model does not transfer to another. If you sized your chunks at 512 tokens against GPT, the same chunking strategy produces meaningfully larger passages on Claude.

Vocabulary size and training corpus drive almost all of it. A larger vocabulary holds more whole words, so it needs fewer pieces per word. A vocabulary trained heavily on English handles English efficiently and fragments everything else. The Cambridge and Hong Kong team behind arXiv:2601.11518 traces both effects across ten production tokenizers.

Take one word. Researchers at Cambridge and the University of Hong Kong tested ten tokenizers and found that antidisestablishmentarianism becomes 9 tokens on Claude and 5 on Gemini and Grok, with the full range running from 4 to 9 depending on the model. Our own counts agree: Gemini's countTokens returns 5 for that word and Anthropic's returns 8.

One word is a curiosity. The same effect across a million words is a budget line.

The same study found compression ratios differing by 100% between the extremes on emoji and numeric content, and by 20 to 50% on everything else. It also found that compression varies by nearly a factor of ten between a model's worst domain and its best, and that English essays compress about 25% better than technical writing.

That last point matters more than it looks. Your token cost depends on what kind of text you send, not only how much.

To compare token counts across models fairly, you have to send the exact same bytes through each provider's own counter. TensorZero ran the cleanest public tokenizer comparison of this in April 2026, feeding identical inputs through each API and normalizing against OpenAI.

Source: TensorZero, published April 16, 2026, retrieved August 18, 2026. Inputs were The Iliad for text, Cloudflare's OpenAPI spec for JSON and YAML, and 100 generated tool definitions.

Read the rows and the columns. Across the rows, every model gets worse as content gets more structured. Down the columns, the spread between providers widens from 1.57x on prose to 2.65x on tool schemas.

This partly explains gaps that show up in agent benchmarks. When Composio measured Claude Code against Codex on identical tasks and found 192,000 tokens against 136,000, some of that 1.4x is the harness sending more, and some of it is the tokenizer counting the same work differently.

Multiply the multiplier by the list price and the real cost per million tokens falls out, which is the comparison you actually care about.

Source: TensorZero, published April 16, 2026, retrieved August 18, 2026. Base input token prices only.

Two things break the naive comparison. First, claude-opus-4-7 ends up costing 5.3x what gpt-5.4 costs on tool-heavy work even though their list prices are only 2x apart. Second, the ranking is not stable: Gemini is the cheapest option on text at $2.12 effective, then becomes 46% more expensive than OpenAI once you send tool definitions.

There is no single cheapest model. There is only a cheapest model for a given content mix, which is why "which model is cheaper" is unanswerable without measuring your own traffic. If you have compared agents on token cost, this is the correction to apply to those numbers.

Two more effects sit on top and both cut the other way. Prompt caching discounts repeated context heavily, and long-context tiers raise prices past a threshold. TensorZero's figures cover base input pricing only, and so does the reasoning here.

Structure. Every time.

Prose is what tokenizer vocabularies are built for, so common words compress into single tokens. The same Cambridge tokenizer study measured English essays compressing about 25% better than technical writing, and structured content sitting worse than either. Structured text is the opposite: braces, quoted keys, colons, indentation, and closing tags produce short tokens carrying almost no meaning. A JSON key like "created_at" spends tokens on two quote marks and an underscore before it spends any on the concept.

This is the mechanism behind a cost most agent developers have already met. Tool schemas are pure structure, which is why they land at the 2.65x end of the table and why loading many of them is expensive before an agent does any work. We have covered what MCP servers cost against a CLI in detail, and the tokenizer is one half of why those numbers are as bad as they are.

The same effect shows up in miniature elsewhere. A Claude Code skill costs roughly 100 tokens just to advertise itself at startup, and that overhead is mostly frontmatter rather than instruction.

Which brings us to the most structured input anyone routinely hands a model.

No published benchmark covered this, so we ran one.

We took 15 real pages spanning the kinds of things agents actually fetch, scraped each one twice through Firecrawl in a single run, once as untouched rawHtml and once as markdown with onlyMainContent: true, then counted both with each provider's own tokenizer. GPT counts come from tiktoken with the o200k_base encoding, which runs locally and is exact. Claude counts come from Anthropic's count_tokens endpoint with the 7-token message wrapper subtracted. Gemini counts come from Google's countTokens on gemini-3.1-pro-preview, which adds no wrapper.

Source: Firecrawl benchmark, retrieved August 18, 2026. Eight of the fifteen pages measured, ordered by raw HTML size. Specs and raw counts in the article's research folder.

The aggregate: 4,103,706 GPT tokens as raw HTML against 190,817 as markdown, a 21.5x reduction. Median per page was 19x, and the spread ran from 2.6x on a Hacker News front page to 283x on Vercel's homepage.

The reduction holds on every tokenizer: 21.5x on GPT, 20.8x on Claude, 22.5x on Gemini.

The cost story is the obvious one. The feasibility story is the sharper one: five of the fifteen pages exceed 200,000 tokens as raw HTML, which means they do not fit in a 200k context window at all. As markdown, not one page came close. Format is not only what you pay, it is whether the request is possible.

Why the enormous outliers? The Stripe reference and the Vercel homepage are both JavaScript-heavy pages that ship inline scripts, serialized state, and styling in the document. Almost none of that is content. That is the same boilerplate problem that degrades retrieval quality in a web index, showing up here as a bill.

Now the part that surprised us. We expected cleaning the page to shrink the gap between tokenizers, since tag soup should be the hard case. Instead the two non-OpenAI tokenizers moved in opposite directions.

Source: Firecrawl benchmark, 4.1M HTML tokens and 191k markdown tokens across 15 pages, counted with each provider's own tokenizer, retrieved August 18, 2026.

Claude went from 1.168 tokens per GPT token on raw HTML to 1.210 on markdown, a gap that widened on 13 of the 15 pages. Gemini went the other way, from 1.249 down to 1.191. Gemini is the most expensive tokenizer in the set on raw HTML and the middle one on markdown.

The explanation is that HTML is repetitive ASCII markup that every tokenizer handles roughly equally, so cleaning a page strips out the easy part and leaves the prose, where vocabularies genuinely differ. Which way that lands depends on the specific vocabulary, and it is not predictable from first principles.

So there is no rule of thumb here, which is the practical point. Format and tokenizer are separate levers, and how they interact depends on the pair you happen to be using. What does hold across all three tokenizers is the size of the format lever: roughly 21x, against a tokenizer spread of about 1.2x. Cleaning your input does not reliably buy a better exchange rate, but it always buys you an order of magnitude less to exchange. For the mechanics of the conversion, see our guide to scraping a website to markdown.

If you crawl beyond English, this is the largest tokenization effect you will meet.

Petrov and colleagues showed at NeurIPS 2023 that the worst-served languages cost more than twelve times what English costs for equivalent content, and that the disparity is created at the tokenization stage, in their words "well before the language model sees any data at all." The same premium hits latency and how much text fits in the window.

Source: Tokenization efficiency of foundational LLMs for the Ukrainian language, PMC12380774, retrieved August 18, 2026. The Claude 3 figure is approximated by the study's authors, since Anthropic publishes no offline tokenizer.

Newer tokenizers with larger vocabularies narrow the gap without closing it. GPT-4o needs 1.98 tokens per Ukrainian word against 1.07 for English, while the older GPT-3.5 and GPT-4 vocabulary needs 3.32. If your crawl targets non-English sites, tokenizer choice is worth more than a few percentage points of list price.

One caveat against assuming the obvious: the Cambridge study found no significant correlation between a language's compression ratio and how common it is in Common Crawl. You cannot predict the premium from how widely spoken a language is. You have to measure it.

Stop estimating. Every provider ships a count tokens API or a local library that is exact, and they disagree with each other, which is the whole point.

# OpenAI: local and exact, no API call
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
gpt_tokens = len(enc.encode(text))
 
# Anthropic: subtract the 7-token message wrapper for a bare-string count
import anthropic
client = anthropic.Anthropic()
claude_tokens = client.messages.count_tokens(
    model="claude-sonnet-4-5",
    messages=[{"role": "user", "content": text}],
).input_tokens - 7
 
# Google
from google import genai
gemini_tokens = genai.Client().models.count_tokens(
    model="gemini-3.1-pro-preview", contents=text
).total_tokens

Two traps worth naming. Anthropic's endpoint rejects payloads past roughly 100,000 characters, so long documents need chunking and summing; at 80,000-character chunks we measured the boundary error at 8 tokens in 24,452, or 0.03%. And do not reach for the four-characters-per-token shortcut. The Cambridge team called that heuristic "inadequate and oversimplified", measuring 0.35 to 0.45 words per token on randomly sampled English words against the 0.75 the rule promises.

Once you are counting properly, track it in production. Per-provider token spend is a standard metric in most LLM observability tools, and it is the only way to catch a content-mix shift moving your effective price.

You cannot change a provider's tokenizer. You can change two things that matter more.

Pick the model against measured cost, not list price. Run a representative sample of your real traffic through all three counters, multiply by list price, and compare that. If your workload is tool-heavy, the answer will likely differ from the one the pricing page implies.

Fix the input format first. Our benchmark puts this at 21.5x, against a tokenizer spread of roughly 1.2x. For token efficiency, switching models is a rounding error next to not sending tag soup.

In practice that means three things, and Firecrawl is built around each one:

• Send markdown, not HTML. Firecrawl's /scrape returns clean markdown by default, with onlyMainContent: true stripping navigation, footers, and inline scripts. That is the 21.5x.
• Send excerpts, not documents, when you only need an answer. The question and highlights formats return grounded answers or verbatim passages instead of the whole page.
• Keep retrieved pages from accumulating. Scraped content stays in context for every subsequent turn, which is exactly how web research inflates a session.

If your agent runs in Claude Code specifically, our 12 ways to cut token consumption covers the harness-level tactics that stack on top of these.

Token counts are provider-specific measurements dressed up as a universal unit, and treating them as universal is how a cheaper model produces a larger bill. Measure your own content through each provider's counter before you commit to one.

Then look at what you are sending. On the fifteen pages we tested, the format of the input mattered twenty times more than the choice of model.