LLM Token Optimization: 9 Techniques in the Order to Apply Them

Fix the biggest input first, lay the prompt out so the provider can cache it, and stop the model from writing more than you read. Routing, batching, and compression come after those three, and every change needs an eval behind it, because a cheap request that fails costs more than an expensive one that works.

Much of an agent's token bill is content it has already sent. Model APIs are stateless, so each turn resends the whole conversation: the system prompt, the tool definitions, earlier replies, and any page the agent fetched. A page read on turn one of an eight-turn task is billed eight times.

Longju Bai, a PhD researcher at the University of Michigan, and colleagues measured this in the first systematic study of agent token consumption, across eight frontier models on SWE-bench Verified. Agentic coding tasks consumed 1,000x more tokens than code chat or code reasoning, and most of that cost came from input tokens.

The techniques below apply to apps and agents you build on the OpenAI or Anthropic APIs. If you want to cut token usage inside Claude Code itself, see 12 ways to cut token consumption in Claude Code.

LLM token optimization is reducing how many tokens an application sends to and receives from a model, and how much each of those tokens costs, without lowering the quality of the result. In practice, LLM cost optimization comes down to two levers: shrinking the token count (send less, generate less) and shrinking the unit price (cache, batch, route to a cheaper model).

AI token cost depends on which meter a token lands on, because providers bill input, output, and cached tokens separately at very different prices. These are from OpenAI's pricing page and Anthropic's pricing page on October 13, 2026:

Standard short-context prices per million tokens. Anthropic's cache write is the 5-minute tier. OpenAI's long-context rates double input and raise output by 1.5x, and Haiku 5.5 rises to $0.50 input and $2.50 output for prompts over 100,000 tokens.

Output is 5x input on every row from both providers. A cache read costs 1/10th to 1/40th of a fresh input token depending on the model, and 1/20th on both mid-tier models. The gap between the top and bottom model is 100x on input at both providers.

The same text also becomes a different number of tokens on each provider, sometimes by more than 2x, which we measured in AI tokens: what they are, how they're counted, and why it costs you.

In an agent or RAG app, a single call usually carries five layers:

1. Static prefix: system prompt, instructions, few-shot examples. Identical on every call.
2. Tool definitions: every tool schema the model can call. Anthropic adds a further 286 tokens of tool-use system prompt on Opus, Sonnet, and Haiku 5.5 whenever tools are present.
3. Conversation history: every prior user message, assistant reply, tool call, and tool result.
4. Retrieved or fetched content: RAG chunks, scraped pages, file contents, API responses.
5. Output: the visible answer plus any reasoning or thinking tokens, all billed at the output rate.

Layers 1 and 2 are the same on every call, which makes them ideal for caching. Layers 3 and 4 grow with every turn, which makes them the source of runaway bills. Layer 5 is the most expensive per token and the slowest to produce.

An eight-turn agent that reads one 12,000-token page per turn and writes 800 tokens per turn, on top of a 12,000-token prefix, processes 550,400 input tokens for one task, of which only 96,000 are new page content.

Finished work per token is the metric I would track too.

The early techniques are cheap to apply and carry no quality risk. The later ones save less per hour of work or need evaluation before you trust them.

To reduce LLM token usage, you first need it broken down per task. Every provider returns usage on every response. On Anthropic, read input_tokens, output_tokens, cache_creation_input_tokens, and cache_read_input_tokens. On OpenAI, read cached_tokens inside the usage details. Log them per request, tagged with the feature, the task ID, and the model.

Three numbers tell you where to start:

• Input tokens per task, split by layer. If retrieved content is 80% of input, that is your first fix.
• Output tokens per task. At 5x the input price, a verbose model can cost more than a long prompt.
• Cache hit rate. Cached read tokens divided by total input tokens. If this is low on a multi-turn workload, your prompt layout is breaking the cache.

Thariq Shihipar, a member of the Claude Code team at Anthropic, wrote that the Claude Code team runs alerts on prompt cache hit rate and declares SEVs if it drops too low, because a few percentage points of cache misses can dramatically change cost and latency.

For counting before you send, use tiktoken locally for OpenAI models, Anthropic's /v1/messages/count_tokens endpoint, or Gemini's countTokens. Estimating from character counts is off by enough to matter, which we showed in the tokenization benchmark.

Our roundup of LLM observability tools covers dashboards that track per-request token usage.

Retrieved and fetched content is the largest layer in most agents and the easiest to shrink, because most of it was never needed.

The first cut is format. In our tokenization benchmark, fifteen real pages came to 4,103,706 GPT tokens as raw HTML and 190,817 as clean markdown from Firecrawl, a 21.5x reduction. For document Q&A or RAG ingestion, markdown is the right place to stop, because the model needs the whole page.

An agent that needs one fact from a page can skip the page. We tested this head to head against Claude Code's built-in WebFetch tool in Claude Code uses too many tokens on web search, using the same prompts through both:

Source: Firecrawl, measured with Claude Code's /context and tool output lengths.

Every character saved on a tool result is saved again on every later turn that resends the history.

Web search is the input most agents can't do without when they need current information, such as today's prices, docs for a library released last month, or news. A model only knows what it saw before its training cutoff, so without search it answers from stale knowledge. Search is also one of the heaviest tool calls, because each result can carry a full page, which makes the format a search provider returns a cost decision.

On Openbenchmarks' web search for coding agents board, updated September 26, 2026, a fixed coding agent solves the same tasks through each search provider while the harness records the median LLM tokens per task:

Perplexity completed more tasks than Firecrawl, using about 18% more tokens. On the search-and-fetch board, Firecrawl's 17,379 median tokens per task came in below Exa, Parallel, Tavily, and Perplexity.

For data with an official source, such as SEC filings or economic series, Firecrawl Alexandria returns typed records from 100+ data providers through the same search call, so the agent gets the numbers without a page around them.

In code, the difference is one format option:

import { Firecrawl } from "firecrawl";
 
const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });
 
// Whole page as clean markdown: right for RAG ingestion
const page = await firecrawl.scrape("https://replit.com/pricing", {
  formats: ["markdown"],
  onlyMainContent: true,
});
 
// One answer: right for an agent tool call
const fact = await firecrawl.scrape("https://replit.com/pricing", {
  formats: [{ type: "query", prompt: "How much does the Core plan cost per month?" }],
});
 
console.log(fact.answer);

For several fields at once, the json format with a schema returns only those fields, as typed values you can validate.

Answers and summaries from any tool, including Claude Code's own WebFetch summarizer, are lossy by design. Use a direct answer or a JSON schema for facts your code can check, and keep markdown for anything the model must reason over in full.

Outside the web, the same applies to a database tool that returns the three matching rows instead of the table, a file tool that returns a diff, and a test runner that returns the failing test instead of the full log.

Prompt caching is the largest unit-price discount available, and a prompt that changes near the start of the request forfeits it without any error.

Both major providers cache by prefix. The provider stores the processed state of your request up to a point, and the next request reuses it only if everything before that point is byte-identical. On Anthropic, you mark breakpoints with cache_control (up to four) or turn on automatic caching. A 5-minute write costs 1.25x base input, a 1-hour write costs 2x, and a read costs 0.05x on Opus 5.5 and Sonnet 5.5, 0.025x on Fable 5.1, and 0.1x on older models, per Anthropic's caching docs. The minimum cacheable prompt is 512 tokens on current models.

OpenAI caching is automatic for prompts of 1,024 tokens or more, with cached input discounted by up to 95% on recent models and a prompt_cache_key parameter to steer related requests to the same cache. OpenAI now also lists explicit cache writes at 1.25x input. As one HN commenter explained in the thread on ngrok's prompt caching explainer, a chatbot's requests can get routed to different machines, so implicit caching misses unless you set the key to something stable like a user ID.

Prompt order decides the hit rate, and Thariq's write-up of how Claude Code is built around caching covers it in detail:

The rules from it that apply to any app:

• Static content first, dynamic content last. System prompt and tools, then project-level context, then session context, then messages.
• Never put a timestamp in the system prompt. Pass changing facts like the date or a file change in the next user message instead.
• Never add, remove, or reorder tools mid-session. Tool definitions sit in the prefix, so changing one invalidates the cache for the whole conversation.
• Don't switch models mid-session. Caches are per model. At 100k tokens into an Opus conversation, switching to Haiku for an easy question costs more than letting Opus answer, because Haiku has to rebuild the cache from scratch.
• Fork side work from the same prefix. Compaction and summarization calls should reuse the parent's system prompt, tools, and history, then append the new instruction at the end.

The first controlled study of this, "Don't Break the Cache" (January 2026), ran more than 500 sessions of a web research agent with 10,000-token system prompts on OpenAI, Anthropic, and Google models. Caching cut API cost by 41-80% and time to first token by 13-31%. Which parts you cache also mattered. Caching only the system prompt, or excluding dynamic tool results from the cached block, gave more consistent savings than caching the full context, which in some configurations made latency worse. The authors also found the savings scale linearly with prompt size once a prompt clears the provider's minimum cacheable length.

Cache lifetime (TTL) is a tradeoff. The 1-hour write costs 60% more than the 5-minute write, so it only pays when the gap between requests is longer than five minutes. An issue filed against Claude Code in April 2026, which reached the Hacker News front page with 552 points, estimated that 17.1% of one user's spend was waste after Claude Code's cache TTL moved from 1 hour to 5 minutes. The reply on the issue said the change was intentional and that 1 hour everywhere would cost more, because many calls are one-shot and 1-hour writes cost more than 5-minute writes. Measure the time between turns in your own traffic and pick the TTL that matches it.

Semantic caching is a separate layer that stores whole responses keyed by the meaning of the query and skips the model entirely on a hit. It helps for FAQ-style traffic with lots of near-duplicate questions, and does little for agents, where almost every request is unique. Prefix caching helps both.

Output tokens cost 5x input per token and are generated one at a time, so they dominate latency even in agents where input dominates the bill. Reasoning tokens bill at the output rate too.

Four controls, in order of effort:

• Set max_tokens on every call. Without it, nothing stops a long answer you did not need.
• Lower reasoning effort where the task is easy. Anthropic's effort parameter takes low, medium, high, xhigh, or max and applies to all output tokens, including text, tool call arguments, and thinking. Classification and extraction rarely need more than low. OpenAI exposes a similar reasoning effort setting on its reasoning models.
• Ask for structured output. A JSON object with the fields you need is shorter than prose that contains them, and you skip a parsing step.
• Tell the model the length. "Answer in two sentences" paired with a max_tokens buffer works better than a hard cap alone, which can truncate mid-sentence.

Use the API parameters instead of asking in the prompt. A November 2025 study of reasoning models, Srivastava et al., found that instructions like "use no more than 100 tokens in thinking" did not reduce overthinking, because models either ignored them or lost accuracy. The same study found that batching several queries into one prompt cut reasoning tokens by 76% on average (2,950 to 710) across 13 benchmarks on DeepSeek-R1 and o1, while preserving or improving accuracy. If your workload has many small independent questions, grouping them into one request is worth testing.

Effort controls pay off because, as a 2025 evaluation of four thinking models found, these models waste computation on easy problems and earn their cost only on harder ones.

Changing the thinking or effort configuration invalidates cached message blocks on Anthropic, so pick a setting per feature and keep it stable instead of tuning it per request.

Tool definitions are pure structure, and structure tokenizes worst. Anthropic's docs put a typical setup with GitHub, Slack, Sentry, Grafana, and Splunk servers at about 55,000 tokens of definitions before the model does any work, and note that tool selection accuracy drops past 30 to 50 tools.

Anthropic's tool search tool defers them, sending lightweight stubs and loading a full schema only when the model searches for it, which the docs say cuts definition tokens by over 85%. Because the stubs never change, the cached prefix stays stable, which you would lose by adding and removing tools as needed.

Tool results are the other half. A February 2026 project called context-mode, which hit 570 points on Hacker News, runs tool calls in a sandbox and returns only stdout. Its author reports a Playwright snapshot dropping from 56 KB to 299 bytes and a full session from 315 KB to 5.4 KB.

Whether tools reach the agent as MCP servers or CLI commands also changes the overhead. We compared the two in MCP vs CLI for AI agents.

Conversation history grows on every turn and fills the context window, and a long history costs money on the bill and accuracy in the answer.

Chroma's context rot study evaluated 18 models and found performance degrading as input length grows, even on simple retrieval tasks.

JetBrains researchers compared context strategies inside SWE-agent on SWE-bench Verified in "The Complexity Trap", presented at a NeurIPS 2025 workshop. Replacing older tool observations with a placeholder halved cost relative to the raw agent and matched, sometimes slightly beat, the solve rate of LLM summarization. A hybrid of the two cut cost a further 7% against masking alone.

For agents, most context window optimization comes down to deciding which old tool results to drop. Two server-side tools handle this on Anthropic:

• Context editing clears old tool results once context passes a threshold (100,000 input tokens by default, keeping the last 3 tool uses), replacing each with a placeholder. It can also clear old thinking blocks. The docs' worked example takes a request from 70,000 input tokens to 25,000.
• Compaction summarizes the conversation when it nears the window limit and continues from the summary. Built the naive way, with a separate call and a different system prompt, compaction pays full price for the whole history. Built cache-safe, it reuses the parent prefix and only the summary instruction is new.

Clearing a tool result invalidates the cached prefix from that point onward, which is why Anthropic's docs recommend a clear_at_least value large enough to make each rebuild worthwhile. It trades one cache rebuild for a permanently smaller context. That trade pays off when the cleared content is large and the session is long. It costs more than it saves on short sessions.

For deciding what belongs in context at all, see context engineering vs prompt engineering.

Haiku 5.5 input costs 1/20th of Sonnet 5.5 and 1/40th of Opus 5.5. For classification, extraction, routing, and short summaries, a small model often passes the same eval.

RouteLLM trained routers on human preference data and cut cost by over 2x in some benchmark settings without lowering response quality. Earlier work on cascades, FrugalGPT, sent each query to cheap models first and escalated only when a scorer judged the answer unreliable, matching GPT-4's accuracy with up to 98% cost reduction on its test sets. Real traffic usually lands well short of the upper bound, because hard queries still need the large model and the escalation logic has to be tuned per task.

Three rules keep routing from backfiring:

• Route per task type, decided up front. A fixed mapping from feature to model is easy to evaluate and keeps each session on one model.
• Hand off through subagents instead of switching. Claude Code sends search-heavy work to Haiku-based Explore agents with a short handoff message, so the main session's cache stays intact.
• Count retries. A cheaper model that fails 20% of the time and triggers a retry on the large model can cost more than sending everything to the large model.

Price per token also interacts with caching. In our Fable 5.1 vs Fable 5 runs, the newer model came out cheaper only on cache-heavy work, because its cache reads cost 75% less while its input and output prices stayed the same. Compare models on your own traffic shape before switching.

Both Anthropic's Message Batches API and OpenAI's Batch API charge 50% of standard prices in exchange for asynchronous processing within 24 hours. Anthropic accepts up to 100,000 requests per batch and says most finish within an hour. OpenAI accepts up to 50,000, and its Flex tier charges the same 50% rate for requests that can tolerate slower processing. Its Fast tier goes the other way, at 2x standard prices.

Anthropic states that batch and caching discounts stack. Nightly enrichment, evals, backfills, content classification, and research jobs that run on a schedule all qualify. Anything a user is waiting on does not.

Prompt compression removes low-information tokens from the input with a small model before the large model sees it. LLMLingua reports up to 20x compression with little performance loss, and LongLLMLingua reports a 21.4% accuracy gain on NaturalQuestions with about 4x fewer tokens, because the compressor also removes distractors.

Those numbers come from benchmark tasks. In production, compression adds a model call and latency, and the compressed text is harder to debug. It earns its place on long retrieved contexts that you cannot shrink at the source. If the tool that fetches the content can return only the answer or the fields you need, do that instead, because extraction at fetch time keeps the extra text out of your model's context entirely.

For retrieval pipelines, the safer form of compression is retrieving fewer, better chunks. Our guide to chunking strategies for RAG covers how chunk size and boundaries change what gets retrieved.

Take an eight-turn research agent running 10,000 tasks a month on Claude Sonnet 5.5. Each turn reads one page and writes a reply, on top of a 12,000-token static prefix. Prices are Anthropic's list prices from October 13, 2026.

Assumes every turn after the first reads the prior prefix from cache and writes only the new page and reply. Batch row applies only to jobs nobody is waiting on.

Fed raw HTML at 120,000 tokens per page instead, the same agent's LLM API cost would be $89,408 a month before any other change.

The first change cut the most because it shrank the layer that every later turn resends. Caching then cut the remaining bill by roughly 60% by changing the order of the prompt and leaving its content alone.

Agents need limits that a single API call does not, because an agent decides how many calls to make. One example from this year is a widely shared report of $1.3M in OpenAI tokens in 30 days on one developer's agent workflows. A commenter pointed out that the figure was raw API cost with fast mode on, and closer to $300k without it.

In the Bai et al. study, runs of the same task differed by up to 30x in total tokens, accuracy peaked at intermediate cost and flattened above it, and models asked to predict their own token usage beforehand underestimated it, with correlations no higher than 0.39. The budget has to come from outside the agent.

A September 2026 analysis of 1,200 Claude Code and Mini-SWE-Agent trajectories, Hu et al., found three repeat behaviors behind much of the waste: re-retrieving information the agent already had, generating near-duplicate scripts, and re-running the same tests. They showed up in 79-98% of tasks and accounted for up to 22.75% of task cost. Developer-written skills that told the agent how to handle those situations cut cost by up to 41.73%, about twice what agent-generated skills achieved.

The budgets that work in practice:

• Per-task token ceiling. Sum input and output from the usage fields as the agent runs, and end the task or escalate to a human when it passes the limit.
• Turn cap. Most tasks finish in a predictable number of turns. As a starting heuristic, flag tasks that run past three times the median and tune the threshold against task outcomes.
• Per-tool result size limit. Truncate or summarize any tool result above a threshold before it enters history, since it will be resent on every later turn.
• Cache hit rate floor. Alert when it drops. A sudden fall usually means someone added a dynamic value to the system prompt or changed the tool list.
• Cost per successful task. Divide spend by tasks that passed your checks, so a cheaper configuration that fails more often shows up as more expensive.

For the loop design side of this, loop engineering covers how to structure agent loops so they terminate.

Caching can lock in a bad context. Developer @irl_danB replied to Thariq's caching thread arguing that designing around the cache can mean keeping worse context to keep requests cheap, and that a cleaner context would mean fewer requests in the first place. Thariq's answer was that caching is what makes generous rate limits possible. My read is to cache the parts that are stable and correct, and clear the parts that are stale even when clearing costs a cache rebuild.

The other techniques have failure modes too:

• Lossy extraction drops facts. Any summary or extracted answer can miss or misread a value. Validate extracted values against a schema or a second source when they drive decisions.
• Over-compression lowers accuracy. Measure task success rate on the same eval set before and after every change. A 70% token cut that drops success from 95% to 80% usually costs more in retries and human review than it saves.
• Routing fails on the hard tail. The queries a small model gets wrong are often the ones that matter most. Route by task type with an eval per route.
• TTL mismatches cost money both ways. A 1-hour cache on one-shot calls pays a 2x write for nothing, and a 5-minute cache on a workload with ten-minute gaps rebuilds on every turn.

Most of the savings in this guide come from not paying full price for the same pages, schemas, and history on every turn. Send less of it, cache what has to repeat, and track cost per finished task.