Prompt Caching: How It Works, What It Costs, and How to Keep the Cache Hitting

Caching is a prefix match, so put everything that never changes at the start of the request and everything that changes at the end. Then measure your hit rate per request and alert when it drops.

In April 2026, the AI gateway Requesty measured prompt cache hit rates across its traffic. Requests to Claude through Anthropic's own API read 77% of their input tokens from cache. Requests to Claude through Google's Vertex read 24%. The models were the same, and the gateway called the gap a configuration problem.

Anthropic's Claude Code team alerts on cache hit rate and treats a drop as an incident, and OpenAI shipped a dashboard and a miss-diagnostics field for it in September 2026.

It is often said in engineering that 'Cache Rules Everything Around Me', and the same rule holds for agents.

— Thariq Shihipar, member of the Claude Code team at Anthropic

For the broader cost picture, LLM token optimization covers caching alongside eight other techniques.

Prompt caching is an LLM API feature that stores the processed state of the beginning of a prompt so later requests starting with the same tokens skip reprocessing them. Cached tokens cost between 0.025x and 0.1x the normal input price on current Anthropic and OpenAI models, and responses start sooner because the model has less input to process.

Model APIs are stateless. Every turn of a chat or agent loop resends the system prompt, the tool definitions, and the whole conversation so far, and the model processes all of it again. In an April 2026 study of eight frontier models on SWE-bench Verified, Longju Bai and colleagues found agentic coding tasks consumed 1,000x more tokens than code chat, with input tokens driving most of the cost.

Prompt caching never changes the output. Anthropic's docs state the response is identical to what you would get without caching.

An LLM handles a request in two phases:

• Prefill reads the whole input in parallel and, at every layer, computes a key vector and a value vector for every token.
• Decode then generates the output one token at a time, and each new token attends to the keys and values of everything before it.

Those stored key and value vectors are the KV cache.

Within a single request, the KV cache saves the model from recomputing earlier tokens at each decode step. Prompt caching extends that across requests. The provider keeps the KV tensors for a prompt prefix after the request ends, and reuses them when the next request starts the same way. OpenAI's guide states that the cache stores KV tensors rather than the tokens.

Reuse only works for a prefix because attention in these models is causal. The key and value vectors for token 500 depend on tokens 1 through 499 and nothing after. Change token 12, and the vectors for every token from 12 onward come out different, so the saved state is useless from that point. Change only the last message, and everything before it is still valid.

Providers implement the lookup with hashes. On Anthropic, marking a block with cache_control writes one cache entry keyed by a cumulative hash of everything up to that block. On the next request, the API computes the hash at your breakpoint, and if there is no match it walks backward up to 20 blocks looking for an entry an earlier request wrote.

A three-turn conversation with Anthropic's automatic caching, which moves the breakpoint to the last block on every request, looks like this:

Request 1:  [system + tools][user 1]                                 -> write all
Request 2:  [system + tools][user 1][asst 1][user 2]                 -> read through user 1, write asst 1 + user 2
Request 3:  [system + tools][user 1][asst 1][user 2][asst 2][user 3] -> read through user 2, write asst 2 + user 3

Self-hosted inference engines do the same thing in memory. vLLM's automatic prefix caching hashes fixed-size blocks of KV cache so identical prefixes share blocks, and SGLang's RadixAttention keeps cached prefixes in a radix tree so requests that share a beginning share the work.

Semantic caching helps when many users ask nearly the same question. Agents rarely do, because each request carries its own history, so prefix caching does almost all the work there.

The three major APIs all cache by prefix, but they differ in who places the breakpoints, how long entries live, and whether a write costs extra:

Sources: Anthropic prompt caching docs, OpenAI prompt caching guide, and Gemini context caching docs, all read October 18, 2026.

Claude prompt caching is opt-in. The quickest setup is automatic caching, launched February 19, 2026: add one top-level cache_control field, and the API caches everything up to the last cacheable block and moves that point forward as the conversation grows.

import anthropic
 
client = anthropic.Anthropic()
 
response = client.messages.create(
    model="claude-sonnet-5-5",
    max_tokens=1024,
    cache_control={"type": "ephemeral"},  # automatic caching
    system="You are a support agent for Acme. <policies>...</policies>",
    messages=[{"role": "user", "content": "Can I get a refund after 30 days?"}],
)
 
u = response.usage
print(u.cache_creation_input_tokens, u.cache_read_input_tokens, u.input_tokens)

Explicit breakpoints go on individual blocks when sections change at different rates, for example tools that never change, a document that changes daily, and a conversation that changes every turn. You get four breakpoints per request, and the automatic one uses a slot. Cache prefixes build in a fixed order, tools, then system, then messages, and a change at one level invalidates that level and everything after it.

Three Anthropic details trip people up:

• Lifetime counts from the start of the request. The docs give the example of a response that takes 4 minutes to stream: the follow-up must start within about 1 minute of it finishing to hit the 5-minute cache.
• Parallel requests don't share a cache until the first response starts. If you fan out ten requests with the same prefix at once, all ten write. Send one, wait for it to start responding, then send the rest.
• Some settings live inside the cached prefix. Changing tool definitions invalidates everything. Toggling web search, citations, or fast mode invalidates the system and message caches. Changing tool_choice, images, the thinking configuration, or the effort setting invalidates message blocks.

Mid-conversation system messages let you send role: "system" updates inside messages without rewriting the cached prefix, and the September 2026 release notes list betas that do the same for per-message effort changes and for defining new tools inside a message.

OpenAI prompt caching is on by default. On GPT-5.6 and later, implicit mode places a breakpoint at the end of the latest eligible message, and explicit mode lets you mark breakpoints yourself, with up to four cache writes per request. Content after your last explicit breakpoint is billed at the plain input rate with no write charge, which is useful for content you don't expect to reuse.

from openai import OpenAI
 
client = OpenAI()
 
response = client.responses.create(
    model="gpt-6.1-sol",
    prompt_cache_options={"mode": "explicit"},
    input=[
        {
            "role": "developer",
            "content": [{
                "type": "input_text",
                "text": SYSTEM_PROMPT_AND_REFERENCE_DOCS,
                "prompt_cache_breakpoint": {"mode": "explicit"},
            }],
        },
        {"role": "user", "content": "Summarize the refund policy."},
    ],
)
 
print(response.usage.input_tokens_details.cached_tokens)

Cached state lives on individual machines, and traffic above 15 requests per minute can overflow to machines that don't hold the entry. On models before GPT-5.6, a stable prompt_cache_key such as a user or session ID helps route related requests to the same cache; on GPT-5.6 and later, OpenAI routes automatically and the key is only for separating cache accounting.

OpenAI's September 22, 2026 announcement also added a way to change reasoning effort mid-conversation without breaking the cache: append a configuration_update input item and leave the request-level reasoning.effort unchanged. To restrict tools for one turn, use allowed_tools or tool_choice: "none" instead of removing definitions.

Gemini's API caches implicitly for Gemini 2.5 and newer models, in both stateful and stateless conversations, with no configuration. Google's advice is to put large, common content at the start of the prompt and send requests with similar prefixes close together. Cache hits appear in usage.total_cached_tokens.

Gemini's higher minimum (4,096 tokens on current models) means short prompts never cache, and implicit caching makes no hit guarantee.

xAI also discounts cached input. Its September 2026 release notes list Grok 4.7 at $2 input and $0.50 cached input per million tokens, so a cached token costs a quarter of a fresh one.

These are list prices per million tokens from Anthropic's caching docs and OpenAI's pricing page on October 18, 2026:

Standard short-context rates. OpenAI's write column is its single cache-write rate.

Read prices have been falling. Anthropic cut cache reads on Fable 5.1 to 0.025x at launch on September 1, 2026, and on October 7 halved Sonnet 5.5 reads from $0.20 to $0.10. Write prices stayed the same, so writes now carry a larger share of the bill. In our Fable 5.1 vs Fable 5 runs, the newer model only came out cheaper on cache-heavy work for this reason.

Think of a cached prefix in multiples of its normal input cost. With a write premium of w and a read price of r, caching the prefix and reusing it n times costs w + n·r, against 1 + n without caching.

The 1-hour tier only pays when gaps between requests are longer than 5 minutes, and only if the prefix is reused at least twice. If requests come more often than every 5 minutes, the 5-minute cache refreshes itself for free on each hit.

A 5-minute cache on requests that arrive more than 5 minutes apart is a loss. Each request misses, writes again at 1.25x, and you pay 25% more than you would with no caching at all.

BenchLM ran this as a live test on September 14, 2026, sending 87 calls each to Claude Sonnet 5 and GPT-5.6 Terra through OpenRouter with a shared long system prompt. The first Sonnet call cost 24.7% more than the uncached equivalent, the second cost 89.1% less, and input spend across all 87 calls fell 87.8%, with 97.9% of input tokens read from cache.

Take a support agent on Claude Sonnet 5.5 with a 20,000-token static prefix (system prompt plus tool definitions). Each of its 10 turns adds 3,000 tokens of user message and tool results and writes a 500-token reply. Over the session, it processes 387,500 input tokens.

Anthropic list prices, October 18, 2026. Output is $0.05 of each total. With a stable prefix, 85.9% of input tokens are read from cache.

In the third row, a timestamp at the top of the system prompt changes the hash on every request, so every turn writes the full context at 1.25x and nothing is ever read. The API returns no error when this happens.

On LangChain's Deep Agents eval suite in June 2026, prompt caching cut token cost by 77% on Claude Haiku 4.5 using explicit breakpoints, 80% on GPT-5.4 mini with OpenAI's automatic caching, and 49% on Gemini 3.5 Flash with implicit caching. The longer the trajectory, the more the cached prefix gets reused.

Claude Code orders its requests as static system prompt and tools (shared across all users), then project context such as CLAUDE.md (shared within a project), then session context, then messages. The Claude Code caching docs describe the same three layers: a change to the conversation leaves the system prompt and project context cached, and a change to the system prompt invalidates everything.

For any app, the order is: tools, instructions, few-shot examples and reference documents, per-user context, conversation history, the new message.

Timestamps, request IDs, user names, and feature flags at the top of a prompt break the cache on every request. Pass them in the latest user message or tool result instead. Claude Code adds a <system-reminder> tag to the next message with updates like the current date rather than editing the system prompt.

On Anthropic, also place your breakpoint on the last block that stays identical. If you put cache_control on a block that includes the changing value, the docs explain, the API writes a new entry at that block on every request and the backward lookup never finds a prior write at an earlier position.

The cache matches bytes, so two JSON objects with the same content in a different key order are different prefixes. Anthropic's troubleshooting list calls out languages like Swift and Go that randomize key order during JSON conversion. Sort keys when you build tool schemas and tool results, and don't regenerate tool descriptions from templates that can change between requests.

Tool definitions sit at the very start of the prefix, so adding, removing, or reordering one invalidates the entire cache. Claude Code keeps every tool loaded and models state changes as tool calls instead. Turning on plan mode calls an EnterPlanMode tool and adds a system message, and the tool definitions stay the same.

For large tool catalogs, defer instead of removing. Anthropic's tool search sends lightweight stubs and loads a full schema only when the model asks for it, so the stubs stay the same on every request. On OpenAI, keep the full tools list and narrow it per turn with allowed_tools.

Adding web access mid-session has the same cost. Unless your harness defers tool loading, connecting an MCP server adds tool definitions and invalidates the cache, while a CLI the agent runs through its existing shell tool, such as the Firecrawl CLI, adds none. We compared the two approaches in MCP vs CLI for AI agents.

Each model has its own cache. Thariq's example was that 100,000 tokens into an Opus conversation, asking Haiku an easy question costs more than letting Opus answer, because Haiku has to rebuild the cache.

At Anthropic's October 2026 rates, the answer depends on how long the context is, because Haiku 5.5 charges more for prompts over 100,000 tokens:

Even when Haiku is cheaper, it has to prefill the whole context before it answers, so the first token arrives later. When you do need a cheaper model, hand it a short summary through a subagent rather than switching the main session.

Effort and thinking settings are rendered into the prompt on many models, so changing them mid-session can invalidate the cache. The Claude Code docs note that on Opus 5.5, Sonnet 5.5, Haiku 5.5, and Fable 5.1, with an API key or a Claude subscription, the cache stays intact by default when effort changes. On other models, or through your own API calls, use the provider's mid-conversation mechanism: OpenAI's configuration_update item or Anthropic's per-message effort beta.

Compaction, summarization, and background checks are separate API calls. Built the simple way, with a different system prompt and no tools, they miss the cache and pay full price for the whole conversation. Claude Code's compaction reuses the parent's exact system prompt, tools, and history, then appends the summary instruction as a new user message, so only that instruction is new.

Measure the time between turns in your own logs. If most gaps are under 5 minutes, use the 5-minute cache. If users or agents routinely pause for 5 to 60 minutes, the 1-hour tier saves a full rewrite each time.

On Hacker News in April, Boris Cherny of the Claude Code team explained that returning to a session after more than an hour idle is a full cache miss on every message, and that a 900,000-token context would write more than 900,000 tokens to cache in one turn. A reply in the 942-point thread objected to calling that a corner case, since leaving sessions idle over lunch is normal. If your product lets users resume old sessions, warn them or compact before resuming.

The first request after a deploy or a cache expiry pays the full write and the slowest time to first token. Anthropic lets you send a request with max_tokens: 0, which writes the cache at your breakpoints and returns immediately with no output. OpenAI added prewarming in September as well. Fire one at startup or on a schedule shorter than the TTL, with the same thinking and effort settings your real requests use.

The Claude Code docs describe three gateway behaviors: forwarding markers unchanged, rejecting them with a 400, or silently removing them, in which case the entire conversation bills as uncached input on every turn. Converting block-form system content to a plain string drops the marker the same way. A stripped marker could explain part of the Vertex gap in Requesty's data.

Model routers have the same problem. In a June 2026 Show HN for a model router, commenters pointed out that every switch is a fresh cache miss, and one described handing plans to a single model that held 99.9% cache hits over a long session.

Append fetched pages as tool results at the end of the conversation and leave them there. Re-fetching a page and splicing the new copy into an earlier position changes the prefix from that point on, and live pages change between loads.

To measure how often, we scraped eight pages with Firecrawl twice, 90 seconds apart, with its page cache turned off (maxAge: 0), then twice more with the cache on:

Firecrawl scrape with onlyMainContent, October 18, 2026. Cached runs used maxAge of one day.

Firecrawl stores scraped pages and, by default, returns a copy up to two days old instead of scraping again. When the same page has to go back into a prompt, for example when several workers build the same cached document block or a restarted process rebuilds it, a non-zero maxAge gives every caller the same bytes and so the same prefix hash. Keep maxAge: 0 for content that has to be live, like prices.

Put the page content in the prompt and leave out the rest of the API response. Each Firecrawl response carries a unique scrapeId in its metadata, so a prompt built from the raw JSON changes on every call even when the page has not.

Keep each result small. Cache reads are cheap, but every new token is written once at 1.25x or 2x and then read on every later turn. In an August 2026 analysis of 63,880 billed Claude Code requests, cache writes carried 76% of the cache-input cost inside a typical session, even with an aggregate hit rate of 96.59%. In our Claude Code web research test, a stock-price lookup through Firecrawl's question format came back as 6 characters, against 295 characters from Claude Code's fetch tool for the same fact.

For document Q&A, load a whole documentation site into the prefix once, cache it, and ask many questions against it:

import anthropic
from firecrawl import Firecrawl
 
firecrawl = Firecrawl(api_key="fc-YOUR-API-KEY")
claude = anthropic.Anthropic()
 
# Crawl the docs once and join the pages as clean markdown
crawl = firecrawl.crawl(
    "https://docs.example.com",
    limit=50,
    scrape_options={"formats": ["markdown"], "only_main_content": True, "max_age": 86_400_000},
)
docs = "\n\n".join(page.markdown for page in crawl.data if page.markdown)
 
def ask(question: str) -> str:
    response = claude.messages.create(
        model="claude-sonnet-5-5",
        max_tokens=1024,
        system=[
            {"type": "text", "text": "Answer only from the documentation below."},
            {
                "type": "text",
                "text": f"<docs>\n{docs}\n</docs>",
                "cache_control": {"type": "ephemeral", "ttl": "1h"},
            },
        ],
        messages=[{"role": "user", "content": question}],
    )
    return response.content[0].text
 
print(ask("How do I rotate an API key?"))
print(ask("What are the rate limits on the free plan?"))  # reads the docs from cache

The crawl output stays fixed for the life of the cache because you build docs once and reuse the same string, and max_age (one day here) means a restart or a second worker rebuilds the same bytes. Clean markdown also keeps the cached block small. In our tokenization benchmark, fifteen pages came to 4.1 million tokens as raw HTML and 190,817 tokens as markdown. The 1-hour TTL fits a docs assistant where questions arrive minutes apart.

Cache hit rate is cached input tokens divided by total input tokens, computed over a window of requests. Every provider returns the numbers on each response:

• Anthropic: cache_read_input_tokens / (cache_read_input_tokens + cache_creation_input_tokens + input_tokens). Anthropic's input_tokens counts only tokens after the last breakpoint, so all three belong in the denominator.
• OpenAI: usage.input_tokens_details.cached_tokens / usage.input_tokens, or the Prompt Caching Dashboard.
• Gemini: usage.total_cached_tokens against total input tokens.

Track the median per session as well as the aggregate. In the 63,880-request Claude Code analysis, the aggregate was 96.59% and the median session 86.41%, because a few very long sessions with near-perfect reuse pulled the total up.

When the rate drops, both major providers now tell you why:

• Anthropic cache diagnostics, generally available since September 23, 2026: include a diagnostics object with the previous response's ID, and the API returns a cache_miss_reason saying whether the model, system prompt, tools, or message history diverged. Fingerprints contain only hashes and token-count estimates.
• OpenAI prompt cache diagnostics: compares a request with a recent response and returns a reason such as tools_changed with the number of affected tokens.

Teams quoted in OpenAI's GPT-6 caching announcement reported large changes from small fixes. Manus said its OpenAI cache hit rate went from about 85% to consistently above 90% in under a week. Wordsmith moved its session agents to explicit breakpoints and saw hit rates rise from 83% to 91%, cache writes fall by about two-thirds, and inference costs drop 36%. Strawberry Browser said a few percentage points of hit rate cut its costs 20%. GitHub said it had cut the share of Copilot prompt tokens needing fresh processing by more than 50%.

In a July 2026 proxy study that reached 706 points on Hacker News, OpenCode's request prefix was byte-identical in every captured run, while Claude Code rewrote tens of thousands of cache tokens mid-session and wrote up to 54x more cache tokens than OpenCode on the same task. Both harnesses set breakpoints correctly, so the gap came from how stable each kept its prefix.

Provider caches require exact prefixes, and much of this year's research is about loosening that requirement for agents, whose prompts shift in ways a prefix cache can't reuse:

• Cache-aware context management. TokenPilot (June 2026) points out that pruning and evicting context in agents changes the prompt layout and causes prefix mismatches, so token savings trade off against cache continuity. Its approach stabilizes the prefix and evicts in conservative batches, and reduced costs by 56-87% on PinchBench and Claw-Eval.
• Caching tools independently of order. Agents send tool and skill schemas in different combinations and orders, which defeats prefix caching. ReCache (August 2026) caches each schema separately with its own positions, matching dense performance (82.3% vs 82.4% invocation F1) with a 3.655x faster time to first token.
• Position-independent caching. Irminsul (May 2026) notes that agent workloads put identical tokens at shifted positions each turn, voiding prefix caches. Keying the cache by content instead of position recovered up to about 83% of prompt tokens beyond exact-prefix matching on agentic traffic.
• Lifecycle-aware eviction. KVTether (September 2026) tells the serving layer which messages an agent has discarded or will reuse soon, lowering estimated task cost by 33-40% against LMCache and MORI.
• Reuse is not always free of side effects. Request Order Matters (October 2026) shows that non-prefix reuse with selective recomputation can make an unchanged prompt give different answers depending on earlier requests, with answer variation across orderings as high as 69%. The paper studies non-prefix reuse, which is a different technique from the exact prefix caching providers sell today.
• Why the discount exists. "Can I Buy Your KV Cache?" (June 2026) measured reuse at 9-50x cheaper in compute than prefill on Qwen3-4B, and argues the 0.1x cache-read price sits comfortably inside that range.

• Short prompts. Below 512 to 4,096 tokens, depending on provider and model, nothing caches.
• One-shot requests with unique context. If nothing is reused within the TTL, a write premium is a pure cost.
• Sparse traffic. Requests further apart than the cache lifetime miss every time, and on providers with a write premium they cost more than no caching.
• Prompts that must change at the top. Per-user system prompts can only share a cache within that user's requests. Move per-user content below the shared instructions so the shared part still caches across users.
• Output-heavy workloads. Caching only discounts input. If output dominates your bill, cap output and reasoning first.

Designing around the cache can also go too far, if it means keeping a worse context because it is cheap to resend. Cache the parts that are stable and correct, and clear stale tool results even when that costs a rebuild. Context engineering vs prompt engineering covers deciding what belongs in context in the first place.