AI 101 — What is prompt caching?

Prompt caching is a way to make an AI model skip re-reading text it has already read. The provider saves its internal working state for the opening section of your prompt, and the next request that starts with the exact same text picks up from there — cheaper and faster.

It sounds like a developer-housekeeping detail. It is now one of the largest line items in how AI gets priced, and one of the most common ways a working product quietly runs up a bill.

Why it matters right now

On September 22, OpenAI halved the API prices of its GPT-6 Sol and Luna models, hours after Anthropic cut Opus 5.5 pricing by 40%. OpenAI attributed its cut to better prompt caching and cheaper inference rather than to smaller models, and cached input reads now carry a 90% discount. When two frontier labs are racing to cut prices on the same Tuesday, caching is a big part of where the money came from.

The other direction is uglier. A developer running OpenAI's Codex agent on Amazon Bedrock reported that 3,656 requests produced 171.94 million cache-write tokens and effectively zero cache hits, because the CLI could not opt into explicit prompt caching at all — cache writes were about 85% of the model's estimated spend, roughly $1,182 of $1,386 over four days. Nothing was broken. The agent simply paid full price to rebuild the same state on every turn.

That is the whole argument for understanding this one: caching turns re-reading from a cost you pay every time into a cost you pay once.

The mental model: read, then write

A model does two separate jobs per request, and only one of them gets cached.

First it reads your prompt — every token of the system instructions, tool definitions, and the documents or conversation history you pasted in. Then it writes the answer. For the model to read token number 40,000, it must keep intermediate state for tokens 1 through 39,999: the "key-value" states that let attention refer back to earlier text. Those states are the thing being cached.

Writing is priced higher per token, so it feels like the expensive half. But an agentic coding session sends enormous prefixes over and over — the same instruction block, the same tool list, the same growing conversation — and each repeat would otherwise rebuild all of that state from scratch. That is the waste caching removes.

Prefix matching is literal and unforgiving. The provider compares your new prompt against saved entries from the beginning and reuses the longest identical stretch it finds. Change one character early and everything after that point stops matching. Change something at the very end and you keep nearly all of it.

Prompt caching is the API-level product; underneath, it is the KV cache that determines how much a model can hold in context at once. Serving-side systems manage the same state by different rules — we covered the surprise that plain LRU beat three smarter KV-cache policies on 68,000 real agent requests — while the commercial APIs bill it as a discount.

The kitchen analogy

Picture a restaurant before the dinner rush. The cook does mise en place: buns toasted, lettuce washed, sauces portioned, the same prep every service. When an order lands, the cook assembles the burger in seconds because the slow work is already done.

Prompt caching is that prep counter. The stable front of your prompt — instructions, tools, a document you ask about again and again — gets prepped once and reused all shift. The changing part at the end is the order: that always has to be assembled fresh, and it is why a cache hit never makes the answer appear instantly.

Two failure modes follow from the analogy. If a customer swaps the bun, every bun on the counter is junk — that is what a changed prefix does. And if the kitchen preps a hundred burgers nobody orders, the prep cost more than cooking from scratch. Caching is not free; it is a bet that the same prefix comes back soon.

Common misconceptions

"It gives the model memory." No. The saved state is the provider's, not the model's, and it holds no knowledge of your other conversations. A cache hit also never changes the answer — the model computes the same thing, it just skips rebuilding state to do it.

"It's on by default, so I'm benefiting." OpenAI and Google both cache automatically on current models, and a hit still requires an identical prefix: same system prompt, same tool list, same order, no inserted timestamp. Developers lose most of their hits to a date string at the top of a system prompt.

"It makes responses faster." It shortens the wait before the first word — the reading phase. Generating the answer costs exactly what it did before. vLLM, the most widely used open serving engine, says this plainly about its own prefix caching: no gain when most time goes into writing the answer.

"Caching is free money." Writes can cost more than normal input. OpenAI charges 1.25× the uncached input rate to write on its newest models; Anthropic charges 1.25× for a five-minute cache and 2× for the hour-long one, against a 90% discount on hits. If your prefix never repeats, you have paid a premium for nothing.

"Any prefix works." There are minimum lengths before anything is cacheable at all — 1,024 tokens on GPT-5.6 and later, roughly 2,048 to 4,096 on recent Gemini models depending on size — and saved state lives on individual machines, so heavy traffic can route a request to a machine without your entry and miss.

Where to learn more

The vendor docs are unusually good here, and short. Anthropic's prompt caching page documents the two TTLs and what invalidates a cache. OpenAI's guide explains breakpoints, minimums, and machine routing. Google's context caching page covers its implicit mode. And if you run your own models, vLLM's automatic prefix caching page is the practical version of everything above.

The operational half is worth an hour if you own a budget: our guide to cutting an LLM bill without switching models puts caching first in order of payoff — and starts with measuring your cached-token share instead of assuming it.

Related reading: What is a context window? is the limit caching exists to work around · What is model quantization? is the other big lever on inference cost · and What is a token in AI? is what you are being billed per million of.

Have you ever found a caching bill that made no sense — writes with no hits, or a discount that never showed up? Tell us in the comments.

Sources: Anthropic — Prompt caching · OpenAI — Prompt caching · Google — Context caching · vLLM — Automatic Prefix Caching · openai/codex issue #37674 (GitHub)