Prompt caching
Also known as: context caching, prompt cache, LLM prompt caching
Prompt caching is a feature of AI APIs that saves the processed version of the beginning of a prompt, such as long instructions, a big document, or tool definitions, so later requests that start the same way don't have to process it again.
Cached input is billed at a steep discount and returns faster. While prompt caching can cut costs dramatically for chatbots and agents, it only works when the repeated part stays exactly the same and comes first.
Prompt caching, explained at three levels
At home
Imagine a teacher who gives every class the same 20-minute safety briefing before a short new lesson. Prompt caching is like recording the briefing once and replaying it instantly, so only the new part takes real effort. AI apps often resend the same long instructions or documents with every question, and caching lets the AI provider skip reprocessing that repeated opening part.
At work
Prompt caching is a feature of AI APIs that stores the processed version of the repeated opening of a prompt, such as system instructions, tool definitions, or a long document. Later requests that start the same way reuse it, so cached input is billed at a steep discount, often 50 to 90 percent off, and responses start faster. For high-volume AI apps, it can cut bills substantially.
At home
Before an AI model writes a word, it processes every token of your prompt into internal data. If you upload a long manual and ask several questions, the app sends the manual again with each question. With caching, the provider keeps the processed manual for a while after your first question, so follow-ups only need your new question processed. That's why later answers can come back noticeably faster.
At work
Caching works on prompt prefixes: the cached portion must match exactly, token for token, from the very start. So structure prompts with stable content first, such as the system prompt, tool definitions, and reference documents, and changing content last, such as the user's newest message. Some providers, such as OpenAI, cache automatically above a minimum length, while others, such as Anthropic, let developers mark cache points explicitly. Caches expire after a time-to-live, often about five minutes, with longer options on some platforms.
At home
A cache only helps if the beginning of the prompt is identical each time. Put something that changes, like today's date or the user's name, at the very top, and every token after it no longer matches, so the whole prompt is processed and billed at full price again. While the fix is just moving that line to the end, many apps never notice. In my opinion, it's a good example of how small design choices can quietly double costs.
At work
Under the hood, providers save the KV cache, the attention data computed for the prefix, and reuse it when a new request matches. Cost optimization means measuring cache hit rates, ordering content by stability, and weighing time-to-live against cache-write charges, which some providers bill above the normal input rate. Agent loops benefit most, since each step resends the growing history; appending only at the end keeps the prefix valid. Providers differ on automatic versus explicit caching, minimum lengths, and pricing.
Real-world examples
At home
Lena uploaded a 200-page manual to an AI assistant and asked several questions. The first answer took a while, but later ones came back quickly because the manual was cached.
At work
Carlos manages the AI budget at a Toronto software company. Reordering their support bot's prompts so the long policy instructions came first raised cache hits and cut the monthly API bill by more than half.
How it compares
| feature | Prompt caching | Response caching |
|---|---|---|
| What's reused | Processed prompt prefix | A previous final answer |
| Answers | Freshly generated each time | Identical to before |
| Works when | Start of prompt matches | Whole request matches |
| Offered by | AI providers | Usually your own app |
Prompt caching saves work but still generates new answers. Response caching reuses the old answer.
For analysts
For any AI feature with long, repeated instructions, I'd ask the team to measure the cache hit rate. It's often the single cheapest optimization available.
History & origin
Language models process every token of a prompt before answering, so long system prompts and documents were paid for again on every request. Google introduced 'context caching' for Gemini in 2024, and Anthropic launched prompt caching for Claude in August 2024, advertising cost reductions of up to 90 percent and lower latency on long prompts. OpenAI added automatic prompt caching in October 2024 with a 50 percent discount on cached input. Caching became especially important for AI coding agents and multi-step agents, which resend large amounts of context on every step. In my view, prompt caching is one of the least glamorous and most valuable cost features in AI, and many teams leave money on the table by ignoring it.