KV cache
Also known as: key-value cache, KV caching, attention cache
The KV cache, short for key-value cache, is memory an AI language model uses to store calculations about earlier tokens in a conversation so it doesn't have to repeat them for every new word it writes.
It makes text generation far faster. While the KV cache is essential for speed, it grows with every token in the context and can use more GPU memory than the model itself, which is why long conversations are expensive to serve.
KV cache, explained at three levels
At home
When writing a story, you don't reread the whole thing before each new word; you remember what came before. The KV cache is the AI's way of remembering so it can keep writing quickly.
At work
The KV cache makes AI responses faster, but it uses memory that affects how many users a server can handle at once.
At home
Each word in a conversation adds a bit to the cache. Very long chats mean a bigger cache, which can slow things down or cost more.
At work
Cache size depends on layers, attention heads, head dimension, context length, and batch size. Serving engines manage it with paging, eviction, and prefix sharing, which also powers prompt caching across requests.
At home
When running AI on your own computer, a long context can run out of memory even if the model fits. In my opinion, that surprises many people trying local AI for the first time.
At work
Techniques to reduce cache cost include grouped-query and latent attention, KV cache quantization, sliding-window attention, offloading to CPU memory, and disaggregated serving that separates prompt processing from token generation. Cache memory often limits concurrency more than compute does.
Real-world examples
At home
Thomas ran an AI model on his laptop. Short chats were fine, but when he pasted in a long report, memory filled up because the KV cache grew with every token.
At work
Aisha is an ML engineer at a Vancouver AI startup. Switching to a serving engine with paged KV cache management let each GPU handle roughly twice as many users at once.
How it compares
| feature | KV cache | Prompt caching |
|---|---|---|
| What it stores | Attention data for tokens in one sequence | Reusable prefix across requests |
| Lifespan | During a request | Minutes, across requests |
| Who manages it | The serving engine | The provider, exposed as a feature |
| Relationship | Building block | Built on saved KV cache |
The KV cache speeds up one response. Prompt caching reuses it across many.
For analysts
When budgeting for self-hosted AI, I'd size GPU memory for the KV cache at your expected context length and user count, not just the model weights.
History & origin
The transformer architecture, introduced by Google researchers in 2017, computes 'keys' and 'values' for each token as part of attention. Because past tokens don't change while generating, systems began storing these results instead of recomputing them. As models and context windows grew, the cache became a major bottleneck. Noam Shazeer proposed multi-query attention in 2019, and Google researchers introduced grouped-query attention in 2023, both shrinking the cache by sharing keys and values. In 2023, Woosuk Kwon and colleagues at UC Berkeley introduced PagedAttention and the vLLM serving engine, managing the cache like computer memory pages and greatly improving throughput. DeepSeek's multi-head latent attention, used from 2024, compressed the cache further. In my view, many of the biggest serving-cost improvements came from fighting the KV cache, not from new models.