KV cache
Also known as: key-value cache, KV caching, attention cache
The KV cache, short for key-value cache, is memory an AI language model uses to store calculations about earlier tokens in a conversation so it doesn't have to repeat them for every new word it writes.
It makes text generation far faster. While the KV cache is essential for speed, it grows with every token in the context and can use more GPU memory than the model itself, which is why long conversations are expensive to serve.
KV cache, explained at three levels
At home
When you write a story, you don't reread every earlier sentence before adding each new word; you keep the gist in your head. The KV cache is the AI's version of that. As a language model writes, it saves notes about every token so far, so each new word only requires work on that word, not reprocessing the entire conversation from the start.
At work
The KV cache, short for key-value cache, is GPU memory where a language model stores calculations about earlier tokens so it doesn't repeat them while generating each new token. It makes AI responses much faster. For a business running AI, it matters because the cache takes memory for every active conversation, which limits how many users each expensive GPU can serve at once.
At home
Inside a language model, each token gets turned into three sets of numbers: a query, a key, and a value. To choose the next word, the model compares the new token's query with the keys of every earlier token, then blends their values. Earlier tokens' keys and values never change, so the model saves them instead of recalculating. Each word in your chat adds a slice to that store, so long chats need more memory.
At work
Cache size per token is two (keys and values) times layers times key-value heads times head dimension times bytes per number. For a large model, that can reach hundreds of kilobytes per token, so a 100,000-token conversation may need tens of gigabytes. Serving engines such as vLLM manage it with paging, which stores the cache in fixed blocks to avoid wasted memory, eviction of idle conversations, and prefix sharing across requests, which is what powers prompt caching.
At home
On your own computer, the model weights and the KV cache share the same memory. A model that loads fine can still crash or slow to a crawl when you paste in a long document, because the cache for that context needs gigabytes of its own. Tools such as llama.cpp let you cap the context length or compress the cache to fit. In my opinion, that surprises many people trying local AI for the first time.
At work
Architectures shrink the cache directly: grouped-query attention shares keys and values across heads, DeepSeek's multi-head latent attention compresses them, and sliding-window attention keeps only recent tokens in some layers. Serving systems add KV quantization to 8 or 4 bits, offloading to CPU memory, and disaggregated serving, which runs prompt processing and token generation on separate GPUs and transfers the cache between them. Because cache memory usually limits concurrency more than compute does, I'd size GPUs for it explicitly.
Real-world examples
At home
Thomas ran an AI model on his laptop. Short chats were fine, but when he pasted in a long report, memory filled up because the KV cache grew with every token.
At work
Aisha is an ML engineer at a Vancouver AI startup. Switching to a serving engine with paged KV cache management let each GPU handle roughly twice as many users at once.
How it compares
| feature | KV cache | Prompt caching |
|---|---|---|
| What it stores | Attention data for tokens in one sequence | Reusable prefix across requests |
| Lifespan | During a request | Minutes, across requests |
| Who manages it | The serving engine | The provider, exposed as a feature |
| Relationship | Building block | Built on saved KV cache |
The KV cache speeds up one response. Prompt caching reuses it across many.
For analysts
When budgeting for self-hosted AI, I'd size GPU memory for the KV cache at your expected context length and user count, not just the model weights.
History & origin
The transformer architecture, introduced by Google researchers in 2017, computes 'keys' and 'values' for each token as part of attention. Because past tokens don't change while generating, systems began storing these results instead of recomputing them. As models and context windows grew, the cache became a major bottleneck. Noam Shazeer proposed multi-query attention in 2019, and Google researchers introduced grouped-query attention in 2023, both shrinking the cache by sharing keys and values. In 2023, Woosuk Kwon and colleagues at UC Berkeley introduced PagedAttention and the vLLM serving engine, managing the cache like computer memory pages and greatly improving throughput. DeepSeek's multi-head latent attention, used from 2024, compressed the cache further. In my view, many of the biggest serving-cost improvements came from fighting the KV cache, not from new models.