The KV Cache Tax: Why Long Context Eats Your VRAM

There are two bills for running a language model, and most people only see one. The first is the weights: the model file itself, fixed in size, easy to plan for. The second grows every time you use the model. It is called the KV cache, and it is the reason a 7B model that fits comfortably on your 8 GB card suddenly runs out of memory the moment you paste in a long document.

Understanding this second bill changes how you pick a model, how much context you can afford, and whether that shiny 128K context window on the model card is something you can actually use.

What the cache actually is

When a model reads your prompt, it converts every token into two sets of numbers, called keys and values, at every layer. It then stores them instead of recomputing them for each new token it generates. That stored state is the KV cache. Without it, generating each token would require re-reading the entire prompt from scratch, which would be unusably slow. With it, the model pays in memory instead of time.

The important property: the cache grows with every token in the context window. The weights stay the same size no matter what you do. The cache does not.

The back-of-the-envelope math

For each token, the cache stores two numbers (a key and a value) per attention head dimension, per layer, in some number of bytes. Written out: 2 times the number of layers times the hidden size times the bytes per number. At the usual 16-bit precision, a classic 7B-style model with 32 layers and a hidden size of 4096 needs about half a megabyte per token.

Multiply that out. Four thousand tokens of context costs roughly 2 GB. Thirty-two thousand tokens costs roughly 16 GB. That is more than the Q4-quantized weights themselves, which sit around 4 GB. The context can dwarf the model.

Modern models often do better than this. Many use grouped-query attention, which shares keys and values across head groups and can cut the cache to a quarter of the naive size. Some runtimes also compress the cache to 8-bit. So treat the formula as an upper bound and check your specific model and runtime. But the scaling law holds regardless: double the context, double the cache.

Why this surprises people

Model cards and benchmark posts usually describe memory in the best case: the weights plus a short prompt. Nobody advertises the 32K-context number, because it is worse. So a reader sees "runs on 8 GB" and reasonably assumes any prompt fits in 8 GB. Then they load a 40-page PDF into a retrieval pipeline, or hand an agent a long conversation history, and the run dies with an out-of-memory error.

The failure mode is predictable. Short chats are fine. Medium documents get sluggish as the system starts shuffling. Long documents crash. If your setup works for chat but dies on documents, the KV cache is the first suspect, not the model size.

Spend the context budget on purpose

Once you see the cache as a budget, the decisions get simpler. First, know your two numbers: the model's layer count and key-value size, which the model card usually lists, and your runtime's cache precision. Second, decide how much context you actually need. Most retrieval setups do fine at 8K or 16K; the full 128K window is for the days you truly need it, and on consumer hardware those days come with a smaller model or a shorter leash.

Third, remember the tradeoff this creates between model size and context length. A 14B model with 8K of context can need less total memory than a 7B model with 64K of context. Bigger is not always heavier. The weights-versus-cache split is what decides.

Before you commit to a model and a context length, run the numbers through the VRAM calculator on llmhardware.io. It accounts for weights, quantization, and context together, so you find out about the KV cache tax before the out-of-memory error finds you.