Why a 1M-Token Context Changes Your VRAM Math

Why a 1M-Token Context Changes Your VRAM Math

When a model ships with a million-token context window, the headline number everyone talks about is parameters. Mistral Large 4 is a 1.05-trillion-parameter mixture-of-experts model with about 49 billion active parameters per token and a 1M-token context. The weight math is straightforward. The part that surprises people is the second bill: the memory for the context itself.

That second bill is the KV cache. When a model generates text, it computes a key and a value for every token it has seen, then stores them so it does not recompute them for the next token. That cache lives in VRAM right next to the weights, and it grows with every token in your context window. It is the reason a model that fits on your GPU at short context can run out of memory at long context.

The formula is simple. For each token, the cache holds 2 (keys plus values) times the layer count times the number of KV heads times the head size times the bytes per value. Take a typical 70B-class model: 80 layers, 8 KV heads, a head size of 128, running in 16-bit. That works out to about 320 kilobytes per token. Multiply by a million tokens and you get roughly 320 gigabytes of cache. Just for the context. The weights are extra.

Read that number again. A model whose weights fit on two 80GB cards can still be completely unusable if you actually fill its context window. Most people size their GPU for the weights and meet the KV cache at load time, ten minutes into a run, in the form of an out-of-memory error. Then they learn that context length is a hardware question, not just a capability question.

It scales down in a straight line, which helps with planning. That same 70B-class model at a 128K context needs about 40 gigabytes of cache. Still serious, but a very different machine than the 320GB that a full 1M-token window demands. A smaller 8B-class model (32 layers, 8 KV heads) costs about 128 kilobytes per token, so 128K of context takes roughly 16 gigabytes of cache. You can see why long-document work on a laptop tops out fast.

This is also why the context numbers in announcements deserve a second look. A 1M-token window is a real capability, but it assumes you can feed it. If your actual work is 50-page PDFs and long chat histories, you will never touch the full window, and you should size the machine for the context you use, not the context on the spec sheet.

You have levers when the math does not fit. The blunt one is a shorter context: cap the window at what your documents actually need instead of running the maximum. Quantizing the cache to 8 bits cuts it in half, and most inference engines support that with a small quality cost. Some architectures use sliding-window attention, so old tokens age out of the cache automatically. And offloading part of the cache to CPU RAM works when you can tolerate slower generation.

There is one more subtlety worth knowing. The cache only grows with the tokens you actually feed in, not the maximum the model supports. Setting a 1M-token limit costs nothing by itself. The bill arrives when you fill it. That means a machine that handles your 200K-token legal documents fine today will not suddenly need an upgrade because the next model release advertises a bigger window.

The practical move is to do both halves of this math before you buy or rent the GPU. Estimate the weights, estimate the cache for the context size you really use, add a margin, and then pick the card. The calculator exists for exactly this reason: here is the VRAM calculator. Plug in your model and your real context length and see both bills before you spend anything.

border="0" width="400" data-original-height="1280" data-original-width="1920" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgchMC99nf5togs7VuBjr7NlefQMhIbIfQR9Pt51AgYZKcr1ypAh_pZqYDNUs44PMe9AM-TkzAXKW1bdco9NPAgEr7_Qhyvsrkz9FMQ7BpRrmjbnF0EOw3NIMj2g1Yr_XTQ4TJO8pm_UyTY807cc5d4RJ1vXZO3JugHLHPDsc7kmhAE6fpG0ZBNDczry4k/s400/2026-10-11-runopensourcellm.jpg"/>