What Can You Run on an RTX 5070 12GB?

The RTX 5070 is the card a lot of people actually own. It launched with 12GB of VRAM, and 12GB is the awkward middle of the local-LLM world: plenty for some models, not enough for others, and the line between them is not obvious.

Here is what fits, what almost fits, and what needs a workaround.

Your real budget is about 10.5GB

The card says 12GB, but the model does not get all of it. The GPU driver, the CUDA runtime, and the inference engine take roughly a gigabyte before you load anything. Plan on 10.5GB of usable space, and leave a little headroom on top of that for the KV cache, the scratch memory the model uses to remember the conversation.

A useful rule of thumb: at Q4 quantization, a model needs roughly half a gigabyte per billion parameters. At Q8, closer to a full gigabyte per billion. These are ballpark numbers, but they get you in the right neighborhood.

What fits comfortably

7-8B models at Q8. An 8B model at Q8 lands around 8.5GB, which leaves real room for context. Llama 3.1 8B and its many fine-tunes live here. This is the sweet spot for a 12GB card: good quality, fast tokens, room to breathe.

12B models at Q4. A 12B model at Q4 needs roughly 7GB for the weights. Gemma 3 12B and the Mistral Nemo class sit here. You get noticeably better reasoning than 8B models with context to spare.

14B models at Q4, at short context. Qwen3 14B at Q4 is roughly 8GB of weights. It fits, but the KV cache starts to matter. Keep the context window modest and it runs fine.

What does not fit

20-24B dense models at Q4. Mistral Small 24B at Q4 is around 14GB of weights. That is over budget before you generate a single token. No amount of clever prompting fixes physics.

32B and up, dense. A 32B model at Q4 needs about 18GB. Not close.

Big MoE models, even efficient ones. This is the trap. GLM-5.3-Flash has only 18B active parameters per token, which sounds small, but all 320B of its weights have to live in memory. At Q4 that is roughly 180GB. The active parameter count decides how fast it runs, not how much memory it needs. MoE giants are API-only for a 12GB card.

The workarounds

If you really want a bigger model, partial offloading is the honest option. Tools like llama.cpp can keep part of a 20B+ model in VRAM and spill the rest to system RAM. It works, but the CPU-side layers run much slower, so expect a real speed penalty. For chat it is tolerable. For anything latency-sensitive, it is not.

The other workaround is dropping to Q3 or Q2 quantization, which shrinks the weights further. The quality cost is real at that point: expect degraded reasoning and more hallucinations. For most people, a smaller model at Q4 beats a bigger model at Q2.

Context length is the silent second bill

Everything above assumed a normal conversation. Push to 64K or 128K tokens of context and the KV cache can eat several gigabytes on its own, which shrinks the weights budget. On a 12GB card, long context and big models do not mix. Pick one.

The takeaway

For a 12GB card, the practical ceiling is a 14B model at Q4 with moderate context. The comfortable zone is 8B at Q8 or 12B at Q4. That covers the models most people actually run day to day, and the quality jump from 8B to 14B is real.

Before you download anything, run your exact model and context length through a VRAM calculator. The ballpark rules in this post get you oriented; the calculator gives you the number before you spend an hour downloading weights that will not fit.