Run a 70B LLM on a 24GB GPU With Offloading

A 70-billion-parameter model at 4-bit quantization needs about 40 gigabytes of memory. A 24-gigabyte card cannot hold it. But many people run 70B models on 24GB cards anyway, thanks to offloading: the runtime keeps some layers of the model on the graphics card and parks the rest in system RAM, shuttling weights back and forth as it generates text.

The idea is simple once you see how inference works. A transformer model is a stack of identical layers. To generate each token, the computation passes through every layer in order. So you do not need the whole model in video memory at the same instant; you only need the layer you are computing right now, plus the key-value cache for the conversation. Tools like llama.cpp exploit this by letting you choose how many layers live on the graphics card (GPU layers) and how many stay in system memory.

The practical effect is a sliding scale. Push more layers onto the card and generation gets faster. Pull layers back into RAM and speed drops, because each layer parked in RAM has to travel over the PCIe bus for every token. A full 70B model at 4-bit quantization might generate 15 to 20 tokens per second entirely on a 24GB card if it fit (it does not). With half the layers offloaded to RAM, expect something like 6 to 10 tokens per second. That is slow for writing a novel, but perfectly usable for chat, coding help, or drafting emails.

System RAM becomes the new constraint. The layers you offload still live somewhere, and that somewhere is your main memory. A 70B model at 4-bit quantization needs roughly 40GB total, so if your card holds 24GB, the remaining 16GB plus overhead has to sit in RAM. In practice you want 64GB of system RAM for comfortable 70B offloading: the model, the cache, your operating system, and your other apps all need room. This is why Macs with unified memory have an odd advantage here. On Apple Silicon, the CPU and GPU share one memory pool, so "offloading" is closer to a rename than a real transfer.

The setup is straightforward in llama.cpp. You load the model file, set the number of GPU layers to something your card can hold (for a 24GB card and a 70B 4-bit model, try 40 to 50 layers out of roughly 80), and the rest runs on the CPU. Ollama does partial offloading automatically and tends to fill the card as far as it can.

There are tradeoffs worth knowing before you commit. First, speed depends heavily on your PCIe generation and your CPU. An old CPU doing the layer math becomes the bottleneck, and offloading stops helping. Second, the key-value cache for long conversations also competes for card memory, so a long chat with lots of offloading gets slower as context grows. Third, quantization matters: an 8-bit model is twice the size and offloads twice as much, so it pays to start at 4-bit quantization (Q4) for big models.

So when does offloading beat the alternatives? If your choice is between running a 70B model slowly or running a 13B model fast, try the 70B with offloading first. The bigger model is usually smarter even at half speed. But if you need interactive speeds for real-time work, or your system RAM is under 32GB, you are better off with a model that fits the card whole: a 14B to 24B model at 4-bit quantization will feel much snappier and often does the job.

One more practical tip: measure before you tune. Change the GPU layer count in steps and watch your tokens per second. There is usually a sweet spot where adding more layers stops helping because the CPU side or the memory bus becomes the limit.

If you are sizing hardware for a model, check your numbers before you buy. The VRAM calculator shows what fits on your card at each quantization level, including the headroom offloading needs for the key-value cache, so you can plan the split between card and RAM instead of guessing.