Memory Bandwidth: The Real Bottleneck for LLM Inference

When people shop for AI hardware, they fixate on VRAM size. But the metric that actually determines how fast your LLM generates tokens is memory bandwidth — the rate at which data flows between VRAM and the GPU's compute cores. Understanding this changes how you shop for hardware.

Key insight: LLM inference is memory-bandwidth-bound, not compute-bound. A GPU with 2x the bandwidth runs LLMs roughly 2x faster, even if its FLOPS rating is lower. LLMHardware.io tracks bandwidth specs for every GPU alongside real-world tokens/second benchmarks.

Why Bandwidth Matters More Than FLOPS

During token generation (the autoregressive decode phase), the GPU must load the entire model's weight matrices from VRAM for every single token. A 7B parameter model in FP16 requires loading 14GB of data per token. At 500 GB/s bandwidth, that takes 28ms — a hard ceiling of ~35 tokens/second regardless of compute power.

Bandwidth by GPU Class

Consumer GPUs range from ~288 GB/s (RTX 4060) to 1,008 GB/s (RTX 4090). Apple Silicon unifies CPU and GPU memory — the M3 Max gets 400 GB/s across all 96GB, while the M4 Ultra hits 800 GB/s. Professional cards like the H100 SXM reach 3,350 GB/s, explaining why data center inference costs have plummeted.

Optimizing for Bandwidth

Quantization is the primary lever. Reducing from FP16 to Q4_K_M cuts model size by ~75%, proportionally reducing bandwidth demands. Flash Attention 2 reduces KV cache memory traffic. Continuous batching in vLLM and TGI amortizes bandwidth costs across multiple concurrent requests, which is why server-grade inference throughput can be much higher than single-user throughput.