llama.cpp Flash Attention: How to Halve Your Context VRAM Usage

Flash attention is the single most impactful optimization flag in llama.cpp for long-context inference. Enabling it reduces KV cache memory usage by 30-50%, letting you run longer contexts on the same GPU. Here is everything you need to know.

What Is Flash Attention?

Flash Attention is an algorithm that computes attention in a memory-efficient way by tiling the computation across GPU SRAM. Instead of materializing the full attention matrix in VRAM, it processes blocks — dramatically reducing peak memory usage.

Memory Savings: Real Numbers

Context Length	Without Flash Attn	With Flash Attn	VRAM Saved
8K tokens	~2 GB KV	~1.2 GB KV	800 MB
32K tokens	~8 GB KV	~4.5 GB KV	3.5 GB
128K tokens	~32 GB KV	~18 GB KV	14 GB

Enabling Flash Attention

# Build with flash attention (requires CUDA 11.8+ or ROCm 5.6+)
cmake -B build   -DGGML_CUDA=ON   -DGGML_CUDA_FORCE_MMQ=OFF   -DCMAKE_CUDA_ARCHITECTURES=86,89  # RTX 3090=86, RTX 4090=89
cmake --build build -j12

# Run with flash attention enabled
./build/bin/llama-server   -m ./models/qwen3.6-32b-q4.gguf   --n-gpu-layers 999   --flash-attn \           # THE FLAG
  --ctx-size 65536 \       # now feasible on RTX 4090
  --host 0.0.0.0 --port 8080

What to Watch Out For

• Not all models support it: Most transformer models work, but some custom architectures may give incorrect results. Test with a known-good prompt first.
• Speed effect varies: Flash attention is faster at long contexts (64K+) but neutral at short contexts. Do not expect speed gains at 4K context.
• CUDA only at maximum efficiency: The CPU/Metal backend has a basic FA implementation; CUDA benefits most.

Combine with KV Cache Type for Maximum VRAM Efficiency

./build/bin/llama-server   -m ./model.gguf   --n-gpu-layers 999   --flash-attn   --cache-type-k q8_0 \   # quantize KV cache to 8-bit (halves KV VRAM)
  --cache-type-v q8_0   --ctx-size 131072 \      # 128K context now realistic on 24GB
  --host 0.0.0.0 --port 8080

For hardware comparisons showing how different GPUs perform with flash attention enabled at long contexts, see LLMHardware.io's long-context benchmark page.