Mistral 7B Optimization Guide: Getting 150 Tokens/sec on RTX 4090

Mistral 7B is one of the most efficient small language models ever released. With the right quantization and llama.cpp settings, an RTX 4090 can generate over 150 tokens per second — faster than any person can read. Here is how to get the absolute maximum performance.

Baseline Setup

ollama pull mistral:7b-instruct-q8_0
ollama run mistral:7b-instruct-q8_0
# Expected: ~100 tok/s on RTX 4090

Optimized llama.cpp Build

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp

# RTX 4090 (Ada Lovelace sm89) with all optimizations
cmake -B build   -DGGML_CUDA=ON   -DGGML_CUDA_FORCE_MMQ=OFF   -DGGML_CUDA_FA_ALL_QUANTS=ON   -DCMAKE_CUDA_ARCHITECTURES=89
cmake --build build --config Release -j16

Maximum Speed Configuration

./build/bin/llama-server   -m ./models/mistral-7b-instruct-v0.3-Q8_0.gguf   --n-gpu-layers 999   --flash-attn   --ctx-size 8192 \         # smaller context = faster
  --batch-size 2048   --ubatch-size 512   --threads 8   --host 0.0.0.0 --port 8080

# Benchmark
./build/bin/llama-bench   -m ./models/mistral-7b-q8.gguf   -n 128 -ngl 999 -fa 1

Performance Tuning Flags Explained

Flag	Effect	Recommendation
--flash-attn	Reduces KV VRAM 30%	Always on
--batch-size 2048	Prompt processing speed	2048 for RTX 4090
--ubatch-size 512	Micro-batch for VRAM balance	512 default
--n-gpu-layers 999	Full GPU offload	Always max

For comparing Mistral 7B speed against other budget model options on the same hardware, see LLMHardware.io's 7B model benchmark comparison.