How to Benchmark Your Local LLM: A Practical Testing Guide

When you set up a new model or change inference settings, how do you know if it's actually better? Systematic benchmarking answers that question. This guide covers both performance benchmarking (tokens/second) and quality evaluation (how good are the outputs), giving you a complete picture of your local AI setup.

Reference point: Tokens/second numbers only mean something compared to hardware benchmarks. LLMHardware.io maintains a benchmark database so you can compare your results to expected performance for your GPU.

Performance Benchmarking

llama.cpp includes a built-in benchmark: ./llama-bench -m model.gguf. This reports prompt processing speed (PP tokens/second) and text generation speed (TG tokens/second) for standard batch sizes. Always benchmark with the same prompt length and batch size for comparisons to be valid. Warm up the GPU with one run before taking measurements — the first run includes GPU memory allocation overhead.

Quality Evaluation

For quick quality checks, use a standard test suite: 5 factual Q&A questions, 2 coding problems, 1 math word problem, and 1 multi-step reasoning task. Grade outputs yourself on a 1-5 scale. Keep the same test suite to compare across models and quantization levels. For systematic evaluation, lm-evaluation-harness runs standardized benchmarks (MMLU, HellaSwag, ARC) against any OpenAI-compatible endpoint including local servers.

Profiling Memory and GPU Utilization

Monitor with watch -n 1 nvidia-smi during inference. Look for VRAM utilization (should be high — close to model size), GPU utilization (should spike to 80-100% during token generation), and memory bandwidth utilization. Low GPU utilization combined with low VRAM usage suggests CPU bottlenecking from mixed inference. High VRAM with low GPU utilization suggests a software or batching configuration issue.