GGUF, GPTQ or MLX: Which Model File to Download

Open a popular model on Hugging Face and you will often find the same weights packaged three or four different ways. For a home setup, the right file depends less on which format is "best" and more on which program you plan to run it with. Match the file to the app and most of the confusion goes away.

GGUF is the default for llama.cpp, LM Studio and Ollama

GGUF is a single-file format built for llama.cpp and the GGML library. Hugging Face's documentation says it was developed by Georgi Gerganov, who also wrote llama.cpp, and that models trained in frameworks like PyTorch can be converted to GGUF for those engines. LM Studio's docs list llama.cpp (GGUF) as the way it runs models on Mac, Windows and Linux, and Ollama's import docs cover bringing in a GGUF file directly.

The names inside a GGUF repo tell you the quantization. Q4_K_M is a 4-bit quant and the usual starting point. Q5_K_M and Q6_K spend more memory for a little more quality, and Q8_0 is 8-bit and close to the original weights. The llmhardware.io sizing rule uses about 0.5 bytes per parameter for Q4_K_M, 0.625 for Q5_K_M and 1.0 for 8-bit. So a 14B model goes from roughly 10.4 GB at Q4 to roughly 18.8 GB at 8-bit once overhead is added, which is the difference between a 12 GB card and a 24 GB one.

GGUF has one more feature that suits home hardware. When a model does not fit in VRAM, llama.cpp can keep some layers on the GPU and run the rest from system RAM. In llama-server the setting is -ngl (also --n-gpu-layers), and the README says it takes an exact layer count, auto or all, with auto as the default. A split model runs slower, but it runs.

GPTQ is a GPU serving format

GPTQ comes from a 2022 paper by Elias Frantar and colleagues, "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers." It is a method for compressing weights after training. On Hugging Face you usually meet it as a folder of safetensors files with a quantization config, and those checkpoints load in Python serving stacks. vLLM, for example, lists GPTQ support through GPTQModel in its quantization docs.

If you run Ollama or LM Studio, a GPTQ folder is the wrong download. It makes sense when you already serve models with vLLM or Hugging Face Transformers on a GPU with enough memory to hold the model.

MLX is for Apple Silicon

MLX is Apple's machine learning framework for its own chips, published as an open-source repo by Apple's machine learning research team. MLX-format models are only useful on an M-series Mac. LM Studio's documentation says that on Apple Silicon Macs it can also run models with MLX, next to its GGUF engine. On a Mac, you can download both versions of the same model at the same bit width and keep whichever responds faster in your own workflow.

A short decision list

• Ollama on any operating system: GGUF.
• LM Studio on Windows or Linux: GGUF.
• LM Studio on an Apple Silicon Mac: GGUF or MLX.
• vLLM or Transformers on a GPU server: GPTQ, or another format your server supports.

Whatever the format, size the download against your memory before you pull it. The format changes how the file loads, not how many weights it holds. A 32B model at 4-bit still needs about 21 GB with overhead, which means a 24 GB card.

The GGUF vs GPTQ guide on llmhardware.io has the tool support table and more on how each format loads.

Sources: Hugging Face Hub GGUF docs, GPTQ paper (arXiv 2210.17323), vLLM quantization docs, LM Studio docs, Ollama import docs, llama.cpp server README, MLX repository. Memory figures are formula estimates.