The LLM Guy

A used RTX 3090 is still the cheapest common way to get 24 GB of VRAM on one NVIDIA card. NVIDIA lists it with 24 GB of GDDR6X on a 384-bit bus, the same capacity as the RTX 4090. For local language models, capacity decides which models fit, so the older card opens the same model sizes as the newer one. What fits in 24 GB The llmhardware.io formula is parameters in billions × bytes per parameter × 1.2, plus 2 GB for the system. Run common sizes through it: A 32B model at Q4 comes to about 21.2 GB. It fits. A 27B model at Q4 comes to about 18.2 GB. It fits with room for a longer context. A 14B model at 8-bit comes to about 18.8 GB. It fits. An 8B model at full 16-bit precision comes to about 21.2 GB. It fits. A 70B model at Q4 comes to about 44 GB. It does not. That 32B line is the reason people still hunt for this card. A 16 GB card misses the 21.2 GB estimate, so the jump from 16 GB to 24 GB is the jump from 14B-class models to 32B-class ones. Where the newer cards di...

Ollama has a long command list, but day to day you will use a handful of them. They cover the whole loop: download a model, run it, check where it landed in memory, unload it, and clean up the disk. Every command below comes from Ollama's own CLI reference and FAQ. Get a model and run it ollama pull qwen3:8b downloads a model without starting it, which is handy when you want a large download done before you sit down to work. ollama run qwen3:8b starts a chat with it in the terminal. You can also put a prompt in quotes after the model name, which turns run into a one-shot command you can call from a script. See what is loaded ollama ps lists running models, and the column to read is PROCESSOR. Ollama's FAQ says 100% GPU means the model loaded entirely into the GPU, 100% CPU means it loaded entirely into system memory, and a split such as 48%/52% CPU/GPU means it sits partly in each. A split is the most common reason a local model suddenly feels slow, and it usually ...

Three kinds of desktop machine now put 128 GB of memory within reach of a home user: NVIDIA's DGX Spark, Apple's Mac Studio, and mini PCs built on AMD's Ryzen AI Max+ 395, the chip nicknamed Strix Halo. All three let the GPU work from one big shared memory pool, so a 70B model at 4-bit, about 44 GB by the llmhardware.io formula, fits on any of them. Fitting is the easy part. How fast the model answers depends mostly on memory bandwidth, and that number differs by more than four times across these machines. What the spec sheets say DGX Spark: 64 GB or 128 GB of LPDDR5x on a 256-bit interface, 273 GB/s, per NVIDIA. Ryzen AI Max+ 395: up to 128 GB of LPDDR5x-8000 on a 256-bit bus, per AMD. That works out to about 256 GB/s. Mac Studio with M5 Max: 460 GB/s on the 32-core GPU chip or 614 GB/s on the 40-core GPU chip, with up to 128 GB, per Apple. Mac Studio with M5 Ultra: 1.2 TB/s, with 96 GB configurable to 256 GB or 512 GB, per Apple. Why bandwidth sets the pace ...

Google's Gemma 3 comes in five sizes: 270M, 1B, 4B, 12B and 27B parameters. The model card lists a 128K-token context for the 4B, 12B and 27B models and 32K for the two smallest, and it describes the family as multimodal, taking text and images as input. On a home GPU the real question is which size fits, and how much room is left once you start feeding it long documents or pictures. Sizing Gemma 3 with one formula llmhardware.io sizes a model as parameters in billions × bytes per parameter × 1.2, plus 2 GB for the system. At Q4_K_M a weight takes about half a byte, and at 8-bit it takes one. Run the sizes through it and you get: Gemma 3 1B: about 2.6 GB at Q4. Gemma 3 4B: about 4.4 GB at Q4, 6.8 GB at 8-bit. Gemma 3 12B: about 9.2 GB at Q4, 16.4 GB at 8-bit. Gemma 3 27B: about 18.2 GB at Q4, 34.4 GB at 8-bit. Card by card An 8 GB card such as the RTX 4060 runs Gemma 3 4B comfortably, even at 8-bit. It does not hold the 12B at Q4, which needs about 9.2 GB. A 12 GB...

Open a popular model on Hugging Face and you will often find the same weights packaged three or four different ways. For a home setup, the right file depends less on which format is "best" and more on which program you plan to run it with. Match the file to the app and most of the confusion goes away. GGUF is the default for llama.cpp, LM Studio and Ollama GGUF is a single-file format built for llama.cpp and the GGML library. Hugging Face's documentation says it was developed by Georgi Gerganov, who also wrote llama.cpp, and that models trained in frameworks like PyTorch can be converted to GGUF for those engines. LM Studio's docs list llama.cpp (GGUF) as the way it runs models on Mac, Windows and Linux, and Ollama's import docs cover bringing in a GGUF file directly. The names inside a GGUF repo tell you the quantization. Q4_K_M is a 4-bit quant and the usual starting point. Q5_K_M and Q6_K spend more memory for a little more quality, and Q8_0 is 8-bit and ...

Qwen3 comes in eight open-weight sizes, and the one you can run comes down to a single number: how much memory your graphics card has. Alibaba's Qwen team released six dense models (0.6B, 1.7B, 4B, 8B, 14B and 32B) and two mixture-of-experts models (30B-A3B and 235B-A22B), all under the Apache 2.0 license. Picking the right one takes about a minute of arithmetic. The formula The rough fit check I use is VRAM (GB) = parameters in billions × bytes per parameter × 1.2 + 2 . At Q4_K_M, the most common 4-bit GGUF quant, a parameter takes about half a byte. At 8-bit it takes one byte. The 1.2 covers the KV cache and runtime overhead at a modest 4K to 8K context, and the extra 2 GB is held back for the operating system and the inference engine itself. Qwen3 sizes by card Here is what that formula gives at Q4_K_M: Qwen3-4B: about 4.4 GB. Fits any 8 GB card, and the 8-bit version fits too. Qwen3-8B: about 6.8 GB. Fits an 8 GB card at a short context. Qwen3-14B: about 10.4 GB. W...