The LLM Guy

Microsoft's Phi-4 is a 14-billion-parameter model. Its model card describes a dense decoder-only Transformer with a 16K-token context window, released on December 12, 2024. Dense means every weight is read for every token, so the memory math is simple: at 4-bit it fits a 12 GB graphics card. Using the whole context window on that card is where it gets tight. The weights: about 10.4 GB at Q4 llmhardware.io sizes a model as parameters in billions × bytes per parameter × 1.2, plus 2 GB for the system. At Q4_K_M a weight takes about half a byte, so Phi-4 comes to about 10.4 GB. At 8-bit it is about 18.8 GB, and at FP16 about 35.6 GB. Ollama lists the default phi4 download at 9.1 GB, which is the file alone, before the runtime and the context take their share. That sorts the cards. An 8 GB card does not hold Phi-4 at Q4. A 12 GB card does, with a little room left. A 16 GB card such as the RTX 4060 Ti 16GB or RTX 5060 Ti 16GB holds it with space for a longer context, but not the ...

Most people who start running models at home install either Ollama or LM Studio first. Both run the same kinds of open-weight models on your own machine, and both are free to download. The difference is how you work with them, and that decides which one to start with. Ollama: a command-line tool with a local API Ollama runs as a background server, and you drive it from the terminal. ollama pull qwen3:8b downloads a model, ollama run qwen3:8b starts a chat, and ollama ps shows what is loaded and whether it fits in your GPU. The same server answers HTTP requests on localhost:11434 , which is what makes Ollama a good base for other tools: chat front ends, coding assistants and scripts can all point at that one address. Settings live in environment variables. Ollama's FAQ says the default context window is 4096 tokens, and OLLAMA_CONTEXT_LENGTH raises it. That is easy to script and easy to forget, so note what you changed. LM Studio: a desktop app with a model browser LM...

A used RTX 3090 is still the cheapest common way to get 24 GB of VRAM on one NVIDIA card. NVIDIA lists it with 24 GB of GDDR6X on a 384-bit bus, the same capacity as the RTX 4090. For local language models, capacity decides which models fit, so the older card opens the same model sizes as the newer one. What fits in 24 GB The llmhardware.io formula is parameters in billions × bytes per parameter × 1.2, plus 2 GB for the system. Run common sizes through it: A 32B model at Q4 comes to about 21.2 GB. It fits. A 27B model at Q4 comes to about 18.2 GB. It fits with room for a longer context. A 14B model at 8-bit comes to about 18.8 GB. It fits. An 8B model at full 16-bit precision comes to about 21.2 GB. It fits. A 70B model at Q4 comes to about 44 GB. It does not. That 32B line is the reason people still hunt for this card. A 16 GB card misses the 21.2 GB estimate, so the jump from 16 GB to 24 GB is the jump from 14B-class models to 32B-class ones. Where the newer cards di...

Ollama has a long command list, but day to day you will use a handful of them. They cover the whole loop: download a model, run it, check where it landed in memory, unload it, and clean up the disk. Every command below comes from Ollama's own CLI reference and FAQ. Get a model and run it ollama pull qwen3:8b downloads a model without starting it, which is handy when you want a large download done before you sit down to work. ollama run qwen3:8b starts a chat with it in the terminal. You can also put a prompt in quotes after the model name, which turns run into a one-shot command you can call from a script. See what is loaded ollama ps lists running models, and the column to read is PROCESSOR. Ollama's FAQ says 100% GPU means the model loaded entirely into the GPU, 100% CPU means it loaded entirely into system memory, and a split such as 48%/52% CPU/GPU means it sits partly in each. A split is the most common reason a local model suddenly feels slow, and it usually ...

Three kinds of desktop machine now put 128 GB of memory within reach of a home user: NVIDIA's DGX Spark, Apple's Mac Studio, and mini PCs built on AMD's Ryzen AI Max+ 395, the chip nicknamed Strix Halo. All three let the GPU work from one big shared memory pool, so a 70B model at 4-bit, about 44 GB by the llmhardware.io formula, fits on any of them. Fitting is the easy part. How fast the model answers depends mostly on memory bandwidth, and that number differs by more than four times across these machines. What the spec sheets say DGX Spark: 64 GB or 128 GB of LPDDR5x on a 256-bit interface, 273 GB/s, per NVIDIA. Ryzen AI Max+ 395: up to 128 GB of LPDDR5x-8000 on a 256-bit bus, per AMD. That works out to about 256 GB/s. Mac Studio with M5 Max: 460 GB/s on the 32-core GPU chip or 614 GB/s on the 40-core GPU chip, with up to 128 GB, per Apple. Mac Studio with M5 Ultra: 1.2 TB/s, with 96 GB configurable to 256 GB or 512 GB, per Apple. Why bandwidth sets the pace ...

Google's Gemma 3 comes in five sizes: 270M, 1B, 4B, 12B and 27B parameters. The model card lists a 128K-token context for the 4B, 12B and 27B models and 32K for the two smallest, and it describes the family as multimodal, taking text and images as input. On a home GPU the real question is which size fits, and how much room is left once you start feeding it long documents or pictures. Sizing Gemma 3 with one formula llmhardware.io sizes a model as parameters in billions × bytes per parameter × 1.2, plus 2 GB for the system. At Q4_K_M a weight takes about half a byte, and at 8-bit it takes one. Run the sizes through it and you get: Gemma 3 1B: about 2.6 GB at Q4. Gemma 3 4B: about 4.4 GB at Q4, 6.8 GB at 8-bit. Gemma 3 12B: about 9.2 GB at Q4, 16.4 GB at 8-bit. Gemma 3 27B: about 18.2 GB at Q4, 34.4 GB at 8-bit. Card by card An 8 GB card such as the RTX 4060 runs Gemma 3 4B comfortably, even at 8-bit. It does not hold the 12B at Q4, which needs about 9.2 GB. A 12 GB...

Open a popular model on Hugging Face and you will often find the same weights packaged three or four different ways. For a home setup, the right file depends less on which format is "best" and more on which program you plan to run it with. Match the file to the app and most of the confusion goes away. GGUF is the default for llama.cpp, LM Studio and Ollama GGUF is a single-file format built for llama.cpp and the GGML library. Hugging Face's documentation says it was developed by Georgi Gerganov, who also wrote llama.cpp, and that models trained in frameworks like PyTorch can be converted to GGUF for those engines. LM Studio's docs list llama.cpp (GGUF) as the way it runs models on Mac, Windows and Linux, and Ollama's import docs cover bringing in a GGUF file directly. The names inside a GGUF repo tell you the quantization. Q4_K_M is a 4-bit quant and the usual starting point. Q5_K_M and Q6_K spend more memory for a little more quality, and Q8_0 is 8-bit and ...

Qwen3 comes in eight open-weight sizes, and the one you can run comes down to a single number: how much memory your graphics card has. Alibaba's Qwen team released six dense models (0.6B, 1.7B, 4B, 8B, 14B and 32B) and two mixture-of-experts models (30B-A3B and 235B-A22B), all under the Apache 2.0 license. Picking the right one takes about a minute of arithmetic. The formula The rough fit check I use is VRAM (GB) = parameters in billions × bytes per parameter × 1.2 + 2 . At Q4_K_M, the most common 4-bit GGUF quant, a parameter takes about half a byte. At 8-bit it takes one byte. The 1.2 covers the KV cache and runtime overhead at a modest 4K to 8K context, and the extra 2 GB is held back for the operating system and the inference engine itself. Qwen3 sizes by card Here is what that formula gives at Q4_K_M: Qwen3-4B: about 4.4 GB. Fits any 8 GB card, and the 8-bit version fits too. Qwen3-8B: about 6.8 GB. Fits an 8 GB card at a short context. Qwen3-14B: about 10.4 GB. W...

You load a model, ask it something, and the answer crawls out one word at a time. The model "works," but it feels broken. Often the cause is simple: the model didn't fit in your GPU's memory, and part of it quietly moved into system RAM. The good news is you don't have to guess. Both of the popular local runners will tell you exactly where the model landed, if you know where to look. Check it in Ollama with one command Load your model, send it a prompt, then open a second terminal and run: ollama ps Look at the PROCESSOR column. The Ollama FAQ spells out what it means: 100% GPU means the whole model is in video memory. This is what you want. 100% CPU means it is running entirely from system memory. A split like 48%/52% CPU/GPU means part of the model is on the GPU and part is in system RAM. That split is the one that catches people. The model still answers, so it's easy to miss. But every token now has to touch the slower memory too, and g...

A 70-billion-parameter model at 4-bit quantization needs about 40 gigabytes of memory. A 24-gigabyte card cannot hold it. But many people run 70B models on 24GB cards anyway, thanks to offloading: the runtime keeps some layers of the model on the graphics card and parks the rest in system RAM, shuttling weights back and forth as it generates text. The idea is simple once you see how inference works. A transformer model is a stack of identical layers. To generate each token, the computation passes through every layer in order. So you do not need the whole model in video memory at the same instant; you only need the layer you are computing right now, plus the key-value cache for the conversation. Tools like llama.cpp exploit this by letting you choose how many layers live on the graphics card (GPU layers) and how many stay in system memory. The practical effect is a sliding scale. Push more layers onto the card and generation gets faster. Pull layers back into RAM and speed drops, bec...

Mistral AI put out a preview of its biggest model yet on October 6: Mistral Large 4, nicknamed "le Chonk." It is a natively multimodal mixture-of-experts model with about 1 trillion parameters in total and 52 billion active per token. The weights are not out yet. Mistral says they land by the end of October. The headline numbers come with a catch that matters for anyone running models at home. The 52 billion active parameters decide how fast the model runs, not how much memory it needs. All one trillion weights have to sit in memory, which puts Large 4 at roughly half a terabyte for the weights alone at Q4 quantization, before you generate a single token. For comparison, GLM-5.3-Flash is 320B total parameters, which still works out to about 180GB at Q4, so it lives in data centers too. Large 4 is a multi-GPU cluster job, not a desktop. Mistral knows this. The company trained the model from scratch on 3,800 of its own Grace Blackwell GPUs in European data centers, and ...

The RTX 5070 is the card a lot of people actually own. It launched with 12GB of VRAM, and 12GB is the awkward middle of the local-LLM world: plenty for some models, not enough for others, and the line between them is not obvious. Here is what fits, what almost fits, and what needs a workaround. Your real budget is about 10.5GB The card says 12GB, but the model does not get all of it. The GPU driver, the CUDA runtime, and the inference engine take roughly a gigabyte before you load anything. Plan on 10.5GB of usable space, and leave a little headroom on top of that for the KV cache, the scratch memory the model uses to remember the conversation. A useful rule of thumb: at Q4 quantization, a model needs roughly half a gigabyte per billion parameters. At Q8, closer to a full gigabyte per billion. These are ballpark numbers, but they get you in the right neighborhood. What fits comfortably 7-8B models at Q8. An 8B model at Q8 lands around 8.5GB, which leaves real room for context. ...

There are two bills for running a language model, and most people only see one. The first is the weights: the model file itself, fixed in size, easy to plan for. The second grows every time you use the model. It is called the KV cache, and it is the reason a 7B model that fits comfortably on your 8 GB card suddenly runs out of memory the moment you paste in a long document. Understanding this second bill changes how you pick a model, how much context you can afford, and whether that shiny 128K context window on the model card is something you can actually use. What the cache actually is When a model reads your prompt, it converts every token into two sets of numbers, called keys and values, at every layer. It then stores them instead of recomputing them for each new token it generates. That stored state is the KV cache. Without it, generating each token would require re-reading the entire prompt from scratch, which would be unusably slow. With it, the model pays in memory instead...

The most interesting benchmark number this week belongs to the smallest model in the news. H2O.ai has released H2O-Lightning-4B, a 4-billion-parameter open-weight model under the Apache 2.0 license, available now at huggingface.co/h2oai/h2o-lightning-4b. The company says it ranks first on the JevBench open-weights leaderboard, with roughly one-third the latency and two-thirds the cost of the reference model it is measured against. Four billion parameters will not win a frontier coding benchmark, and H2O is not pretending it will. The pitch is different. According to the company's announcement, the score it watches most is calibration: when the model says it is 80 percent confident, it should be right about 80 percent of the time. A team can then automate the confident calls and send the uncertain ones to a person. That framing is worth paying attention to, because it is the shape of how small models actually get used. Nobody deploys a 4B model expecting it to write a novel. They de...

Why a 1M-Token Context Changes Your VRAM Math When a model ships with a million-token context window, the headline number everyone talks about is parameters. Mistral Large 4 is a 1.05-trillion-parameter mixture-of-experts model with about 49 billion active parameters per token and a 1M-token context. The weight math is straightforward. The part that surprises people is the second bill: the memory for the context itself. That second bill is the KV cache. When a model generates text, it computes a key and a value for every token it has seen, then stores them so it does not recompute them for the next token. That cache lives in VRAM right next to the weights, and it grows with every token in your context window. It is the reason a model that fits on your GPU at short context can run out of memory at long context. The formula is simple. For each token, the cache holds 2 (keys plus values) times the layer count times the number of KV heads times the head size times the bytes per value. T...

Cloudflare's Clef: Models That Skip the Chat and Just Decide Think about what happens inside an AI agent. Before it writes anything, it makes a dozen small calls: is this ticket urgent, which team owns it, is this crawler a bot or a person, does this tool call look safe. Most teams run those calls through a chat model, pay chat-model prices, and wait for chat-model latency to answer a yes-or-no question. Cloudflare thinks that is wasteful. On October 1, the company released two models called Clef and Clef-flash that do not write anything at all. They read an input state and a set of typed questions, then return an answer with a confidence score. No prose, no pleasantries, just the decision. The design is deliberate. Clef is built on a 27-billion-parameter Qwen backbone, and Clef-flash uses a 9-billion-parameter one. Both freeze that backbone and train low-rank adapters on top, which kept training cheap and the models fast. A prefill-only architecture scores the possible answers ...

  This month's big model releases each came with two numbers. Reflection Beam: 501 billion parameters, 23 billion active. Mistral Large 4: 1 trillion parameters, around 50 billion active. These are the companies' own figures from their October announcements. If you run models locally, you need to know which number your hardware cares about. The active number sets your speed. In a mixture-of-experts (MoE) model, only a slice of the network fires for each token. Inference cost behaves closer to a 23B model than a 501B one. That is why these giant models can still serve cheaply. But the total number sets your memory. Every parameter has to sit in video memory (VRAM) or unified memory before a single token is generated, firing or not. Beam's 23B active sounds workstation-friendly until you do the math: all 501B parameters at 8-bit precision need roughly 500 GB of memory. At 4-bit quantization, about half that. Either way, it is a multi-GPU server, not a desktop. So read model s...

A new open-weight giant just showed up. On October 5, Reflection AI announced Beam, its first open-weight model: a sparse mixture-of-experts transformer with 501 billion total parameters and 23 billion active per token. It is text-only, built for coding, reasoning, and agentic work, and it carries a 1-million-token context window. The weights land under an Apache 2.0 license later this month, after final red-teaming. For now, early access is a waitlist. The headline number that matters is not 23 billion. It is 501 billion. In a mixture-of-experts model only a fraction of the parameters fire on each token, but all of them have to sit in memory. Rule of thumb: expect roughly 500 GB at 8-bit precision, or about half that with a 4-bit community quantization. That is multi-GPU server territory, not a home workstation. So what should you actually do? Do not plan a build around weights that are not public yet. Set up your evaluation set and serving path now with a model you can already down...

Mistral announced Mistral Large 4 on October 6, nicknaming it "Le Chonk", and the headline number is hard to miss: 1.05 trillion total parameters. Before you close the tab, the number that matters for local runners is the second one: 49 billion active parameters per token. That split is the whole point of a mixture-of-experts (MoE) design. The model holds a trillion parameters of capacity, but each token only travels through a small set of expert sub-networks. Inference cost tracks the 49B active, not the 1T total. It also ships with a 1 million-token context window and multimodal input. Mistral calls it the best open-weights model from the US or Europe, and it is pitching the release at cybersecurity and sovereign deployments that cannot send data to a cloud API. The catch: this is a public preview, not a download page. Mistral says the open weights follow later this month, under a license it has not published yet. Treat the benchmarks as company-reported until independent...

A base MacBook Air with 16GB of unified memory is one of the most common machines people try local LLMs (large language models) on. It works well, once you know what actually fits. bXZ3gc_EthZQ9e58R1kh8qjh4t0NWLhnSlOCO76eg8d4WjtB8-EKOSlp-IL56H5lUChMog6RGVPZH38rgnu65KkfEzymt8T3-TcxR4AGbjQFe_hpnc24gqMTAsN1DVwinpY/s1920/2026-10-06-runopensourcellm.jpg" style="display: block; padding: 1em 0px; text-align: center;"> A base MacBook Air with 16GB of unified memory is one of the most common machines people try local LLMs (large language models) on. It works well, once you know what actually fits. Unified memory is shared between the CPU and GPU, so the number on the box is not the number you get. macOS takes a couple of gigabytes, your browser takes some more, and what is left for the model is usually 12 to 13GB in practice. That is enough for a good setup. A 7B or 8B model at Q4 quantization (4-bit) needs roughly 5 to 6GB for weights, plus another couple of gigabytes for ...

A new open-weight model worth knowing about dropped on October 3. Aleph Alpha, the German AI company, released Kolibri: a 78.1-billion-parameter mixture-of-experts model for English and German, with weights on Hugging Face under the Apache 2.0 license. The headline number is misleading in a good way. Only 3.46 billion parameters activate per token, about 4.4 percent of the total. That is the mixture-of-experts trick: a big model that costs a fraction of its size at inference time. The other headline is context: up to one million tokens, with four adjustable reasoning settings. So what does it take to run it? The official FP8 checkpoint is about 78 gigabytes, which puts full precision on a single B200, H200, or a pair of H100s, served through vLLM. That is datacenter or serious workstation territory. Home setups are not out of the picture, just further down the quantization ladder. At 4-bit, the same weights shrink to roughly 40 gigabytes, in range of a high-memory workstation or an...

Every model download on Hugging Face makes you pick a quantization: Q4_K_M, Q8_0, F16. Those labels control two things: how much disk space the model needs and how much video memory (VRAM) it takes to run. The right pick decides whether a model fits on your graphics card at all. Quantization shrinks the model's weights. A full-precision 8B model needs about 16 GB just to load. The same model at Q4 fits in about 5.5 GB, and Q8 needs around 9 GB. That is the entire difference between needing a 24 GB card and getting by with a mid-range 12 GB one. The quality loss is smaller than most people expect. For chat and general questions, Q4 gives you nearly everything the full model offers. The gap shows up in hard reasoning and code generation, where Q8 holds up better. My rule: start with Q4. If the answers wobble on your hardest prompts, step up to Q8 and pay the bigger VRAM bill. Memory math has one more term: context. A long conversation fills the key-value cache, which eats VRAM on top...