How to — run a local LLM
Running a large language model on your own machine means your prompts don't go to a cloud provider, there's no per-token bill, and the assistant still answers when the network drops. Here's how to go from zero to a working local model in six moves, even if you've never done it before.
1. Size the model to the memory you actually have
The single decision that decides the whole project is memory — the model's files have to sit in RAM (or your graphics card's memory) while it runs, alongside everything else your computer is doing. A small model around eight-billion parameters, compressed into a few gigabytes of memory, runs comfortably on a modern laptop. Something in the 30–70-billion range wants a workstation or a machine with a serious GPU. Ignore hype about parameter counts and ask one question first: how much memory do I have free? That number, not ambition, picks your model.
2. Get a runner, don't build a pipeline
You don't need to write code, configure a server, or touch a terminal to run a local model. Start with a "runner" — an app that downloads models, loads them into memory, and gives you a chat window. Ollama and LM Studio are the two most popular starting points; either will get you running in minutes and both run on CPU alone (slower) or GPU (much faster). The point of move two is to resist the urge to over-engineer. Get the simplest thing that answers a prompt first; you can add integrations later.
3. Pick an open-weights model and a quantized copy
Once your runner is installed, its catalog lists today's mainstream open-weights families — Llama, Qwen, Gemma, Mistral — each in several sizes. Here's the one piece of jargon you need: quantization. It's a compression step that shrinks a model's memory footprint at a small cost to quality. A model offered "as 4-bit" or "8-bit" is the quantized version. Your rule of thumb: download the largest quantization your free memory can hold, because bigger and higher-fidelity generally means smarter results — at the cost of slower responses. Let the runner tell you which version fits rather than guessing.
4. Let your GPU do the heavy lifting if you have one
Most local runs hit a wall at speed, not capability. If your machine has a decent graphics card, make sure the runner is actually using it — quality of life changes from one-sentence-per-minute to normal conversation speed. If you're on a CPU-only laptop, that's fine too: the model still works, it just answers more slowly. Speed is a real part of "running a local LLM," so set it up for the hardware you have rather than the one you wish you had.
5. Set expectations before you trust its output
Local models are genuinely good at summarising, drafting, and routine coding — and genuinely weaker than the frontier cloud models on hard reasoning and obscure, up-to-date knowledge. That's not a bug; it's the trade you're making for privacy, cost, and offline access. When one of us burned a week on Qwen at home, the consensus was clear: it's capable, but you have to know its limits — the community's verdict after a week with local Qwen was that it holds up for routine work. Remember the model only "sees" as much as its context window allows, so keep conversations focused and don't expect it to recall details from earlier in a very long session.
6. Give it a real job and judge it on that
The final move separates curiosity from usefulness. Give the model a task you'd actually ask a paid assistant to do — condensing a long document into bullet points, drafting an email, explaining a piece of code you're reading. The local model is a success if it handles that task well enough for you. If it isn't good enough, that's not a setup failure; it usually means the model is too small to know the answer or the quantization is too aggressive, and climbing one size up is the fix.
Don't do this
Don't buy new hardware before you've run the model on what you already own — most people discover a mid-size quantized model is "good enough" on hardware they already had. And don't arbitrarily trust a model file: download from your runner's own catalog or the project's official release, never a random mirror, because open-source weights can hide backdoors. We saw that risk made concrete this week — a model can carry a backdoor that arms on a chosen date, so trust your download's source.
How you'll know it worked
You'll know when two things are true at once: the model answers a real question for you with the network off, and your chat history stays on the machine. If you can say "I got a reasonable answer, offline, for free, and nothing left this computer" — you've done it.
What convinced you local was worth it — privacy, cost, or just tinkering? Tell us in the comments.
Sources: Ollama · LM Studio · llama.cpp (GitHub) · Llama (Meta) · Qwen (Alibaba)
