AI 101 — What is model quantization?

A large language model is a giant bag of numbers. Quantization is the trick of shrinking those numbers so the model takes up less memory and runs faster — usually with only a small, barely-noticeable drop in quality.

If you have ever tried to run a powerful open-weight model on your own laptop and watched it choke, quantization is the reason some people can and some can't. It is also why a phone can now hold an assistant that, a couple of years ago, needed a rack of servers. This explainer is about the single most practical lever for putting AI on cheaper hardware — and it sits right at the intersection of the open-weight models we covered earlier and the "run it yourself" trend that keeps showing up in the news.

The mental model

A model's "weights" are the millions or billions of numbers it learned during training. In their full form, most of those numbers are stored in 16-bit floating point — what engineers call FP16. Each number gets 16 bits (two bytes) of memory. A 7-billion-parameter model in FP16 therefore needs about 14 gigabytes of memory just to hold the weights, before you add the overhead of actually running it. That is more than most consumer laptops have free, and it is a serious chunk of a graphics card.

Quantization converts many of those 16-bit numbers into smaller ones — 8-bit integers (INT8), 4-bit integers (INT4), or similar. Halve the bits, roughly halve the memory; go to a quarter of the bits, roughly quarter the memory. A 7B model that needed 14 GB can land near 4 GB in 4-bit form, small enough for a modest laptop GPU or even some phones. Smaller numbers also move through the chip faster, so the model tends to generate text more quickly and draws less power. The trade-off is precision: a number rounded from "13.4132" to "13" loses detail, and enough rounding can make the model's answers slightly worse or occasionally odd.

The kitchen analogy

Think of a recipe written with a hyper-precise digital scale: "add 13.4132 grams of flour." That level of detail is faithful, but it is also heavy to write down, store, and follow. Quantization is switching to measuring in whole grams — "add 13 grams." The cake comes out essentially the same, the recipe fits on a much smaller card, and you can carry it in your pocket. Push too far, though — "add a fistful" instead of a measured gram — and the cake starts to fall apart. In AI terms, aggressive quantization (2-bit or stranger) is the fistful stage: it saves the most space but the quality gets flaky.

Common misconceptions

"Quantization makes the model dumber." Not exactly. Good quantization methods keep most of the important numbers sharp and only compress the ones that don't matter much. On a 4-bit or 8-bit model, most people cannot tell the difference from the full version on everyday tasks. The drop is real but often small.

"Lower bits is always better." Lower-bit means smaller and faster, but each step down risks more quality loss. There is a sweet spot (often 4-bit or 8-bit for consumer use) rather than a race to the smallest number.

"It is the same as compression like a ZIP file." No. A ZIP file loses nothing — you get every original bit back. Quantization is lossy: the smaller numbers permanently discard some precision. You trade a little accuracy for a lot of savings, which is a different bargain.

"Only hobbyists care." Every cloud provider quietly quantizes models to cut serving costs, and on-device assistants on your phone rely on it. It is an industry-wide cost lever, not a garage hack.

Where to learn more

If you want to go deeper, the two methods you will hear named most are GPTQ and AWQ, both post-training recipes for squeezing models down to 4-bit without retraining them from scratch. The llama.cpp project and Hugging Face's documentation explain how to pick a quantization level and what each one costs in quality. And if you are deciding between running a model yourself versus calling an API, our explainer on open-weight models covers the bigger picture, while what is a context window explains the memory budget a quantized model helps you fit inside.

Which matters more to you when running AI yourself — saving memory or keeping every decimal of quality? Tell us in the comments.

Sources: GPTQ (arXiv) · AWQ (arXiv) · Hugging Face — Quantization · llama.cpp