AI 101 — What is an NPU (neural processing unit)?
An NPU is a chip built for one job: running the math that neural networks do, at a fraction of the power a general-purpose CPU or a graphics GPU would burn. The name covers anything from the block inside your phone's processor to a USB stick in a robot to a rack card in a datacenter. Vendors also call it an AI accelerator or a neural engine, and it is the hardware reason "on-device AI" stopped being a marketing phrase.
Why this matters right now
The NPU is the quiet spec war of 2026. Microsoft set an actual floor for its Copilot+ PC class: an NPU that can do more than 40 trillion operations per second, or TOPS. Apple's M5, announced in October 2025, claims over 4x the peak GPU compute performance for AI compared to M4, because each GPU core now carries its own "Neural Accelerator" alongside a faster 16-core Neural Engine. Qualcomm says the Hexagon NPU in its next Snapdragon can run a Mixture-of-Experts model with up to 30 billion parameters entirely on a phone. Huawei is making the same claim for its Kirin 9050 Pro. Every one of those announcements is a bet that the next decade of AI runs partly on hardware you own — and the NPU is the part that makes the economics work.
The mental model
A modern chip has three kinds of compute, and the difference is what they give up. A CPU is a brilliant generalist: a handful of very clever cores that can run any program, chase branches, and handle messy logic — and it does that work one step at a time. A GPU is thousands of simple lanes doing the same easy instruction across a huge pile of data at once; still programmable, still powerful, still hungry. An NPU is the third option: it stops pretending to be general. It is wired almost entirely for multiply-accumulate operations at low precision — 8-bit integers, 4-bit, sometimes fp8 — because that is what neural networks spend nearly all their time doing. Strip out the branch prediction, the caches, the floating-point flexibility, and a given watt buys many more useful operations. That is the whole idea: specialization in exchange for efficiency.
The catch is that the arithmetic is usually not the limit. Running a language model means streaming billions of weight numbers from memory into those multiply-accumulate lanes, over and over. When the data cannot arrive fast enough, the silicon sits idle no matter how many operations per second it is rated for. This is why memory bandwidth — not TOPS — decides how fast a model actually runs, and why a laptop with unified memory can beat a desktop with a bigger accelerator on the same task.
A kitchen analogy
Picture a restaurant. The CPU is one classically trained chef: can cook literally anything, works one pan at a time. The GPU is a line of two hundred cooks all executing the same simple step in parallel — brilliant for scale, expensive to feed. The NPU is a tortilla press. Dough goes in one side, flat tortillas come out the other, forty times faster than any human could roll them, for a fraction of the effort — and it cannot make a single dish that is not a tortilla. Presses are cheap, tireless, and permanently one-trick. Put one in every kitchen and tortillas become free; that is exactly what has happened to the specific math AI runs.
Common misconceptions
"More TOPS means a faster AI." Vendors count differently, and the number is a peak, not a sustained rate. Qualcomm's own guidance notes that TOPS does not specify which operations, and on NPUs it is usually INT8 multiplications and additions — a different unit than a GPU's rating. Two chips with identical TOPS can differ by multiples in tokens per second, because one of them is bandwidth-starved.
"The NPU replaces the GPU for AI." It does not. NPUs are overwhelmingly inference devices; frontier training still happens on datacenter GPUs and on a handful of specialized accelerators. And an NPU only helps when a model compiles cleanly onto it — which is why Apple's popular MLX framework runs models on the CPU and GPU of an M-series Mac rather than on the Neural Engine at all.
"On-device AI means small models." Not necessarily — it means sparse models. The 30-billion-parameter phone demos are Mixture-of-Experts: only about 3 billion parameters are active for any given token, and that small active slice is what fits inside a phone's memory budget and power envelope.
"Apple silicon's Neural Engine is a mystery black box." It used to be. A reverse-engineering effort published this month by Eileen Yoon mapped the M1 engine's internals — 16 fixed-function cores, no instruction set, a 1 MiB kernel memory — and argued the chip is pinned by memory bandwidth rather than compute: peak throughput needs roughly 162 operations per byte fetched, and the kernel path tops out near 38 GB/s. Her reading is that the engine was architected around the predictable weight reuse of vision networks, an assumption transformers broke. Treat that as a well-evidenced outside analysis, not confirmed documentation — Apple has not commented.
Where to learn more
The two numbers worth learning to read skeptically are TOPS (operations per second, counted differently by every vendor) and memory bandwidth (gigabytes per second delivered to the chip) — the second one is almost always the real limit. Quantization is the sibling trick that shrinks weights enough to fit an NPU's narrow memory path, and the datacenter version of the same bandwidth problem is the stacked-memory story.
Related reading: What is model quantization? · Local LLMs vs cloud APIs · What is a mixture of experts?
If your phone could run a real model offline, what's the first task you'd take off the cloud — and what would you never trust to on-device AI? Tell us in the comments.
