Apple's Neural Engine loses half its bandwidth at exactly 1 MiB
The Mac's AI accelerator has a hidden speed cliff, and one developer has now measured it to within 64 bytes — then found a workaround that recovers most of the throughput.
Apple's Neural Engine collapses its own memory bandwidth whenever a weight-streaming transfer lands on an exact multiple of 1 MiB. Reverse-engineer Eileen Yoon profiled the engine's DRAM throughput during single-token decoding on an M3 and found the numbers make no sense at first glance: transfers normally run at 45-60 GB/s, but at the pathological sizes they fall to 17-19 GB/s — 16.93 GB/s at one tested tensor dimension against 44.5 GB/s one step away. She logged 2,747 observations across 67 shapes and 41 randomized rounds, ruled out core contention (the collapse appears even with a single core active) and thermal drift, and noticed the dip repeats at every multiple of 2048 — a power-of-two signature that points at a fixed-width hardware counter, not a scheduling quirk. Her diagnosis, laid out in a detailed write-up: the kernel DMA's speculative prefetch ring does its bookkeeping in 14-bit line pointers, and at a 64-byte line granularity one full trip around that ring is exactly 1 MiB. A transfer whose size is an exact multiple of the ring appears "empty" to the lookahead logic, so the chip stops issuing prefetch requests and the whole transfer degrades into stop-and-go fetches. The data still arrives correctly — this is a performance defect, not a correctness one — but the chip forfeits roughly a third to two-fifths of its bandwidth doing it.
The fix is embarrassingly cheap, and that is what makes it a story. Splitting any transfer that lands on 1 MiB into two chunks restores nominal throughput: measured 2.6x faster on affected transfers, and nothing on unaffected ones, which serves as the control case proving the diagnosis. In practice, Yoon reports one model going from 10 to 24 tokens per second and another from 1.36 to 2.97 — a 2x-class gain a compiler gets for free by rounding awkwardly. Since multiples of 1 MiB are common in transformer weight layouts, the finding matters to anyone running models on Apple silicon, which is the largest installed base of NPUs on the planet and a growing refuge for local inference. It also lands the same week her architecture map of the chip explained why the engine struggles with transformers in the first place — we ran that this morning in GPT-6 Astra solves the last FrontierMath Tier 4 problem. Yoon has posted her scripts to the Apple Developer Forums asking others to reproduce the throttle; Apple has not commented, so treat the counter-level explanation as the author's well-evidenced hypothesis, not confirmed silicon documentation.
What to watch: whether the workaround lands in the open-source ANE toolchains first, or whether Apple quietly absorbs it into Core ML's compiler.
If your Mac's AI feels slower than the spec sheet promises, who do you hold responsible — Apple's hardware or Apple's compiler? Tell us in the comments.
Sources: Getting 50 GB/s Back Out of the ANE (Eileen Yoon) · M3 Neural Engine kernelDMA bandwidth throttled at 1 MiB multiples (Apple Developer Forums) · Hacker News discussion · Apple Neural Engine: Architecture, Programming, and Performance (arXiv)
