Deep Dive — The transformer is being taken apart, one layer at a time

For two years the unit of AI infrastructure has been the chip: how many FLOPS, how much memory, how big the rack. In the last three weeks the unit has quietly changed. It is now a piece of a layer.

On August 24, at Hot Chips, Nvidia declared its dedicated inference accelerator — built from the technology it acquired from Groq for a reported $20 billion — to be in full production. The system is designed to sit beside a Vera Rubin rack and take over one specific job: generating tokens. On September 13, China Mobile Cloud announced the first Chinese production system to run a large model across domestic GPUs and brain-inspired chips, and its split is even narrower: attention on the GPU, the feed-forward network on the neuromorphic chip. The vendor claims inference output and energy efficiency both more than double against an all-GPU cluster of the same budget, with business operating cost down more than 40 percent. We flagged that launch this morning in China Mobile Cloud runs attention on GPUs and FFN on brain chips.

Those are two different countries, two different suppliers and two different sets of constraints arriving at the same conclusion: a general-purpose accelerator is the wrong shape for half of what a modern model does. That is an architectural story, an economics story, and — because the Chinese version exists precisely because it cannot buy the American one — a policy story. It deserves to be read as one piece.

Why a layer is two jobs pretending to be one

A transformer block does two very different kinds of work. Attention is the part that looks back: it reads everything that came before, which means it carries state that grows with the conversation, and it spends its time moving that state around rather than multiplying it. Feed-forward networks are the other half — the part with most of the parameters, where the model's actual knowledge is stored as matrix math. In a mixture-of-experts model, which is what both the Chinese and the American systems are aimed at, the FFN is where the model's mass lives. DeepSeek's V4 Flash, the model China Mobile Cloud used to validate its system, carries 284 billion total parameters against 13 billion active per token; nearly all of that mass sits in experts.

So one job is state-heavy and bandwidth-bound, and the other is stateless and compute-bound. Running both on the same silicon means buying a chip that is a compromise for half your workload, every time. That is the intuition behind what researchers now call attention-FFN disaggregation, and it is the same intuition behind the older split between prefill and decode.

The engineering catch is that the split only pays if you size it right, and this year's arXiv paper on the subject — Analytical Provisioning for Attention-FFN Disaggregated LLM Serving under Stochastic Workloads — says so bluntly. Disaggregating lets memory and compute scale independently, the authors write, "but its performance is highly sensitive to the Attention/FFN provisioning ratio: mis-sizing induces step-level blocking and costly device idle time." Their contribution is a closed-form rule for how many attention workers you need per feed-forward worker under real, random traffic, plus a correction for the fact that a synchronized step is only as fast as its slowest worker. Calibrated against request traces, the predicted optimal ratio landed within 10 percent of what simulation said was best.

Read that result as the industry's to-do list. The win is real, and the win is a tuning problem you now have to solve continuously, per model, per workload — because traffic mix changes, context lengths change, and a swapped model changes the ratio.

The American version splits by phase; the Chinese one splits by function

Nvidia's stack is a phase split. Rubin GPUs handle the large-context processing, and the Groq-derived accelerators take the latency-sensitive generation work, with the two "jointly computing every layer." Nvidia packages 256 of the accelerators in a rack linked chip to chip, and cites an Artificial Analysis benchmark in which the rack hit 3,400 output tokens per second on an open Gemma 4 model at a 100,000-token context window — about four times the nearest alternative platform. Nebius is the first cloud customer; the racks go online later this year. The framing Nvidia uses is that agentic systems multiply small delays across chains of tool calls, so decode latency is now the scarce product.

China Mobile Cloud's stack splits twice. Prefill and attention go to a domestic GPU — built with Shanghai GPU maker Iluvatar CoreX — while the feed-forward network goes to a neuromorphic chip from Beijing-based LingXi Technology, whose compute-in-memory design and large on-die storage suit the bandwidth-hungry half of the layer. Getting that to work is why the project names a consortium — China Electronics Technology Group's Nanhu Research Institute, Tsinghua, Peking University — and claims fifteen granted invention patents, six software copyrights and small-batch trial production already running. The partners said the heterogeneous framework will be opened gradually to other domestic GPU and neuromorphic vendors.

The honest read on the Chinese numbers is that they are vendor-supplied, single-model, and measured against a same-budget domestic cluster rather than against Nvidia. That is the correct comparison for the buyer making this decision — nobody in the Chinese procurement path is choosing between Iluvatar and a full Rubin rack — but it is not the comparison a foreign analyst wants. Note also that the team benchmarked its work explicitly against Nvidia's Vera Rubin-plus-Groq combination, which tells you where the American design is being read from.

The economics underneath: memory is the bill, not arithmetic

Why is everyone rearranging silicon at once? Because the expensive part of serving a model has stopped being floating-point operations and started being the cost of holding state.

Consider what we know about memory supply: AI demand has booked up all 2027 DRAM and high-bandwidth memory capacity, a single HBM4 stack reportedly sells for around $392, and Nvidia has been testing versions of its next-generation flagship with as little as 192 GB of memory against the 1 TB originally announced, because HBM supply cannot keep pace. We laid out the mechanics in AI 101 — What is HBM (high-bandwidth memory)?. Chinese accelerator prices jumped 20 to 50 percent in two months on the same squeeze, and DeepSeek's answer to domestic silicon scarcity has been to plan clusters of 160,000 Huawei chips rather than wait for better ones — we covered that in DeepSeek plans a 160,000-chip Huawei cluster in Inner Mongolia.

Against that background, splitting a layer is a way of buying less of the scarce thing. Attention workers are provisioned for memory capacity and bandwidth; feed-forward workers are provisioned for arithmetic and on-chip storage. A chip that keeps expert weights in large on-die SRAM never touches the HBM market for that half of the job, and a specialized decode accelerator is not sized for the terabyte context windows that force the memory purchase. Both designs, in other words, are arguments that the cheapest fourth of an inference bill is the fourth you can move off the most expensive component.

There is a second economics story, and it belongs to Nvidia. The company paid a reported $20 billion for Groq's technology in 2025 — a deal described publicly as a non-exclusive licensing agreement and now, according to reporting on a Department of Justice inquiry, being examined for whether it was structured to avoid antitrust review. Groq itself, having sold its talent and its story to a competitor, closed a $350 million round in August at $3.5 billion, half its September 2025 valuation. Buying the specialist and absorbing it into your rack is a defensive moat and a commoditization of your own acquisition in the same motion.

What skeptics say, and the part that stings

Three objections deserve to be taken seriously.

The first is the round trip. Splitting a layer across two chip classes means the two halves exchange data on every step, for every token, in every layer. That is a scheduling and interconnect burden with no upper bound until someone measures it on real traffic — which is exactly the sensitivity the provisioning paper formalizes. A cluster sized wrong for its workload mix does not become two efficient half-clusters; it becomes two idle ones.

The second is portability. Every model swap re-tunes the ratio, and a heterogeneous stack is only as good as the compiler that hides it. The reason CUDA is a moat is that programmers never think about the hardware. A system whose pitch requires a self-developed model compiler, a bespoke interconnect protocol and a unified engine that splits, schedules and aggregates across two chip classes has rebuilt, in miniature, the integration problem Nvidia spent fifteen years and a software stack solving. The claim that the framework will be opened to other domestic vendors is the most interesting line in the announcement and the least proven.

The third is the version of the "innovation" question. A Chinese state-linked cloud operator describing a system engineered around not being able to buy the best memory, the best node and the best accelerator is describing an export-control workaround, and the press release effectively says so by choosing Nvidia's own heterogeneous design as its benchmark. That is not nothing — architecture is real engineering, and Intel, Arm and AMD have all benefited from decades of doing more with less-favorable silicon. But the framing that splitting mediocre chips beats buying excellent ones flatters the constraint that produced it.

The contrarian counter is the one nobody in the American camp wants said out loud. The same logic that pushes China toward splitting layers is what has Nvidia shipping an accelerator whose job is to make expensive memory unnecessary, and The Information reported in August that Nvidia plans small-batch shipments of an LPU tailored to Chinese customers by the end of 2026, built to comply with US export rules. The bifurcated architecture is the shape export controls produce on both sides of the wall: a market where the winning product is not the strongest general chip but the right specialist chip for the half of a layer the customer can afford.

What to watch

First, whether anyone independent puts the Chinese claim through a public harness. The InferenceX-style benchmarks that made Jalapeño's numbers credible could do the same for a GPU-plus-neuromorphic split; until then the 40-percent cost claim is a vendor statement. Second, whether Nvidia's decode split shows up in prices rather than demos — token-generation speed is only a product when per-million-token economics move, and OpenAI's Cerebras-powered Ultrafast mode at 750 tokens per second already gave buyers a reference point. Third, whether the disaggregation ratio becomes a published, comparable spec, because that is the moment operators start buying attention capacity and feed-forward capacity separately and the rack stops being the unit. Fourth, the DOJ's Groq inquiry, which is the test of whether the licensing-deal structure is a template or a one-off. Fifth, whether Mobile Cloud actually ships the open framework to third-party chip vendors, which is the difference between a national workaround and a market.

If the cheapest way to add AI capacity is to stop buying one kind of expensive chip, who loses more: the memory supplier or the general-purpose accelerator? Tell us in the comments.

Sources: NVIDIA Technical Blog · arXiv: Analytical Provisioning for Attention-FFN Disaggregated LLM Serving · CNRI (央广网) · Bloomberg · The New York Times via Techmeme · The Information via Techmeme