MLPerf v6.1 adds agentic inference tests — and NVIDIA's Rubin shows up
MLCommons opened its twice-yearly scoreboard today with the first tests that measure whole agentic pipelines rather than a single model call — and with performance numbers for hardware that has not shipped yet.
MLCommons released MLPerf Inference v6.1, adding two tests — End-to-end RAG for datacenters and Edge Agentic Inference for devices — and drew a record 30 submitting organizations. The point of the new workloads, per working-group co-chair Miro Hodak, is that query answering has outgrown "an LLM trained on a corpus": buyers need numbers for the multi-step, multi-component pipelines they are actually deploying, and for agentic stacks increasingly hosted on edge devices. The round also includes the largest system ever submitted to the benchmark, 512 accelerators, plus two firsts — a heterogeneous system mixing accelerators from two vendors on one network, and one geographically distributed across the Pacific Ocean.
The silicon list is the other half of the story. Three new parts are available today (AMD's Ryzen AI Max+ 395 and Instinct MI350P, Intel's Arc Pro B70), while NVIDIA's Rubin and the Vera Rubin NVL72 appear in preview — measured results for a platform that has not launched. Gains compounded the way the benchmark's sponsors like to advertise: the best per-accelerator server result on the visual-language-model test improved 2.99× in six months, and the DeepSeek R1 result is 5.7× better than v5.1 a year ago. Six organizations submitted for the first time, including Atlas Inference, Crusoe, ScitiX, VibeHPC and an individual contributor.
What makes this round more than a hardware arms race is the plumbing underneath it. More than half of submitters used MLPerf's new API-centric harness, a true client/server setup that talks to the system under test over standard APIs instead of in-process calls. MLCommons head David Kanter says that harness is the foundation of the next-generation MLPerf Endpoints suite, which will replace Inference for datacenter work entirely. Benchmarks that measure pipelines get gamed differently from benchmarks that measure a model, so the harness change matters more than any single score in it — it moves the target from "how fast is your card" to "how fast is your deployment." We covered the sibling benchmark's turn toward serving earlier this month — MLPerf Storage starts measuring inference, not just training.
Intel published a storage layout that makes ternary models smaller than the 1.58-bit label implies — by taking advantage of the zeros nobody counted. Ternary weights are restricted to −1, 0 and +1, which theory says costs log₂3 ≈ 1.585 bits each and production packing delivers at 1.625 bits after rounding to power-of-two blocks. The Intel team (Evangelos Georganas, Alexander Heinecke and Pradeep Dubey) measured the actual symbol distribution across 29 ternary checkpoints from BitNet, Bonsai, CAT-Q, ParetoQ, TriLM, Maple and BitCPM-CANN, and found zeros account for 29.7% to 51.5% of weights. Their BITCOS layout stores a presence bitmap plus a sign vector only for the non-zeros, costing 2−z bits per weight: 1.485 bits on the sparsest model, beating the five-trit format on 26 of 29 checkpoints and cutting weight traffic 1.16×–1.32× against the 2-bit format in production.
The clever part is that the saving shows up as speed, not just disk. Because decode at small batch sizes is bandwidth-bound, less weight traffic per token is directly fewer milliseconds per token, and the paper ships unpack sequences for AVX-512, AVX2 and Intel's Xe2 GPUs. Against state-of-the-art 2-bit kernels, matrix-vector multiplication runs 1.14×–1.28× faster on a 64-core Emerald Rapids server and 1.13×–1.27× on a 24-core Arrow Lake client; end-to-end vLLM decode over seven ternary checkpoints improves up to 1.18× on CPUs and 1.27× on a discrete Xe2 GPU. It is bit-exact, needs no retraining, and applies to models that already exist — the honest caveat is that on an 8-core client platform with bandwidth to spare, the smaller payload buys nothing because the unpack step is exposed.
Amazon was granted warrants to buy up to $340 million of Generac stock in exchange for backup generators for its data centers, and Generac shares jumped more than 40% after hours. The filing covers 1.69 million shares at $200.93, roughly 3% of Generac's outstanding stock, with about 308,000 vesting immediately and the rest contingent on generator payments; Generac says initial deliveries total $2.4 billion across 2027 and 2028. It is the same playbook Amazon ran last week with Qualcomm, where it took warrants worth up to $4 billion for custom inference silicon — Qualcomm hands Amazon a $4B stake option to win AWS inference silicon — and it repeats across Astera Labs, Plug Power and ATSG.
The signal is that the binding constraint on AI build-outs has moved down the stack from accelerators to the boring hardware that keeps a hall lit: generators, switchgear, transformers. A warrant is a cheap way to buy supply priority without buying the supplier, and it converts a purchase order into equity upside for the vendor — which is why a generator maker's stock moves 40% on a supply deal.
What to watch: whether Anthropic, Google or Meta answer with their own power-supplier warrants, or start buying generation outright.
Is locking up the power grid the most important AI deal of the year? Tell us in the comments.
Sources: MLCommons — MLPerf Inference v6.1 results · MLCommons — Agentic Inference for MLPerf Inference · mlcommons/inference_results_v6.1 (GitHub) · HPCwire · arXiv 2609.16338 — Breaking the 1.58-bit Barrier for Ternary LLMs · Hacker News discussion · CNBC — Amazon obtains right to buy up to $340M of Generac · StreetInsider — Generac strikes $8B supply deal with Amazon
