MLPerf Storage starts measuring inference, not just training
Two releases this week attack the same blind spot: almost everything we can measure about AI infrastructure was built for training, and almost all the money is now being spent on inference.
MLCommons shipped MLPerf Storage v3.0, and the headline is that the suite now tests inference workloads. Alongside the training and checkpointing tests it has run for years, v3.0 adds a KV cache test that measures storage performance for the read/write churn of LLM inference caching, and a vector database test built on an open-source Milvus deployment holding 10 million vectors of roughly 1,500 dimensions each, indexed with DiskANN so the index lives on disk rather than in RAM. The KV cache test runs in three modes — storage-only for Llama-3.1-8B, that 8B workload plus a 4 GiB DRAM cache for hot entries, and a storage-only 70B mode whose cache objects are 2.5 times larger — and it attracted 20 submissions. The vector test has four results so far, which tells you how new the ground is.
That framing matters more than the speeds do. Co-chair Curtis Anderson described the point plainly: a buyer knows how many concurrent conversations they need to run, and the benchmark's job is to tell them what size of storage system supports that number. For eight months every major storage vendor has been shipping some scheme for spilling KV caches out of HBM and DRAM onto disk, all of them claiming to break through the "GPU memory wall" where agentic responses slow to a crawl. Now there is a common ruler, and rulers change purchasing. The suite also added an S3 object storage access layer next to the existing POSIX one — usable for training, checkpointing and some vector tests, and already exercised by about one-sixth of this round's 19 submitters, 11 of them first-timers including Azure, NVIDIA, Nebius and XSKY. The most quietly useful number in the release may be power: on-premises checkpointing writes came in at a median of 14 GB per second per watt, with a best of 201.
Qdrant published the largest open vector search benchmark anyone can actually download — 10.07 billion documents, about 24.5 TB of vectors, with the correct answers computed and released. Building the dataset was the easy half. Establishing ground truth for more than 100,000 queries took over a quadrillion brute-force distance calculations, run on GPUs, to work out the exact top-1,000 nearest neighbours across the full corpus. Qdrant built the set with Vultr from Hugging Face's FineWeb corpus using an Alibaba multilingual embedding model, and it is also releasing two smaller companions — a PubMed multi-vector set with dense, sparse and ColBERT-style representations over the same text, and a multimodal set built from image-caption pairs.
The interesting part is the tooling. Supernova, the framework Qdrant used, is open source and handles all four stages — embedding generation, ground-truth computation, loading, and load testing — and it can drive Qdrant, Milvus and Elasticsearch, not just its sponsor's product. Yes, a vector database vendor has opinions about what a good vector benchmark measures, and the 10-billion headline is aimed at competitors. But most vendors publish a benchmark they win; this one published the answer key and the code to check it, which makes it falsifiable. That is a higher bar than the field usually clears. If you want the audit routine, How to — tell a real benchmark from a marketing one walks through it.
What to watch: whether the vector database test fills out in the next round, and whether S3 access becomes the default submission path as context lengths push past what filesystems comfortably handle.
Which inference bottleneck deserves a standard benchmark next — cache offload, agent tool latency, or power per token? Tell us in the comments.
Sources: Supernova (GitHub)
