Deep Dive — Latent-space pretraining survived 5.7 trillion tokens

For as long as large language models have existed, one rule has been treated as structural rather than optional: predict the next token, and let structure emerge from the statistics. A technical report posted to arXiv on September 9 breaks that rule at a scale where earlier attempts fell apart — and the numbers attached to it are why the open-source community has spent two weeks arguing about a base model nobody can chat with.

The paper is NCP-ArchPreview, from the Intern-NCP Team at Shanghai Artificial Intelligence Laboratory and the LUMIA Lab at Shanghai Jiao Tong University. It trains an 8.9-billion-parameter model on 5.73 trillion tokens of the fully open Dolma-3 corpus while also asking the model to predict discrete concepts that span several tokens at a time. The result the team leads with: the model reaches the final pretraining loss of OLMo-3-7B after spending only 51.3% of the token budget, equivalent to converging 1.95 times faster, with a final loss 0.091 lower than the baseline. The Stage 2 annealing phase still ran 1.51 times faster than the same baseline.

That is not a demo on a toy corpus. The report frames it as the largest latent-space language model trained to date, and the team released the weights, the evaluation code, and the intermediate checkpoints from 100,000 steps through 1.2 million — the training trajectory, not just the destination.

What concept prediction actually does

The architecture is a three-stage pipeline inside a conventional causal Transformer. The bottom 16 layers are a token encoder. Every four consecutive tokens' hidden states get mean-pooled into one continuous concept representation, shrinking the sequence the middle of the network sees to a quarter of its length. An 8-layer Concept Module then predicts the next concept autoregressively, and the top 16 layers decode back to tokens, with the predicted concepts expanded to token resolution, shifted causally by exactly the chunk size so no future token leaks backward, and fused into the decoder residually.

The vocabulary is where the engineering gets interesting. Rather than maintain a flat codebook of impractical size, the model splits each concept vector into 32 segments, each matched to its own sub-codebook of 128 entries. That product quantization yields a concept space with 128^32 nominal combinations for a trivial parameter cost, and it is built from the model's own hidden states rather than bolted on from a separate encoder.

Read plainly, the model learns to sketch an outline while it writes. Next-token prediction remains the interface — generation and serving look like any other autoregressive checkpoint — but during pretraining the model is also graded on whether it anticipated the meaning of the next four tokens, not just their identities. The team attributes the gains to both halves of that: ablations credit the latent architecture and the concept objective separately, and hierarchical residual routing between the three stages costs 0.051% extra compute for a 0.0323 systematic loss reduction.

The compute-aligned comparison is the one that should interest anyone paying for training runs. Against a 40-layer Transformer baseline at matched size, NCP-ArchPreview approaches the baseline's training loss while consuming about 85% of its compute, and the team's scaling-law fits across compute budgets point to a 1.74-times improvement in the compute Pareto frontier. Not a halving of cost — a structural gain, in a field where structural gains are rare.

The downstream numbers, and their ceiling

Lower pretraining loss is a proxy, so the report also runs the model. Across a macro-average of roughly 30 benchmark categories, Stage 1 lands 2.45 points above OLMo-3-7B. The gains are concentrated where reasoning and code are graded: GSM8K rises 5.99 points, from 39.27 to 45.26; GSM-Symbolic adds 3.95; Minerva adds 3.11; HumanEval climbs 4.28 points, from 27.10 to 31.38; the multilingual MultiPL-E MBPP measure adds 5.29.

Read those absolute numbers twice, because they set the ceiling on the claim. A 45% score on grade-school math and a 31% score on HumanEval are respectable for a 7B-class open model and nowhere near frontier. The honest headline is not "frontier capability on half the compute." It is "a better open 7B-class base model for a smaller pretraining bill" — which, at the price of a modern pretraining run, is still worth arguing about.

Two secondary results are, in practical terms, the most transferable things in the paper. The first is a fine-tuning interface that falls out of having a discrete concept vocabulary: freeze all 8.9 billion backbone parameters and update only the quantization module and the concept head, about 17 million parameters. On GSM8K and MATH-500 that raised the average from 30.56 to 34.83, a gain of 4.27 points, ahead of an equally sized LoRA at 3.15 — and general-ability retention went up 0.39 points, where LoRA lost 0.42 and full fine-tuning lost 0.97. Throughput was measured at 15,632 tokens per second per GPU on eight GPUs, 2.02 times full fine-tuning and 1.50 times LoRA, with memory at 51.6% of a card against LoRA's 87.9% in a configuration where full fine-tuning simply ran out of memory.

The second is inference. Injecting the model's chunk-level concept representations into a DFlash2 speculative drafter added roughly 40,000 parameters to the 1.1-billion-parameter drafter and no extra compute to the target model, and improved mean accepted length by 4.17% overall — 7.59% on HumanEval. Concept-level prediction helping a serving stack is not something the efficiency literature usually produces.

What skeptics should say, and what they should not

The obvious objection is that "51.3% of the tokens" is not "half the cost," and the paper's own compute-aligned result proves it: the matched-size comparison consumed 85% of the baseline's compute, not 50%. Tokens are the input to the training bill, not the bill. The 1.95-times convergence figure is a token-budget ratio; the 1.74-times figure is the one that reflects actual compute. Both are good. Only one of them belongs in a headline.

The second objection is precedent. Latent-space language modeling has a graveyard, and the most instructive occupant is Meta's Large Concept Models paper from December 2024, which predicted autoregressively in a sentence-level SONAR embedding space and did not beat token-level baselines at comparable scale. NCP-ArchPreview is the first credible claim that the idea survives multi-trillion-token pretraining rather than collapsing into scaling failure — that is the actual news here, more than any single benchmark delta.

The third is scale. The concept vocabulary, the four-token chunk size, and the 16/8/16 layer split are all choices tuned at 8.9 billion parameters. Latent objectives have a habit of looking efficient until the concept space runs out of resolution, and nothing in the report establishes that the recipe holds at 70 billion or a trillion parameters. The team is upfront about the messy parts — the report documents attention-logit explosions and gradient spikes when a Muon-style matrix optimizer was paired with layer-level query/key normalization, resolved with per-head normalization, and it describes a 100-million-token proxy used to pick annealing recipes, correlating with downstream capability at an R² of up to 0.999. Documentation of failures at this level of detail is unusual, and it is the strongest evidence that the numbers were not selected after the fact.

Finally, this is a base model. There is no instruction tuning, no reinforcement learning stage, no chat behavior — so no evidence yet that the efficiency survives post-training, which is where a pretraining loss advantage usually gets spent.

Why it matters beyond one lab

Pretraining is the single largest line item in AI, and it is the item the build-out thesis rests on. Every efficiency result compounds: a cheaper convergence path does not reduce demand for compute so much as change what a given budget can buy, and the labs with the best recipes get to spend the difference on the next, larger run.

The pattern here is recognizably Chinese. Chinese labs have made algorithmic efficiency a visible part of their releases — DeepSeek's attention and quantization work, Moonshot's serving economics, and Tencent's Apache 2.0 Hy4 preview, whose sparse attention with an index cache is another attempt to make each token cost less. Constrained access to the fastest accelerators is one reason. Talent concentration in architecture research is another. The result is a body of work that treats the training recipe as the deliverable, not a footnote.

That lands at a moment when the economics of the whole sector are inverted: we covered the scissors earlier this month in Deep Dive — Tokens halved in price. The bill to make them didn't, where the selling price of intelligence fell 29% in a single month while memory and accelerator costs went the other way. Efficiency work is the only lever that closes that gap without a price war. And demand keeps arriving to absorb whatever gets cheaper — as Zhipu's 27-fold API revenue growth showed, cheaper tokens get spent, not saved.

So the real stake in this paper is not one leaderboard. It is whether the pretraining recipe stops being a settled question. If concept-level objectives stack on top of token-level ones at larger scales, the labs that only buy compute are running the more expensive version of the same race.

What to watch: whether anyone outside the two Shanghai institutions reproduces the 51.3% convergence figure; whether the next InternLM-generation release ships this objective in the mainline recipe; whether vLLM or SGLang adopt concept-injected drafters, since that result transfers to serving regardless of who wins the architecture argument; and the only test that settles it — a 70-billion-parameter or larger run, where the concept vocabulary either holds its resolution or quietly stops paying for itself.

Would you rather your lab buy more GPUs or rebuild the objective — and does a base-model result like this change that answer? Tell us in the comments.

Sources: NCP-ArchPreview Technical Report (arXiv) · NCP-ArchPreview weights and stage checkpoints (Hugging Face) · NCP evaluation code (LUMIA-Group, GitHub) · Machine Heart (机器之心) · Pandaily · Meta — Large Concept Models (arXiv)