LittleLearner: LLMs can't outgrow their training data
A research team from MPI for Intelligent Systems, ELLIS Tübingen, and ETH Zürich trained language models from scratch on nothing but U.S. elementary-school material — and found that no amount of post-training or clever prompting pushes them past the fifth-grade ceiling their pretraining data set.
Researchers built LittleLearner, three LLMs (0.6B, 1.3B, and 5B parameters) trained from scratch on LittleCurriculum — an 88B-token corpus distilled from FineWeb-Edu through a five-stage filter aligned with Common Core K–5 standards, explicitly excluding concepts, facts, and vocabulary taught above grade 5. Each model ships with a matched "unfiltered" control sharing the same architecture, tokens, and training recipe, so the only difference is the curriculum. The result is a model that is fully fluent — it can chat, explain rain, and solve arithmetic — but draws a hard line in the sand: ask it about quantum entanglement and it answers like a confident ten-year-old. One model famously described Schrödinger's cat as "a cat that has been misbehavin'!"
The headline finding is what the team calls "elicitation, not acquisition." Across their experiments, standard interventions amplified what the curriculum taught but never meaningfully expanded the model's reach beyond it. Scaling helped in-scope performance and extended modestly along the same learning trajectory, but yielded little on problems requiring advanced capabilities outside the exposure. Post-training with GRPO — the same recipe behind many reasoning models — significantly boosted K–5 ability but failed to recover out-of-scope capabilities, even when trained with out-of-scope data. And in-context learning with few-shot prompts and explanations unlocked no new beyond-K–5 reasoning in the 5B model.
Why it matters: the field has long struggled to separate what a model learned from what it merely elicited — whether a new skill was acquired during training or was latent in the data all along. By making the training distribution itself the controlled variable, LittleLearner turns that question into a clean experiment. This is the same trust problem behind MLS-Bench: frontier models can't invent new ML methods: capability claims get murky when you can't audit what the model has seen. The team's sharpest result is the ceiling itself — if post-training can't push a model past its pretraining boundary even with out-of-scope data, then the pretraining corpus is doing far more work than the polishing stages, and data quality deserves more of the credit (and scrutiny) it rarely gets.
What to watch: the team proposes the sandbox as a testbed for RL-driven discovery — whether capabilities can emerge from reward alone when the prior is restricted to K–5 — and for continual learning, where you can literally watch a concept like negative numbers get learned.
If a model can't be pushed past what its training data taught it, what does that say about all the effort spent on post-training? Tell us in the comments.
Sources: LittleLearner project page · LittleLearner (arXiv) · LittleLearner models (Hugging Face)
