Kids learn language on a sliver of an LLM's data — we still don't know why
A child picks up its mother tongue after hearing maybe 100 million words. A frontier model needs trillions. MIT Technology Review takes a long look at the divide researchers call the data-efficiency gap, why it's quietly becoming the industry's biggest problem, and what closing it could unlock for everyone who can't afford to hyperscale.
A child masters its mother tongue with maybe 100 million words of exposure, while the best language models burn through trillions — and nobody can yet explain why. Stanford cognitive scientist Michael C. Frank puts the mismatch bluntly: we still have to "burn down a forest" of data to reproduce, in a machine, what happens in a living room in a year. Meta's open-weight Llama 3.1 alone consumed 15 trillion tokens in pretraining, and frontier labs may be using about ten times that. The uncomfortable direction of travel is that there's only so much internet to train on — researchers suggest the well of easily available data could start to run dry as early as the 2030s. A child proves the ceiling isn't the wall to beat: a preteen in a language-rich home has heard roughly 100 million words, and GPT-2, trained on just 30 million, produces nonsense, not a toddler.
The most concrete progress on the question is coming from a small annual competition called BabyLM. It challenges teams to train a language model on a developmentally plausible corpus of just 100 million words (10 million for the toddler-scale track) drawn from storybooks, dialogue and transcripts of real child-directed speech, then gauges their grammar on benchmarks usually aimed at humans. The results undercut a tidy intuition: curriculum learning — starting simple and ramping up — was the field's most popular approach and simply didn't do much, suggesting transformers don't need neatly ordered data the way we'd guess. More striking, the 2024 winner, GPT-BERT, trained on about 100 million words, beat Meta's Llama 2 70B — pretrained on roughly 15,000 times that much data — on a BabyLM grammar benchmark.
The story's promise runs deeper than efficiency. Researchers are using LLMs as a kind of model organism for their own discipline, testing long-standing theories about whether language learning needs an innate grammar or can emerge from experience alone. Some see baby-scale data as a path to democratizing AI — a way for universities and minority-language communities, where a language like Sami may only have tens of millions of tokens available, to train capable models without hyperscale budgets. Others are chasing the embodied route: Princeton's Uri Hasson has spent five years recording the first 1,000 days of 17 children's lives with cameras and microphones, and a 2024 model trained on 61 hours of a toddler's headcam footage learned to link words to objects with no built-in biases. The frontier labs, for now, aren't racing to borrow tricks from children — the bet is that the next generation of models, the thing that eventually replaces the transformer, will take the lesson.
If a child's 100 million words can beat a model fed 15 trillion, is scale the wrong lever after all? Tell us in the comments.
