Sutton: synthetic data is 'a huge mistake' — LLMs are a quarter of intelligence
Richard Sutton, the reinforcement learning pioneer who wrote the field's most-quoted essay, used a Sequoia Capital interview to take direct aim at the industry's scaling playbook — calling synthetic data "a huge mistake" and arguing large language models capture only about a quarter of what intelligence is.
The reinforcement learning pioneer behind "The Bitter Lesson" — the maxim that algorithms and compute beat hand-encoded human knowledge — sat for a Sequoia Capital conversation alongside Khurram Javed, his co-founder at a new startup called Oak Lab. Sutton's verdict on synthetic data is blunt: labs are exhausting real human text, so they're manufacturing synthetic text to keep training — and he calls that a big error. His "big world hypothesis" argues the world is effectively infinite in complexity, so any synthetic set is a thin slice of it, and producing good synthetic data bottlenecks on human expertise. You still need a domain expert to design convincing training data — engineers at OpenAI or Anthropic on vacation means synthetic data production halts. It's the Bitter Lesson's contradiction reappearing in a new costume.
His second complaint is that today's models are, in his words, dead the moment they ship: the weights never change while you use them. Give a model more context and it uses it, but it isn't learning. Javed points out that even tools that do update weights, like code autocompletion, retrain on millions of users' activity in bulk — diluting any single person's specific need. Sutton frames this as the gap between a system that holds knowledge and one that keeps learning from experience.
That's where Oak Lab comes in. The pair is building on continual backprop, an algorithm they published in Nature: each weight keeps its own learning rate so new data doesn't wipe old knowledge (the catastrophic forgetting problem), and the network continuously spawns fresh randomly-initialized neurons to grow new capacity. Their goal is a foundation model that learns while it runs. Sutton isn't dismissive of LLMs — he calls them a genuine, unexpected breakthrough in language — but he insists fluent language is maybe 20 to 25 percent of intelligence, and that marketing a non-learning system as near-complete intelligence is the strange thinking, not his.
What to watch: whether Oak Lab's continuous-learning bet shows early signs within the year — and whether any large lab quietly adopts per-weight learning rates.
Does Sutton's case for learning-in-use convince you, or is synthetic data still the faster road? Tell us in the comments.
Sources: Pinfu Atlas — Sequoia interview transcript · MIT Technology Review (Chinese) · MSN Tech
