AI 101 — What is backpropagation?

Backpropagation is the algorithm that lets a neural network learn from its mistakes: it measures how wrong the model's answer was, then works backwards through the network to figure out how much each individual connection contributed to that error. Once it knows who is responsible, the model nudges each connection a little in the direction that would have made the answer less wrong. Do that billions of times and you get a language model.

Almost every claim you read about AI training — a model "learned to code," a lab "burned through a data center," an open-weight release that "beats last year's flagship" — is a claim about backpropagation running at enormous scale. It has been the engine of machine learning since 1986, and it is only now getting real competition.

Why it matters right now

Three reasons. First, it explains what the word "training" means when a lab says a model was trained on trillions of words: not memorizing, but running backpropagation over and over until the errors get small. Second, it is the cost center. Training a frontier model is one forward pass plus one backward pass per example, across every parameter, which is why training runs cost hundreds of millions of dollars and why GPUs and high-bandwidth memory are the most contested hardware in the industry.

Third, and most interesting this week: backpropagation's dominance is being questioned. Sakana AI published work on a method called PC-ALM that trains networks up to 1000 layers deep using only local, layer-by-layer corrections — no global backward pass — on small vision benchmarks including MNIST, Fashion-MNIST, and ResNet-18 on CIFAR-10. The result does not threaten backprop at frontier scale today, and the lab's own stated motivation is neuroscience rather than replacing GPUs. But it attacks a forty-year-old assumption that local learning rules must fall apart with depth.

The one-paragraph mental model

A network is a stack of layers, and each layer is a large table of numbers called weights. Training runs in a loop: the model makes a prediction (the forward pass), a loss function scores how wrong that prediction was, and backpropagation computes a gradient — a number for every weight saying which way to turn it, and roughly how hard, to reduce the error. The algorithm's efficiency comes from the chain rule of calculus, applied in reverse order and reused: instead of re-deriving the influence of an early weight from scratch, the backward pass recycles each layer's answer while passing the error signal to the layer before it. A separate step, gradient descent, then actually moves the weights a small amount downhill. Strictly speaking, "backpropagation" is only the gradient calculation, but people use the word for the whole learning loop. An optimizer such as Adam decides the size and shape of each move; a batch is the group of examples whose errors get averaged before a move.

The kitchen analogy

Picture a restaurant with a hundred stations in a line, each cook seasoning the dish before passing it down. A diner sends the final plate back, untasted and unhappy. The head chef does not fire everyone or rewrite every recipe. She works backwards down the line, asking each station: given the plate you received, how much did your handful of salt ruin it? Too much salt, a squeeze of acid, undercooked — each cook gets a specific, small correction, and the correction for station 40 depends on what station 41 did afterwards. Backpropagation is that backward audit. The chain rule is what makes it exact rather than a guess, and the learning rate is the head chef's judgment about how forcefully to reprimand someone: too harsh and the kitchen thrashes around, too gentle and nothing changes before closing.

Common misconceptions

"The model memorizes its training data." It adjusts weights so that its errors get smaller on average. Memorization is a side effect it sometimes suffers from, not the mechanism — which is why a trained model can produce an answer it has never seen and also repeat a training example word for word.

"Backpropagation is what makes a model good at English." It is a general error-correction algorithm, indifferent to language. It trains image classifiers, protein models, and chess networks the same way. What changes is the transformer architecture it runs on and the data it runs against.

"Gradients find the best possible model." They find a downhill direction, not the bottom of the mountain. Training can settle into a good-enough local basin, and in practice that has been fine — a 2015 analysis by LeCun and colleagues argued the feared local-minimum problem mostly does not bite at scale.

"Newer techniques have replaced it." Not in the fundamentals. Fine-tuning, LoRA, and RLHF all still run on backpropagated gradients; they change the error signal, not the audit.

Where to learn more

The original statement of the method is a short Nature letter — "Learning representations by back-propagating errors" by David Rumelhart, Geoffrey Hinton, and Ronald Williams, published 9 October 1986. Hinton received half of the 2024 Nobel Prize in Physics for the work that led there. For the readable version of what the algorithm can and cannot do, the 2015 LeCun-Bengio-Hinton "Deep learning" review in Nature is still the best four pages on the subject. The deeper history is messier than the legend: Seppo Linnainmaa published the reverse-mode trick in 1970 as a way of tracking rounding errors, and Paul Werbos applied it to neural networks in 1982.

Related reading: AI 101 — What is a transformer? · AI 101 — What is RLHF? · AI 101 — What is LoRA?

Local learning rules have lost to backprop on paper for decades — does one epoch on MNIST change how you read the training-cost story? Tell us in the comments.

Sources: Rumelhart, Hinton & Williams — Learning representations by back-propagating errors (Nature) · LeCun, Bengio & Hinton — Deep learning (Nature) · Sakana AI — Augmented Lagrangian Predictive Coding · arXiv:2605.31022 · The Nobel Prize in Physics 2024