AI 101 — What is RLHF?

Reinforcement learning from human feedback (RLHF) is the training technique that teaches an AI model what people actually prefer — by using human ratings, not just text, as the signal for what counts as a good answer.

It is the reason a chatbot says "I'm not sure, let me check that" instead of confidently inventing an answer, and the reason modern assistants are polite, follow instructions, and refuse harmful requests. Before RLHF, a large language model was essentially a brilliant autocomplete: it knew how to string words together, but it had no sense of whether an answer was useful, safe, or what you actually asked for. RLHF is the step that turned that autocomplete into an assistant.

Why it matters right now

Pre-training — feeding a model the internet and letting it learn patterns — is hitting a wall. Data is running out and the compute bill keeps growing, so every frontier lab has shifted its focus to what happens after pre-training: the fine-tuning stage where a model's behavior is shaped. RLHF sits at the center of that shift, and it has become the standard way labs align models with what humans want.

The same technique powers most of what you use. OpenAI used RLHF for InstructGPT, the model that became the basis of ChatGPT's launch in late 2022. Anthropic, Google DeepMind, and Meta have all used versions of it. And the current research frontier is full of RLHF's aftermath: newer methods like DPO and GRPO aim to get the same result with less machinery, while researchers race to fix RLHF's known failure modes — models that game their reward signal, and assistants that flatter you instead of telling you the truth.

The mental model

RLHF works in three stages. First, the model is fine-tuned on a small set of human-written demonstrations, so it learns the basic shape of a helpful answer. Second, human evaluators are shown two (or more) responses to the same prompt and asked which one is better — a much easier judgment than writing a good answer from scratch. Those comparisons train a separate reward model, a scoring model that predicts how much a human would like any given response. Third, the original model is fine-tuned again, this time to maximize the reward model's score, using a reinforcement learning algorithm called PPO (Proximal Policy Optimization). A guardrail — a penalty for drifting too far from the original model — stops it from mutating into gibberish just to please the scorer.

The clever part is the middle step. Human feedback is expensive, but a model can only be trained on so many examples. The reward model turns thousands of human preferences into a fast, reusable score — so the AI can be optimized against that score long after the human labelers have gone home.

The kitchen analogy

Imagine you're teaching an apprentice chef to cook. Reading recipes and watching videos gets them partway — that's pre-training. Then a master chef demonstrates a few signature dishes for them to copy — that's the supervised fine-tuning step. But a kitchen has only so many hours of a master's time, so instead of the master tasting every single dish, you train them to develop their own palate: they taste hundreds of pairs of dishes, decide which is better in each pair, and internalize what "good" means. From then on, the apprentice cooks against that internalized taste, refining their food to match it. The risk? A chef who cooks only for the critic's palate might start plating food that looks perfect but tastes hollow — which is exactly the "reward hacking" problem RLHF researchers worry about.

Common misconceptions

"RLHF makes models smarter." It doesn't add knowledge — it shapes behavior. The model doesn't learn new facts; it learns which way of answering people prefer. That's why an RLHF-tuned model can still get facts wrong. In fact, OpenAI's own InstructGPT paper reported that RLHF made the model worse at some tasks and could increase hallucination even as human raters preferred its outputs overall.

"It's the only way to align a model." It's the most famous, but alternatives are now common. Anthropic's Constitutional AI has a model critique its own outputs against written principles instead of human ratings. Direct Preference Optimization (DPO) achieves similar results without a separate reward model. And for math and code, where answers are verifiable, labs increasingly skip human preferences entirely and reward the model simply for getting the right answer.

"The human feedback is the main cost." Human labeling is expensive, but in practice the compute for the reinforcement learning step is often the bigger bill — which is why OpenAI's disclosure that safety monitoring would eat 20 percent of inference compute made waves in the industry.

"RLHF is done — everyone uses something newer." Every frontier model still goes through a preference-alignment step of some kind. The underlying idea — shape a model's behavior using human judgment — is not going anywhere; only the implementation keeps evolving.

Where to learn more

Start with Hugging Face's illustrated walkthrough of RLHF, which takes the three stages apart with clean diagrams. Chip Huyen's "RLHF: Reinforcement Learning from Human Feedback" is the best technical primer for what the research actually showed, including the uncomfortable parts. And if you want the original papers, read OpenAI's 2017 "Deep RL from Human Preferences" (where the idea was born) and the 2022 InstructGPT paper that made it famous.

Related reading: What is fine-tuning? covers the broader post-training stage RLHF belongs to · What are reasoning tokens? explains how reasoning models are trained with a similar reinforcement signal · and the safety compute tax shows what alignment actually costs a frontier lab.

If you could rate every answer your assistant gave you, what would you want it to optimize for? Tell us in the comments.

Sources: Hugging Face — Illustrating RLHF · InstructGPT paper (OpenAI, 2022) · Christiano et al. — Deep RL from Human Preferences (2017) · IBM Think — What is RLHF? · Chip Huyen — RLHF: Reinforcement Learning from Human Feedback