AI 101 — What is temperature in an LLM?
Temperature is the setting that controls how randomly an AI model picks its next word: near 0 it always takes the most likely word, higher values give unlikely words a real chance, and 1.0 is the default on most APIs. It is the first knob most people turn, and the one most people misunderstand.
Why it matters right now. Temperature has been showing up in the news all week as a variable, not a feature. A Lasso Security study found that switching on SynthID-Text watermarking changed how models behaved — 16.8% of phi-4's tool-call verdicts flipped at a temperature of 1.0 against a net accuracy loss of 2.87 points, and across 21 model-temperature combinations churn averaged 6.5%, which we covered when it landed. The same week, the practical advice for anyone building an eval set was to pin the exact temperature you plan to ship, because a score measured at one setting says nothing about another. Both stories assume you know what the knob does. This is the ten-minute version.
The mental model. A language model never "writes a sentence." At each step it produces one score for every token in its vocabulary — often well over 100,000 numbers, called logits — and a function called softmax turns those scores into probabilities that sum to 1. Temperature divides the scores before that conversion. Divide by a small number and the gaps between scores get exaggerated, so the leader wins by even more. Divide by a big number and the gaps flatten, so tokens the model considered unlikely get a genuine shot. Then one token is drawn from that distribution, glued onto the text, and the entire process runs again for the next token. So temperature does not change what the model knows. It changes how boldly the model bets on what it knows.
The dice analogy. Picture the model handing you a set of loaded dice, loaded exactly to its own confidence. At temperature 0 the die has effectively one face, and the favorite always comes up. At 1.0 you roll the model's honest odds. Crank it to 1.5 or 2.0 and you are shaking the table before every roll: the long shots show up far more often, which is occasionally inspired and frequently nonsense. The model that confidently writes a clean function at 0.2 is the same model that wanders off mid-paragraph at 1.6. Nothing about its knowledge changed — only how willing it was to gamble on a less likely word.
What people get wrong. The biggest one is believing temperature 0 means the same answer twice. In practice it rarely does. One 2024 paper ran five models across eight tasks and ten runs each at settings meant to be deterministic and found accuracy varying by up to 15% between runs, with a gap of up to 70% between the best and worst run — and none of the models produced repeatable results across every task. The cause is not the sampling any more. It is batching and floating-point arithmetic: the same request served alongside a different set of requests can land on a very slightly different number, and near a tie that difference flips the token. Provider APIs add the seed parameter and a fingerprint field to nudge this back toward reproducibility, but "mostly consistent" is the honest promise, not "identical."
The second mistake is treating temperature as a quality dial. Higher is not smarter or more creative in any graded sense; it is more varied, and past a point the variation is just confabulation. Lower is not more accurate either — it is more conventional, which is usually right for code and math and usually dull for prose.
The third is confusing temperature with its neighbors. The top-p setting, from the 2019 nucleus sampling paper, keeps only the smallest set of tokens whose probabilities add up to p and renormalizes; top-k keeps a fixed count. All three reshape the same distribution, which is why provider guidance is to tune one and leave the others alone rather than stack them.
The fourth is assuming the number travels. A default of 1.0 on one API is not the same behavior as 1.0 on another, because each model's training and post-training changed its underlying odds. DeepSeek's own documentation recommends 0.0 for coding and math, 1.0 for data cleaning, 1.3 for general conversation and translation, and 1.5 for creative writing — vendor advice for one family of models, not a law of nature. And some reasoning models do not expose the knob at all, running at a fixed setting because their thinking step already covers the ground temperature used to.
Where to learn more. OpenRouter's parameter reference is the clearest one-page tour of temperature, top-p and top-k with their ranges and defaults, and DeepSeek's parameter page is the shortest real-world example of a lab publishing recommended values by task. For the reproducibility problem, Thinking Machines' write-up on nondeterminism in inference is the best technical account available, and the OpenAI cookbook entry on the seed parameter explains what the fingerprint field is actually for.
Related reading: What is a token in AI? explains the units temperature is choosing between, What are reasoning tokens? covers the models that often fix sampling altogether, and What is AI watermarking? is the provenance layer whose behavior the new churn numbers measured.
If temperature 0 can't guarantee the same answer twice, should AI products promise reproducible output at all — or is telling users to expect variation the more honest option? Tell us in the comments.
Sources: OpenRouter — API parameters · DeepSeek — The temperature parameter · arXiv — Non-Determinism of "Deterministic" LLM Settings · Thinking Machines Lab — Defeating Nondeterminism in LLM Inference · arXiv — The Curious Case of Neural Text Degeneration (nucleus sampling) · OpenAI Cookbook — Reproducible outputs with the seed parameter
