AI 101 — What is a token in AI?

A token is the chunk of text an AI model actually reads and writes — not a word, not a character, but a piece of a word: "un", "believ", "able". Every language model, from the smallest phone-side model to the biggest frontier system, does its work by predicting one token after another. If you understand tokens, you understand why AI bills look the way they do, why some languages cost more than others, and why a model can count the letters in "strawberry" and still get it wrong.

It is also the single most-used word in AI this week that almost nobody explains. Zhipu's first results since its Hong Kong listing showed API revenue up more than 27x, driven by token usage growing over 40x. iFlytek open-sourced two small models carrying a 1-million-token context. Every model card, every pricing page, every agent benchmark is denominated in tokens. The unit is the thing.

The one-paragraph mental model

Text goes into a model through a tokenizer — a small, separate program that chops your message into pieces and looks each one up in a fixed table, its vocabulary. GPT-4's tokenizer has about 100,000 entries; OpenAI's newer o200k table has roughly 200,000; Llama 3's has 128,000. Each entry has an ID number, and the model only ever sees numbers, never letters. The tokenizer is built by an algorithm called byte-pair encoding: start from single characters, find the pair that appears most often in a big pile of training text, merge it into one new token, and repeat until the table is full. Common chunks ("the", "ing", "tion") end up as single tokens. Rare words get split into several. The model then predicts the next token ID, the tokenizer turns it back into text, and the loop repeats — one token at a time, which is why responses stream out word-fragment by word-fragment.

A typewriter with a fixed keyboard

Picture a typewriter whose keyboard has no letters — only 100,000 custom keys, and each key prints a chunk: "the", " ing", "##able", an emoji, half of a rare surname. You cannot type a character that isn't on a key; you can only strike keys. Writing a long word means hitting three or four keys in a row. That is tokenization.

Now the part that surprises people: the keyboard was built by counting what people typed most often. Whoever typed the most got the best keys. In practice, that meant English and other Latin-script languages got efficient single-key chunks, while languages that were under-represented in the training text got many small, awkward fragments for the same sentence.

Why it matters right now

Tokens are the price unit. APIs charge per token, in and out. OpenAI's own guidance estimates one token at roughly four characters, or about three-quarters of an English word — 100 tokens is about 75 words. But that ratio is an average, not a rule: code, JSON, and non-English text all break it. The invisible thinking that reasoning models do before answering is also billed as tokens, which is why a short answer can cost far more than it looks — we covered that in AI 101 — What are reasoning tokens?.

Tokens are the memory unit. A "1-million-token context window" is not a million words — it is roughly 750,000 English words, or considerably fewer if the text is technical or non-English. Judging how much you can paste in requires thinking in tokens, not pages. That is the whole subject of our context-window explainer.

Tokens are not evenly distributed across languages. This is the least-known fact about them. A 2023 NeurIPS paper from Oxford researchers found that the same text translated into different languages can produce token counts up to 15x apart. A companion EMNLP 2023 paper measuring commercial APIs found disparities large enough that speakers of lower-resource languages pay several times more for the same content — and, because models generate one token per step, wait longer for the same answer. Tokenizer design is an economic and fairness decision disguised as an implementation detail.

Common misconceptions

"A token is a word." Roughly three-quarters of one, on average in English. Common short words are single tokens; long or rare words are several; a single space or punctuation mark can be its own token. This is also why models are oddly bad at letter-level tasks — they never see letters, only chunk IDs.

"One word is always the same number of tokens." No. Capitalization and leading spaces change the split: "red", "Red", and " red" can tokenize differently. Numbers are worse — a figure can split across several tokens, which is part of why arithmetic used to be a weak spot.

"All models tokenize the same way." Every model family ships its own tokenizer and vocabulary, so the same sentence costs different token counts on different models. Switching models silently changes your costs and how much fits in the window.

"More tokens means a smarter model." Token count measures length, not intelligence. A bigger vocabulary mostly means fewer tokens for the same text, which is cheaper and leaves more room in the window — not smarter.

Where to learn more

OpenAI's own help page on tokens and how to count them is the clearest short primary source, and it states the character/word estimates providers actually use. Hugging Face's LLM course chapter on byte-pair encoding walks through the merge algorithm with a worked example you can follow without a math background. For the language-fairness angle, the NeurIPS 2023 paper Language Model Tokenizers Introduce Unfairness Between Languages (Petrov, La Malfa, Torr, Bibi) is the foundational study, and its companion project page lets you compare how the same sentence tokenizes across languages and tokenizers — worth five minutes of clicking. The EMNLP 2023 paper Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models (Ahia, Kumar, Gonen et al.) puts price tags on the gap. If you want to feel the difference rather than read about it, any public tokenizer demo will show your own name splitting into pieces.

Related reading: for what tokens cost when a model thinks, see AI 101 — What are reasoning tokens?; for how tokens fill a model's working memory, see AI 101 — What is a context window?; and for the architecture that consumes them one at a time, see AI 101 — What is chain of thought?.

Should AI providers be required to publish how efficiently they tokenize each language? Tell us in the comments.

Sources: OpenAI — Understanding and counting tokens · Hugging Face — Byte-Pair Encoding tokenization · Petrov et al. — Language Model Tokenizers Introduce Unfairness Between Languages (NeurIPS 2023) · Ahia et al. — Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models (EMNLP 2023) · AI Midday — Zhipu's API revenue jumps 27x as China's AI labs prove the token economy