The Frontier

AI 101 — What is multimodal AI?

Multimodal AI is a model that handles more than one kind of input — text, images, audio, video — inside the same system, rather than one model per sense stitched together. "Modality" is just the academic word for a channel of information. Text is a modality. A photograph is a modality. A voice recording is a modality. Multimodal AI reads more than one of them at once and reasons across them.

That last part is the whole trick. You have had software that reads images for years — optical character recognition has been turning scanned pages into text since the 1990s. What is new is a single model that can look at a chart, read the numbers in it, and answer a question about the trend, without anyone writing a rule for charts.

Why it matters right now. Multimodal capability stopped being a headline feature and became table stakes, which is why it keeps appearing in our coverage as an adjective nobody stops to explain. This month Tencent released open-weight models built to search scanned documents by what they look like, preserving tables and layout that a text-only pipeline flattens — Tencent's EVIE turns a scanned page into something you can actually search. ByteDance's Doubao update advertises "multimodal coding" that reads design mockups and screen recordings and turns them into front-end code. Unitree's open humanoid brain is built on Alibaba's Qwen3-VL-4B, and its whole job is turning camera views into movement. None of those products would exist if a model could only read text.

Doctor analyzes lung x-ray wearing a mask in a clinical setting.

The mental model

Text and images are different formats, so how does one model hold both? It converts everything into the same thing: numbers in a shared space. Text goes in as tokens, each one turned into a list of numbers. An image is chopped into a grid of patches, and each patch is turned into a list of numbers too. Once a picture of a dog and the word "dog" both live as coordinates in the same space, the model can compare them directly — and the machinery that does the comparing is the same attention mechanism inside a transformer.

That shared space is the pivot. It is also why the field's first real breakthrough was an embedding model rather than a chatbot: OpenAI's CLIP, published in February 2021, was trained on roughly 400 million image-and-caption pairs to pull matching pictures and text closer together, which let it classify images it had never been trained on. Everything since is scale and polish on that idea.

The kitchen analogy

Picture a restaurant kitchen. A text-only AI is a cook who works from written orders only: every dish request arrives as a note, and if a customer points at something, the note is all the cook ever sees. Multimodal AI is a cook standing in the pass — reading the ticket, watching the plate come back half-eaten, hearing the dining room get loud when a dish lands. Same kitchen, same skill, far more information to work with.

Now the important distinction: there are two ways to staff that pass. You can hire a reader, a taster and a listener and have a manager relay messages between them — that is a pipeline, and it is how early systems bolted vision onto a language model. Or you can hire one cook with eyes, ears and taste. Google called its December 2023 Gemini models "natively multimodal" precisely to claim the second: trained on several modalities from the start, not assembled afterwards. OpenAI's GPT-4o, announced in May 2024, made the same claim from the other direction — accepting any combination of text, audio, image and video and generating text, audio or images back.

Both approaches work. The pipeline is cheaper and easier to debug. The native model tends to handle the seams better — it can notice that the tone of a voice contradicts the words, or that a table's layout carries meaning a transcript destroys.

Common misconceptions

"Multimodal means multiple models." Not necessarily. A native model has one set of weights handling every channel. A pipeline is several models with a translator between them. Both are called multimodal, and the difference shows up in the failure cases.

"If it accepts an image, it's multimodal." Most chat products that let you attach a photo route it through a separate vision model first. That is fine, and it is not the same thing as a model that reasons over image and text jointly.

"More senses means smarter." Adding a modality adds a way to be wrong. Vision-language models hallucinate objects that are not in the picture the same way text models invent citations — often more confidently, because a plausible-sounding description of a photo reads as evidence. A model that sees your x-ray is still a model, not a radiologist.

"It's about generating images." Separate capability, and one that trips people up. Multimodal usually means understanding multiple inputs. Generating pictures is a different skill set built on the same shared-space idea — the embedding that makes "dog" and a dog comparable is also what makes a text prompt steer an image generator.

Where to learn more

The CLIP paper is readable and free, and it is the cleanest explanation of why everything lives in one numeric space. For the native-versus-bolted-on distinction, Google's Gemini announcement and OpenAI's GPT-4o page both state their design choices plainly. If you want the open-source branch, the LLaVA paper from April 2023 is the recipe most open vision-language models still follow: a vision encoder wired to a language model, trained on instruction data.

Should a model that reads your medical scan be held to the same standard as a doctor who misreads it, or is the human who acted on the output always responsible? Tell us in the comments.

Sources: OpenAI — CLIP: Learning Transferable Visual Models From Natural Language Supervision · Google — Introducing Gemini, natively multimodal · OpenAI — Hello GPT-4o · Ars Technica — OpenAI announces GPT-4 · LLaVA — Visual Instruction Tuning