AI 101 — What is RAG?

RAG — retrieval-augmented generation — is a technique that lets an AI model look things up in a document store before it answers, by pulling the most relevant passages into the conversation. Instead of relying only on what it memorized during training, the model gets the right pages placed in front of it and writes its answer with those pages open. It is the standard way companies connect chatbots and agents to their own data.

Why it's in the news

RAG rarely makes headlines on its own — it's the plumbing. But it keeps surfacing in everything the AI industry is building. This week, the leading open-source RAG engine, RAGFlow, crossed 88,000 GitHub stars while repositioning itself as an "agentic context layer": the idea that retrieval is no longer a search box bolted onto a chatbot, but the memory system agents consult as they work. When you read about a company "connecting its chatbot to internal docs," or an agent that "grounds itself in your codebase," RAG is the mechanism doing the lifting. The concept itself is not new — it was named in a 2020 Facebook AI Research paper, "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" — but it has quietly become the workhorse of enterprise AI, and the current wave of agent products is making it more central, not less.

The mental model

A large language model is trained once on a huge body of text, and that's it — its knowledge is frozen at training time. RAG adds a second ingredient: a searchable copy of documents that can be refreshed at any moment. The documents are split into chunks, and each chunk is converted into a vector — a long list of numbers that captures its meaning — so similar passages sit close together in a vector database. When you ask a question, the system converts your question into a vector too, retrieves the closest chunks, and places them in the prompt alongside your question. The model then writes its answer with those passages inside its context window — everything it sees before generating a single word. Two jobs, one pipeline: retrieval finds the right pages, generation writes from them.

An everyday analogy

Imagine a brilliant student who writes excellent essays but has never read your company's files. Ask her to summarize the 2026 vacation policy from memory and she will confidently invent something plausible — the AI equivalent of a hallucination. RAG is the research assistant who hands her the three most relevant pages from the filing cabinet before she starts writing. She still does all the writing, and her raw talent is unchanged — but now every claim traces back to a page she was actually given. That is the whole trick: not a smarter writer, a better-supplied one.

Common misconceptions

"RAG makes the model smarter." It doesn't touch the model's weights — the knowledge and reasoning baked in during training are identical. What changes is the material the model gets to read. It's the difference between writing from memory and writing with sources open.

"RAG is just a vector database." Vector search is the heart of it, but a production RAG pipeline also handles chunking (splitting documents into retrievable pieces), hybrid keyword-plus-semantic retrieval, reranking the candidates, and assembling the final prompt. That is why tools like RAGFlow describe themselves as full context layers rather than search indexes.

"RAG eliminates hallucinations." It reduces them sharply and makes answers traceable — but the model still generates freely. If the retrieved passages are wrong, outdated, or missing, the answer will be too: grounded garbage is still garbage. Worse, the passages themselves can be hostile — retrieved documents have become an attack surface, because instructions hidden in a retrieved page can hijack the model (we explain the mechanics in What is prompt injection?). And there is a subtler limit: even with the right pages in front of it, a model doesn't use them all equally — a 2023 study found LLMs reliably use the beginning and end of long contexts while losing the middle.

"RAG and fine-tuning are competing options." They solve different problems. Fine-tuning changes the model itself — its style, behavior, or specialized skills. RAG changes what the model reads. For facts that change — prices, policies, support tickets — RAG wins, because you update the documents instead of retraining the model. Most serious systems end up using both.

Where to learn more

The original 2020 paper by Lewis and colleagues is short, readable, and where the name comes from. IBM's explainer walks through the pipeline step by step. And if you want to see a real system, RAGFlow's repository is a working, hugely popular example of a modern RAG stack — chunking, embeddings, reranking, the works.

Related reading: RAG is a big reason people download and run models locally — private documents never leave the machine. What are open-weight models? explains how that ecosystem works. And once the retrieved passages are in the context window, the model's reasoning over them is its own process — What is chain of thought? looks at the thinking that happens after the lookup.

If a chatbot ever gave you a confident, wrong answer, RAG is the fix — and the reason it can still happen. What's the worst hallucination you've caught an AI assistant making? Tell us in the comments.

Sources: Lewis et al. — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks · IBM — What is retrieval-augmented generation? · Liu et al. — Lost in the Middle: How Language Models Use Long Contexts · RAGFlow (GitHub) · AI Midday — Open Source Radar: August 14