How to — know if RAG is actually doing anything

Judging a RAG system by reading its answers tells you almost nothing. That is the trap: a fluent, confident, well-cited answer can come from a model that never retrieved a single document — and a genuinely grounded one can still fail because the right text never surfaced. The way to know if retrieval is actually doing anything is to stop grading final outputs and start asking two separate questions. Here is how to do that without building a research lab.

RAG stands for retrieval-augmented generation: a system that searches a knowledge base for relevant passages, then hands those passages to a model and asks it to write an answer grounded in them. The whole point is that the answer should rest on evidence pulled from your sources, not on what the model happened to memorize. So "is RAG actually doing anything" really means one thing: is the answer actually grounded in retrieved text, or is the model just performing? (If the concept is new to you, AI 101 — What is RAG? sets the foundation.)

Split the pipeline before you judge anything

A RAG system does two distinct jobs. First it retrieves — searches an index and returns a short list of passages. Then it generates — an LLM reads those passages and writes an answer. These two jobs fail independently, and a final answer can be wrong because either one broke. So before you form an opinion, get the retrieved context: the exact passages that were handed to the model, not just the pretty answer it produced. If the product or framework you use won't show you the retrieved chunks, that itself is a clue you cannot verify what it's doing. (Retrieval depends on a vector index — see AI 101 — What is a vector database? for the mechanics.)

Pull the answer and see if its claims trace to the retrieved text

Take a handful of real query-answer pairs and check the load-bearing claims against the passages that were actually retrieved. Does every specific fact in the answer map to a sentence in the context? If an answer is smooth and plausible but you cannot find its substance in the retrieved text — the model is almost certainly filling in gaps from memory, which is fake RAG. This is the "faithfulness" check in the evaluation literature: is every claim supported by the context, or was it invented? When support is thin, the system is grounding nothing.

Probe the retriever on its own

You also need to know whether retrieval is doing its job as retrieval, regardless of how well the model writes. Build a small set of questions whose answers definitely exist in your knowledge base, and check whether the right passage actually makes it into the top results. If the relevant chunk never surfaces, every downstream answer is doomed — no amount of prompt tweaking can recover a fact the model was never given. Poor retrieval recall is a ceiling on the entire system; it tends to show up as a lot of confident wrongness that feels like a model problem but is really a search problem.

Add questions that have no answer in the corpus

The sharpest single test for fake RAG is the unanswerable question. Include queries whose answer is genuinely absent from your knowledge base. A system that is truly grounded should hesitate, flag uncertainty, or refuse rather than guess. A model coasting on its own knowledge will happily confabulate — and often fabricate a citation to support it. If you get confident, source-referencing answers to questions your sources cannot answer, retrieval is decoration, not grounding.

Audit the citations, not just the sentences

This is the one most people skip, and it's where fake RAG gets exposed. If the answer cites a source, verify that the cited passage (a) actually exists and (b) was actually retrieved — not merely referenced. A meaningful share of citations in retrieval-augmented agents are hallucinated: the model names a source it never touched, sometimes one that never existed at all. In a grounded pipeline the citation set is the retrieval set. When the two diverge, you've caught the pipeline bluffing. (Loose, invented references are a core form of AI 101 — What is an AI hallucination?.)

Score a fixed, real set instead of judging vibes

Once the spot-checks look sane, make it repeatable. Pull a fixed list of 50–100 real questions from your logs — easy ones, multi-part ones, and the unanswerable kind — and run the same checks on them each time you change anything. Judge answers on a small rubric: does the answer address the question, and is every claim traceable to the retrieved context? These map to the standard metrics (faithfulness, context recall, answer relevance), but you don't need the jargon to run them. The point of a frozen set is that you can re-run it and see a number move instead of trusting a half-remembered impression from last week.

Don't do this

Don't grade RAG by final answers, and don't trust the "RAG-powered" label. The single worst move is set-it-and-forget-it: shipping a claim of grounding once and never checking whether retrieval is actually happening on real traffic. Equally bad is testing only with friendly questions that have obvious answers — that flatters the model's memory and hides whether retrieval contributed anything at all. If you stop at "the answers sound right," you'll never know which broken half caused every failure.

How you'll know it worked

When you've done this, you can point at any bad answer and say which half of the pipeline failed — retrieval missed the text, or generation ignored it — instead of shrugging. Your system will refuse the unanswerable ones instead of fabricating sources. And you'll have a re-runnable set where a regression shows up as a number that moved, not as a nagging feeling. That's the difference between a demo and a system you can actually trust.

Have you caught a RAG-based tool citing a source it never retrieved? Tell us in the comments.

Sources: The original RAG paper · Ragas: metrics for evaluating RAG · Detecting fabricated citations in RAG agents · What building my first RAG app taught me (HackerNoon) · TruLens groundedness