RAG without the hype: when you need vector search and when you don't

RAG is sold as the way to connect a model to your information. It is a specific solution to a specific problem: the model does not have your data and the data does not fit in the context. When the problem is something else, it adds infrastructure and a new source of errors. Before you build a vector index, rule out the alternatives that usually solve it better.

What problem RAG solves

RAG retrieves relevant fragments from a collection and puts them in front of the model so it can answer with them. It is useful when the knowledge is private, changes over time or is too large to include in full.

It does not help the model reason better, follow a procedure or choose well between options. If the information is already in the prompt and the answer is still wrong, the problem is the instructions or the model's capability, not retrieval.

The important change is that you move the failure. Before, you had one system that generated badly. Now you have two: one that picks fragments and another that writes. The retrieval one fails silently, with the confidence of plausible text.

The three alternatives people skip

Full context

If your corpus is a product manual, an internal policy or a small, stable catalogue, paste it in whole. No index, no chunking, no reindexing. Fewer parts, fewer failures.

The limit is not technical, it is economic and about attention: you pay for every call and the model can lose detail in the middle of a huge context. But a long prompt with the right section usually beats a mediocre search engine. Measure before assuming you need infrastructure.

Keyword search

Your database already knows how to search text. A full-text index handles product codes, proper names, legal references and contract numbers better than any embedding, because it looks for exactly what the user typed.

It is cheap, explainable and easy to debug: you can see why a result appeared. Where it fails is with paraphrases and synonyms, and that is where vector search does add value. Combining both is less glamorous and more effective than replacing one with the other.

Structured data

If the question is how many, which ones, how much it cost or what state something is in, the answer lives in a table. RAG over tables turns an exact query into a written approximation.

The solution is a query, an API or a dashboard. If the user needs natural language, add a text-to-SQL layer with validation and permissions, not a document index. Most "chat with our data" projects are reporting projects with a different interface.

How to decide

• The question fits in a stable manual. Business signal: a few pages that rarely change. Start with full context.
• It uses codes, names or exact references. Business signal: the user copies and pastes the term. Start with keyword search.
• It asks for figures, statuses or aggregates. Business signal: the answer is in a table. Start with a structured data query.
• It paraphrases long documents that change. Business signal: support, legal, technical documentation. Start with RAG.
• It needs to cite the source. Business signal: the risk of making things up is high. Start with RAG or a search engine with citations.
• It crosses several documents. Business signal: comparing contracts, synthesising incidents. Start with RAG plus strict evaluation.

A useful test before writing any code: take twenty real questions and try to answer them by hand, pointing to the exact document. If you cannot name it, neither can the search engine. If you need three documents and a subtraction, RAG is not the problem you have.

Chunking: where the project breaks

The most expensive mistake is not choosing the wrong embedding model. It is indexing the wrong corpus: old versions, drafts, scanned PDFs with dirty OCR, duplicates that fill the top results.

• Split by structure, not by character count. A section with its heading is worth more than three fragments that start mid-sentence.
• Store metadata: source, section, date, version, permissions. Without a date, the system will cite the repealed policy with complete confidence.
• Retrieve small and return the parent context. The fragment is for finding; the full section is for answering.
• Fix tables and lists before indexing them. A price table split into loose rows produces nonsensical answers.
• Deduplicate. Five copies of the same paragraph do not add information, only less variety in the top results.
• Exclude what should not be answered: internal notes, personal data, obsolete documents.

Overlap between fragments is a patch for bad splits, not a strategy. If you depend on it for the answers to make sense, the problem is in the boundaries.

Evaluate retrieval and answers separately

They are two systems with two ways of failing. If the right fragment does not show up, no prompt can save it. If it shows up and the answer ignores it, the problem is generation.

For retrieval, build a set of questions with the document that should come up. Measure whether it is in the top results and in what position. Fifty real cases are enough to see the trend.

For the answer, check three things: whether it is grounded in the sources, whether it is complete and whether it cites the right document. Include questions that have no answer in the corpus. The most revealing metric is how often the system admits it does not know.

Do not combine both metrics into a single number. An overall eighty per cent can hide sixty for retrieval and ninety-five for writing, and the fixes are opposite: one calls for a better index, the other for better instructions.

When not to build it

• The corpus fits in the context and rarely changes.
• The question asks for an exact figure that lives in a database.
• Nobody is going to maintain the corpus: RAG amplifies documentation debt, it does not solve it.
• There is no way to evaluate or review answers: you will have a demo that sounds good and is wrong without warning.
• The content changes every hour and you cannot reindex: query the source directly.
• An error has legal or financial consequences and there is no verifiable source: hand it to a person.

RAG does not make the model smarter. It just gives it better papers. If you choose the wrong papers, you will get a better-written lie.

The right question is not whether you can build RAG, but whether your problem is access to information. If the corpus is large, changing and in natural language, go ahead. If not, you have just saved yourself an index nobody was going to maintain.

Keep reading

• Prompt, RAG or fine-tuning: telling a context problem from a behaviour problem.
• Tokens and the context window: how much fits before you need RAG.
• Summarise long documents with AI: working with long documents without an index.
• AI agents: how to design one that does not fail in production: when retrieval is one tool among several.