Prompt, RAG or fine-tuning: how to tell what your AI project needs

You have tried an AI model for a task at work and the result falls short: it makes up facts about your company, it does not follow the format you need or it does not sound like your brand. When you look for solutions, three terms keep coming up: prompt, RAG and fine-tuning. They are not equivalent alternatives. Each fixes a different kind of problem, and choosing the wrong one is expensive.

First, diagnose the problem

OpenAI's guide to improving model accuracy offers a useful distinction. When a model fails, it is usually for one of two reasons:

• It is missing information. It does not know what it needs to know: your prices, your internal policies, what happened last week. This is a context problem.
• It does not behave as you need. It has the information, but answers with the wrong format, tone or reasoning. This is a behaviour problem.

With that diagnosis, the choice becomes much clearer.

Option 1: improve the prompt

The prompt is the instruction you give the model. Improving it means explaining the task better, giving context, showing examples of what you expect (known as few-shot) and fixing the output format. We explain this in detail in How to write a prompt that works the first time.

When to use it: always, and always first. Both OpenAI and Anthropic recommend starting here before thinking about retraining a model. It is immediate, costs nothing but your time and can be changed in seconds.

What it solves: format, tone, level of detail, reasoning steps and a surprising share of quality problems. If you add two or three examples of good answers to the prompt, the model picks up the pattern without any training: this is in-context learning.

Where it falls short: it cannot give the model information it does not have, and if you need hundreds of examples to pin down a behaviour, the prompt becomes huge and expensive.

Option 2: RAG, giving it the information it lacks

RAG (retrieval-augmented generation) is a technique in which, before answering, the system searches your documents for the relevant passages and passes them to the model along with the question. The model answers from that information, not from its memory.

When to use it: when the problem is context. Your manuals, your knowledge base, your catalogue, internal rules or any information that changes frequently.

What it solves: answers based on your real data, updatable without touching the model (you just update the documents) and with the ability to cite where each answer comes from.

Where it falls short: it requires preparing the documents well, splitting them sensibly and evaluating whether the search finds the right passages. And it is often unnecessary: if your documents fit in the context window, you can simply paste them in. We analyse this in depth in RAG without the hype: when you need vector search and when you don't.

Option 3: fine-tuning, changing the behaviour

Fine-tuning means continuing to train an existing model on your own input and output examples so that it adopts a behaviour permanently. Techniques such as LoRA, part of the PEFT (parameter-efficient fine-tuning) family, make this possible by modifying only a small part of the model, which greatly reduces the cost.

When to use it: when you have already got the most out of the prompt and the problem is still one of behaviour. For example, a very specific output format the model does not follow consistently, a very particular writing style or a classification task repeated thousands of times where you want to use a smaller, cheaper model.

What it solves: consistency on bounded tasks, shorter prompts (the behaviour is already learned) and, sometimes, the ability to use a smaller model with results close to a large one.

Where it falls short: you need a quality set of examples, which takes work to prepare; every change means retraining; and it is not a good way to teach new facts. A model fine-tuned on your documents "remembers" them imprecisely and can keep making things up, without telling you where each claim comes from.

The most expensive confusion

The most common mistake is trying to solve an information problem with fine-tuning: "let's train the model on all our documents so it knows them". It usually ends in a costly project that is slow to update and gives unreliable answers. If information is what is missing, the answer is to put it in the context, either by pasting it or with RAG.

The reverse also happens: building a complex RAG system when the problem was that the prompt did not explain the answer format clearly.

In what order to try

1. Write a good prompt with clear instructions, context and an output format. Measure the results on a small set of real cases.
2. Add examples to the prompt if the problem is format or style.
3. Add information to the context if the problem is missing data: first by pasting the documents; if they do not fit or there are many, with RAG.
4. Consider fine-tuning only if, after all of the above, the behaviour is still inconsistent and your usage volume justifies the effort.

The techniques are not mutually exclusive: a customer service assistant can use a carefully written prompt, RAG over the knowledge base and a fine-tuned model to keep to the brand's tone. If you need to structure the decision with your team, the Decide with criteria template helps you put it in writing.

Keep reading

• How to write a prompt that works the first time: always the first step.
• RAG without the hype: when you need vector search and when you don't: when building RAG pays off.
• Open or closed AI models: which models you can fine-tune yourself.
• Why AI makes things up (hallucinations): why missing information and hallucinations go together.
• In the glossary: fine-tuning, RAG, LoRA and in-context learning.

Sources

• OpenAI. Optimizing LLM accuracy. OpenAI API documentation.
• OpenAI. Fine-tuning best practices. OpenAI API documentation.
• Anthropic. Prompt engineering overview. Claude documentation.
• Lewis, P. et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv.
• Hu, E. et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv.