Ask any AI team how they plan to customise a foundation model for their product and you will get one of three answers: prompt engineering, retrieval-augmented generation, or fine-tuning. Confusion about which one to pick is the source of a staggering amount of wasted engineering time. Fine-tuning gets chosen when RAG would have worked better; RAG gets layered on top of problems that a better prompt would have solved for free; prompt engineering gets reached for in situations where it fundamentally cannot deliver. This guide is an opinionated decision tree. It walks through what each technique is actually good at, where each one quietly fails, and how to combine them in the right order for the kind of problem you are trying to solve.
The three techniques, in one sentence each
Prompt engineering: tell the model exactly what you want, how you want it, and give it the context it needs, all inside the prompt. No training, no infrastructure, no new data storage.
Retrieval-augmented generation (RAG): before generation, fetch relevant data from an external store and inject it into the prompt. Facts live outside the model; reasoning happens inside.
Fine-tuning: actually train the model further on examples of the behaviour you want, adjusting its weights so that the new behaviour becomes native.
Those three sentences describe the entire solution space. Everything else is a blend. The art is choosing which one (or which combination) matches your problem.
Start with prompt engineering, always
Ninety percent of real AI product problems can be solved with good prompt engineering alone. This is an unfashionable claim in 2026 because prompt engineering does not sound sophisticated, but it remains true. Before you reach for RAG or fine-tuning, try this:
- Write out exactly what you want the model to do, in natural language, as though you were briefing a new hire.
- Describe the expected input format, the expected output format, and the edge cases.
- Include two or three examples (few-shot prompting) demonstrating good answers.
- Specify constraints explicitly: tone, length, structure, what to avoid.
- If the task involves reasoning, instruct the model to think step by step before answering.
This is not trivial work. A good prompt for a real production task typically runs 300 to 1000 tokens and goes through dozens of iterations before it is reliable. But it costs nothing beyond iteration time, updates instantly, and is fully debuggable. If you can solve your problem with a prompt, you almost always should.
The limit of pure prompt engineering is when the task requires external information the model does not know (product docs, user data, today's news) or when few-shot examples are not enough to reliably steer behaviour. That is when you reach for RAG or fine-tuning.
When RAG beats fine-tuning
RAG shines when the primary gap between the model and your problem is factual knowledge. "My support bot needs to answer questions from our 800 help articles." "My legal assistant must cite actual case law." "My internal chatbot should know what the company's HR policy says." In all these cases, the model is missing facts, not reasoning ability.
RAG wins over fine-tuning in this category for several solid reasons. First, facts change. Your help articles update weekly; your HR policy changes quarterly; your product docs ship with every release. A RAG pipeline absorbs new documents by re-indexing; a fine-tuned model has to be retrained. Second, RAG produces citations naturally — the retrieved chunks tell the user (and the auditor) exactly where the answer came from. Fine-tuned models cannot cite sources because they have internalised the facts opaquely. Third, RAG is cheaper to build and operate. Fine-tuning a competent open-source model on a meaningful dataset costs thousands of dollars and days of compute. A RAG prototype can be built in a weekend.
The mental model: RAG is an external memory system bolted onto a model. Fine-tuning is an attempt to rewrite parts of the model's brain. The former is lower-risk, easier to reverse, and almost always the right starting point for factual grounding.
When fine-tuning beats RAG
Fine-tuning wins when the gap between the model and your problem is about style, format, or skill rather than facts.
Style and voice. You want every response in your brand voice — a specific tone, rhythm, vocabulary. A well-crafted prompt can approximate this, but fine-tuning on thousands of examples of your voice will produce outputs that are natively on-brand without needing a long prompt every time.
Strict output format. You need every response in a precise JSON structure, SQL dialect, or custom DSL with zero deviation. Prompt engineering can get you 95% reliability on format; fine-tuning on a thousand examples of exactly-correct outputs can push that to 99.9%.
Narrow skill that the base model is not good at. The base model cannot produce valid queries in your company's weird proprietary query language. Or it cannot classify tickets into your 50 custom categories well with examples alone. Fine-tuning on labelled examples teaches it the specialised skill.
Latency or cost constraints. A fine-tuned small model can do in 20 milliseconds what a prompted frontier model does in two seconds, at a hundredth the cost. For high-volume production traffic, this alone can justify the fine-tuning investment.
Safety or compliance. You need the model to refuse certain classes of requests reliably, with zero failure rate, across contexts that vary in wording. Fine-tuning baked-in refusals are more reliable than prompt-based ones.
Notice that none of these require new facts. Fine-tuning is a behaviour-shaping tool, not a knowledge-injection tool. Trying to fine-tune facts into a model is expensive and error-prone; trying to RAG in new behaviours is awkward.
The decision tree
A practical flowchart for choosing.
Is the behaviour you want already achievable with a good prompt? If yes, use prompt engineering. Iterate the prompt until it is reliable. Monitor for drift.
If no, is the gap primarily factual knowledge the model lacks? If yes, build a RAG system. Start simple (basic vector search) and add complexity (re-ranking, hybrid search, agentic retrieval) as your evaluation set demands.
If the gap is behavioural — style, format, niche skill — and you have 500+ high-quality labelled examples? Fine-tune. Use LoRA or QLoRA for cost-effective fine-tuning on open-source models; use managed fine-tuning APIs for closed models.
If you need both factual grounding and behavioural shaping? Combine. RAG provides the facts; fine-tuning shapes how those facts get communicated. This is increasingly the dominant production pattern in 2026.
Still stuck? Consider whether the problem is soluble with AI at all, or whether a deterministic solution (a lookup table, a rule engine, a structured API) would serve you better. Not every problem needs an LLM.
Worked example 1: the support chatbot
Concrete scenarios make the decision tree real. Start with a SaaS company wanting an AI support chatbot that answers questions grounded in 500 help-centre articles.
The task is predominantly factual grounding. The help articles change weekly; the model must cite sources; questions vary widely in phrasing. This is a textbook RAG case. Fine-tuning would be the wrong choice — the facts are the hard part, and they change too often. Pure prompt engineering would fail because the model cannot fit 500 articles into context economically, and it would hallucinate on the gaps.
Implementation: chunk the articles by section heading, embed with Cohere embed v3, store in pgvector, retrieve top 10 chunks per query, re-rank to top 5, prompt a Claude Haiku or GPT-4o-mini with a clear instruction to answer only from the context. Add an evaluation harness of 100 known Q&A pairs to catch regressions on every deploy. Total build time for one engineer: about two weeks.
Worked example 2: the code-review assistant
A second scenario. A company wants an AI code-review assistant that flags style violations specific to their internal conventions. Their style guide has 200 rules, many of them subtle and context-dependent. General-purpose code models produce reasonable but non-conformant reviews.
This is a behavioural gap, not a factual one. The model needs to internalise a style, not look up rules. Fine-tuning is the right answer. Collect 2000 labelled examples of code snippets with expert-written reviews in the target style. Fine-tune a mid-size open-source code model (Llama or Qwen-Coder) using LoRA. Serve on a small GPU. Cost: a few hundred dollars for training, a few hundred a month for hosting. Quality: dramatically better than any prompt-engineered equivalent, and a fraction of the ongoing API cost.
Could RAG help here? Marginally — injecting the style guide into every review prompt gets you partway. But for the subtle judgment calls, fine-tuning wins decisively.
Worked example 3: the legal research tool
A third scenario, the kind that mixes all three techniques. A legal-tech startup builds a research tool that answers questions about case law, in the style of a senior partner, with precise citations.
This is the trifecta. The factual grounding (actual case law) requires RAG over a specialised legal corpus. The style (lawyerly, precise, conservative) benefits from fine-tuning a base model on thousands of example answers in the target voice. The orchestration — when to search, what sub-questions to decompose, how to format citations — requires careful prompt engineering on top.
The stack: a fine-tuned open-source model (or a fine-tuned API variant) as the base; a sophisticated RAG pipeline with re-ranking and hybrid search; a carefully crafted system prompt that defines the persona, format, and safety rails. All three techniques working in concert. This is what a serious vertical AI product looks like in 2026.
The hybrid stack that is winning in 2026
Walk into any mature AI product team in 2026 and you will see roughly the same stack, built from all three techniques layered together.
At the bottom is a foundation model — Claude, GPT, Gemini, or an open-source equivalent — chosen for its reasoning quality and multimodal capability.
Above that sits a prompt engineering layer: a carefully crafted system prompt that defines the assistant's persona, capabilities, constraints, and output format. This prompt lives in version control, has its own test suite, and changes through pull requests like any other code.
Above that sits a RAG layer: one or more retrieval pipelines that fetch relevant context at query time from product docs, customer data, knowledge bases, or live APIs. The retrieved context is injected into the prompt before the model sees the user's actual question.
At the top, optionally, sits fine-tuning: either a fine-tuned base model (for cost reduction) or a fine-tuned specialist model for narrow tasks where the generalist model underperforms.
A chat turn typically follows the flow: user asks, query rewriter (LLM call) refines the query, retrieval runs, results rerank, system prompt + retrieved context + user query go to the model, model generates response, response streams to user. This pipeline has roughly five LLM calls and one or two vector queries for a single conversation turn. It is more expensive than a pure prompt-only approach, but it produces dramatically better quality for enterprise tasks.
Common mistakes at each level
Each of the three techniques has characteristic failure modes. Knowing them saves months of frustration.
Prompt engineering mistakes. Vague instructions that leave too much to the model. Inconsistent few-shot examples that teach the wrong pattern. Prompts that balloon past the useful context size. Not versioning prompts in source control. Not testing prompts against an evaluation set.
RAG mistakes. Using default chunking on complex content. Picking the first embedding model that looks cheap. Skipping a re-ranker. Ignoring hybrid search. Building RAG before you have enough documents to justify it. Not evaluating retrieval precision separately from answer quality.
Fine-tuning mistakes. Fine-tuning for facts. Fine-tuning on too few examples (under 500 is almost always premature). Using bad labels that teach the model to hallucinate confidently. Fine-tuning a frontier model when the task does not justify the cost. Forgetting to test that fine-tuning did not degrade general capabilities.
If you avoid these, you are already ahead of most production teams.
Cost and operational comparison
A rough cost comparison for a hypothetical 100,000-query-per-month product.
Pure prompt engineering with a frontier model. Roughly $500-$2000/month for API calls, depending on prompt length and model choice. Zero infrastructure. Fastest to build. No persistent state.
Prompt + RAG with a frontier model. Same API cost plus vector database ($0-$500/month depending on scale), plus embedding API costs ($50-$200/month), plus re-ranker API costs ($100-$500/month). Needs an ingestion pipeline. Dramatically higher quality for knowledge-grounded tasks.
Prompt + RAG + fine-tuned small model. Infrastructure cost to serve the fine-tuned model ($500-$5000/month depending on traffic and hosting), plus training costs (a one-time $500-$10,000). Much lower per-query cost at scale. Best economics for high-volume production traffic.
Startups and prototypes should lean heavily on the first two. Companies at meaningful scale graduate into the third when the savings justify the operational complexity.
What changes in the next two years
Three trends are worth tracking.
Context windows are growing aggressively — million-token and multi-million-token models are normal in 2026. This reduces the need for RAG on smaller corpora; you can sometimes just stuff the whole corpus into the prompt. RAG still wins on cost at scale and on updateability, but the boundary is shifting.
Fine-tuning is getting easier. Managed fine-tuning APIs from OpenAI, Anthropic, and Google let teams fine-tune without ML expertise. Open-source tooling (Axolotl, Unsloth) makes LoRA fine-tuning of Llama-class models approachable for any competent engineer.
Agentic systems are absorbing all three. Modern AI agents dynamically decide when to retrieve, what to retrieve, and how to format output — effectively doing RAG and prompt engineering automatically based on the task. In time, the manual choreography described in this guide may be hidden behind higher-level agent frameworks.
A rough cost-benefit comparison
Weighed in rough numbers, the three techniques stack up as follows.
Prompt engineering: zero build cost, zero infrastructure, instant updates, full debuggability. Maximum quality ceiling is limited by what can fit in context. Best for most new AI projects as a starting point.
RAG: modest build cost (one to four weeks of engineering), modest infrastructure (vector DB, embedding API, re-ranker), updates as fast as you re-ingest, strong quality for knowledge-grounded tasks. Best for factual grounding over changing data.
Fine-tuning: higher build cost (training data collection, training runs, evaluation harness), higher infrastructure (serving a custom model), slower updates, strong quality for stylistic and skill-specific gaps. Best for narrow high-volume behavioural needs.
Pick in that order. Most teams overshoot and wind up with fine-tuned models when a better prompt would have solved the problem at no cost.
Try prompting first, reach for RAG when facts matter, and fine-tune only when you need a new style or skill the base model cannot express.
The short version
Prompt engineering is the default and handles more problems than people give it credit for. RAG is the right tool when the gap is factual knowledge — changing documents, private data, external context. Fine-tuning is the right tool when the gap is behavioural — style, format, niche skills. Production systems combine all three. Reach for the heaviest tool last, not first. And re-evaluate your stack every six months, because what required fine-tuning in 2024 often became a prompt-engineering job by 2026 as base models improved.