Prompting, Retrieval or Fine-Tuning: Pick the Cheapest Fix First

Three engineers hit the same bug: the model keeps getting one thing wrong. One rewrites the prompt, one bolts on a vector store, one starts collecting training examples. All three fixes can work. Only one of them is the cheap fix for the failure you actually have, and guessing wrong costs weeks.

The three levers, and what each one really changes

Each lever changes something different. That difference should drive your choice.

Prompting changes what the model does with the information already in its context window: instructions, output schema, ordering, a few examples. It is text you edit in a file. The limit is context itself — the more rules you stack, the weaker each one gets, and prose rules compete with rules shown by example.

Retrieval changes what enters the context window at request time. It fixes missing, stale or too-large knowledge. It does not fix reasoning, tone or output shape. If the model never saw the fact, no amount of prompt work will produce it reliably.

Fine-tuning changes the weights. It fixes behavior that is consistent across many inputs but hard to state: a house style, a domain tagging scheme, a classification taxonomy. It does not reliably add facts. Fine-tuning to teach facts is the most expensive way to get fluent wrong answers.

Useful frame: every lever has a different unit of change. A prompt is a file you edit. Retrieval is an index you rebuild. Fine-tuning is a training run, a new artifact, an eval pass and a redeploy.

Here is what the cheap end looks like:

# prompts/refund_policy.yaml
version: 7
model: gpt-4.1-mini          # pinned, never "latest"
temperature: 0
system: |
  Answer refund questions using only the policy inside <policy>.
  If the policy does not cover the case, say so and offer a human handoff.
  Never state a refund window that is not written in <policy>.
examples:
  - user: "I bought this 40 days ago, can I get a refund?"
    assistant: "The policy allows refunds within 30 days of purchase, so this is outside the window. I can pass it to a human for a case-by-case review."

Those two lines under examples are a miniature version of fine-tuning. If one example demonstrates the rule, use the example.

Cost, latency, maintenance and staleness

The staleness row is the one people skip. A prompt and an index survive a model swap. A fine-tuned artifact does not: when the base model is retired, your training run is orphaned and you start over. That is a recurring cost, and it is easy to miss when you are only comparing setup effort. The same discipline that keeps failed automations from piling up applies here — see triage habits for automation that fails while you sleep.

Latency cuts the other way. Retrieval adds a round trip before generation, and long prompts add input tokens to every call. Fine-tuning is the only lever that can shrink your prompt, which is why it comes up in cost conversations — but only after you already have a stable task and a working eval set.

{"messages":[{"role":"system","content":"Classify the ticket into: billing, bug, feature_request, churn_risk."},{"role":"user","content":"Invoice shows two charges this month"},{"role":"assistant","content":"billing"}]}

That one line is a file you now version, split, hold examples out from, and re-run whenever the base model changes. Know that before you write a thousand of them.

Symptoms that point to each lever

Match the symptom, not your mood.

Prompting symptoms:

• The model knows the answer but returns the wrong shape: prose when you asked for JSON.
• One instruction inside a long system prompt is consistently ignored.
• Tone is off, or the model refuses cases it should handle.
• The same input produces different formats across runs. That usually means the instruction is ambiguous, not that the model is weak.

Retrieval symptoms:

• Answers are right for common cases and wrong for anything specific to your product.
• The model says it lacks information about something you documented.
• It quotes outdated specifics: a retired price, an old policy window, a renamed SKU.
• Hallucinated details that look plausible, which is the classic sign the fact was never in context.

Fine-tuning symptoms:

• You have restated the same rule several times in the prompt and it still drifts.
• Output must follow a taxonomy or formatting convention that is tedious to describe and easy to demonstrate.
• A high-volume classification or extraction task where prompt tokens dominate the bill.
• Consistency across many generations matters more than per-case nuance.

A rough decision function:

def next_lever(symptom: dict) -> str:
    if not symptom["fact_was_in_context"]:
        return "retrieval"       # prompt work cannot invent the fact
    if symptom["format_drift"] or not symptom["rule_is_statable"]:
        return "prompting"       # tighten the instruction, add one example
    if symptom["same_rule_restated"] and symptom["volume_is_high"]:
        return "fine_tuning"     # last, and only with a held-out eval set
    return "prompting"

Why cheapest-first order almost always wins

The order is prompting, then retrieval, then fine-tuning, because each step costs more to build and much more to undo. Prompting is a file edit you can revert in a second. Retrieval is a few days of chunking and query tuning, and it still survives a model swap. Fine-tuning is a training pipeline plus an eval set plus a re-run when your provider moves, and it locks you to one base model.

Cheapest-first does not mean “exhaust prompting before doing anything else.” It means do not spend retrieval budget on a format bug, and do not spend fine-tuning budget on a knowledge gap. Change the smallest thing whose failure mode matches your symptom, then stop when the symptom is gone.

Two ways people skip ahead and regret it. First, fine-tuning to add facts: the model learns the shape of your answers, not the content, and you get confident answers with wrong details. Second, retrieval as a formatting fix: you build a vector store and the model still returns prose, because the problem was the output schema. If your prompt has grown into a stack of accumulated patches, that is a signal to restructure it, not to add another lever — the same tradeoff as deciding between one big prompt and a chain of steps.

What to do next

1. Write the failure down as a symptom before choosing a lever: “the fact was never in context,” “the shape is wrong,” “the rule is consistent but unstated.” The symptom picks the lever.
2. Capture a small eval set from real logs — a couple dozen cases with expected outputs — and keep it out of any training data. Without it you cannot tell a fix from a regression.
3. Pin the model version and the prompt version, and log both with every request. You cannot compare anything if the model changed silently underneath you.
4. Enforce structured output with a schema before you consider fine-tuning. Format drift is a prompting problem far more often than a training problem.
5. Add retrieval only once you have confirmed the fact is missing from context, and log the retrieved chunks so failures stay diagnosable. If you are wiring that pipeline for the first time, the ultimate AI automation guide walks through a working setup.

If you would rather start from a working baseline than a blank file, the AI Developer Stack Bundle is $79 one-time and bundles a prompt library, agent framework, deployment tooling and tutorials that are pre-configured to work together. It does not decide which lever you need, but it removes the setup tax once you have.

Get AI Developer Stack Bundle

AI Developer Stack Bundle — $79, one-time payment, instant download. See the full breakdown on the review page.

Related reading

• The Ultimate Guide to AI Automation 2026
• How I Built a 10-Product Digital Business
• Building an AI Content Factory