Skip to content
Velloreth
Artificial Intelligence 10 min read

Fine-Tuning vs Prompting: When Each Approach Wins

A practical decision guide comparing fine-tuning and prompting for LLM tasks — what each actually changes, cost tradeoffs, and a checklist for choosing.

By

Prompting changes what the model reads. Fine-tuning changes what the model is. That distinction decides almost every choice in this space, and skipping it is why teams spend six weeks fine-tuning a model that needed a better system prompt.

Both approaches optimize the same output, but they fail differently, cost differently, and go stale differently. Here is how to tell them apart before you commit.

What each one actually does

Prompting is everything you control at inference time: system instructions, examples in context, output schemas, tool definitions, retrieval results. The model’s weights never change. You can edit a prompt in five minutes, roll it back in one, and A/B it the same afternoon.

Fine-tuning updates the model’s weights on examples of the behavior you want, typically with a parameter-efficient method like LoRA or QLoRA that trains small adapter matrices while the base model stays frozen. The result is a new model artifact you version, evaluate, and deploy. A first run is measured in hours and hundreds of dollars, not weeks — but the feedback loop is a training job, not an editor.

PromptingFine-tuning
What changesInput contextModel weights
Time to iterateMinutesHours per run
ReversibilityTrivialNew artifact, new rollout
New facts / docsAdd to context (or RAG)Poor fit — needs retraining
Style, format, toneConstrained by instructionsBaked in, works without prompting
Cost per callPays context tokens every callLower token use, higher fixed cost
LatencyLonger prompt, same modelOften shorter prompt, same or smaller model
EvaluationPrompt-level test setTraining set + held-out eval set
Failure modeModel ignores the instructionModel drifts, overfits, forgets

The decision checklist

Run the questions in order. Most projects stop at the first two.

If your problem is…Lean toward
The model does not know your task or rulesPrompting
You need examples of correct outputFew-shot prompting
Facts change, or must be citedPrompting + retrieval (RAG)
Output format keeps drifting despite instructionsFine-tuning
You need consistent domain style/tone at scaleFine-tuning
Prompt is 4,000 tokens of boilerplate every callFine-tuning (move boilerplate into weights)
A small model must do one narrow job wellFine-tuning
Latency budget is tight and prompt must shrinkFine-tuning
Fewer than a few hundred good examplesPrompting
The task itself is still undefinedNeither — spec it first

The recurring pattern: prompting is for knowledge and rules, fine-tuning is for behavior. If what you need changes weekly, it belongs in context. If it should be true on every call without being stated, it belongs in weights.

A note on prompt distillation

There is a useful middle path teams keep rediscovering. If your system prompt has grown into a 3,000-token document of rules, edge cases, and examples — and every call pays for it — that prompt is a fine-tuning dataset in disguise.

Take the prompt plus a few hundred representative inputs, generate outputs with the prompt in place, then train a model on input → output without the long prompt. The model internalizes the instructions. What you get back is a shorter prompt, lower latency, and lower per-call cost, with the behavior you already validated.

You still keep the original prompt as the source of truth for regeneration. Fine-tuning freezes behavior at a moment in time; prompts stay editable.

When prompting is enough (and it usually is)

Start here, and be honest about what “enough” means:

1. Write the spec. One paragraph: what the output is, what it must never contain, what the model should do when it is unsure. If you cannot write this, fine-tuning will not save you — it automates a behavior you have not defined.

2. Add few-shot examples. Three to five good examples of input → desired output do more than paragraphs of description. This is the highest-leverage ten minutes in the whole workflow.

3. Constrain the output. Ask for JSON with a schema, or a fixed template, and validate it programmatically. A model asked for “some JSON” will eventually give you YAML inside a code fence.

4. Measure on a golden set. Twenty to fifty real examples with known-correct answers. Without this number, “the prompt feels better” is your only signal.

SYSTEM = """You extract invoice line items.
Rules:
- One row per line item, never merge rows.
- Dates as YYYY-MM-DD.
- If a field is missing, use null. Do not guess.
- Output only JSON matching the schema."""

def extract(text: str) -> LineItems:
    raw = client.complete(SYSTEM, user=text, schema=LINE_ITEMS_SCHEMA)
    return LineItems.model_validate_json(raw)  # invalid output = retry

Validation plus one retry loop catches more format drift than any amount of prompt prose.

Signs you have hit the prompting ceiling:

  • You are re-explaining the same rule in every message and it still gets ignored.
  • Performance is stuck around a plateau no example addition moves.
  • The prompt is long enough that it noticeably costs or slows every call.
  • The model gets the format right but the judgment wrong — wrong tone, wrong domain assumptions, inconsistent classification of ambiguous cases.

That last one is the real signal. Prompting fixes “did not understand the task.” It struggles with “understood the task and applies it inconsistently.”

When fine-tuning earns its place

Fine-tuning wins on a narrow set of conditions:

Format and style that must be uniform. Support replies in your voice, SQL in your house style, extraction into a fixed schema. Instructions get you 80% there; examples get you 90%; weights get you the consistency across thousands of calls that spot-checking cannot guarantee.

A narrow task at volume. Classifying tickets, tagging intent, scoring risk — a small model fine-tuned on your labels can match a frontier model’s quality at a small fraction of the cost and latency. This is the strongest economic case for fine-tuning: not beating a big model, but replacing one.

Shrinking a bloated prompt. As covered above — if context tokens dominate your bill, moving instructions into weights is a direct cost reduction.

Consistent behavior under a fixed contract. When output is consumed by downstream code, variance is a bug. Fine-tuning reduces it.

What good training data looks like

The dataset decides the outcome far more than the hyperparameters:

  • Quality over volume. A few hundred cleaned, representative examples beat ten thousand scraped ones. Noisy labels teach noisy behavior.
  • Cover the distribution, not the happy path. Include the messy inputs — typos, missing fields, adversarial phrasing — or the model will fail exactly where your users are hardest.
  • Include the refusals. Show what “I cannot determine this from the document” looks like, or the model will confidently invent answers to stay helpful.
  • Split held-out. Keep 10–20% of data out of training and evaluate only on that. Training loss tells you nothing about generalization.
  • Match your eval to production. If real traffic is 80% edge cases, a test set of clean examples will rate a bad model as good.

A practical loop: build the prompt first, get it to a measured baseline, collect the cases where it fails, turn those failures into training examples, and only then train. Fine-tuning a model you have not yet measured with prompting just automates your current mistakes.

What fine-tuning cannot do

Three failure modes account for most wasted projects:

It cannot reliably add knowledge. Facts trained into weights blend with what the model already believes, are hard to verify, cannot be cited, and are painful to update — changing one price means another training run. Facts belong in retrieval; weights belong to behavior.

It cannot fix an undefined task. If the labelers disagree about what correct looks like, the model learns the disagreement. Fine-tuning amplifies specification, including a bad one.

It cannot exceed the base model’s ceiling for open-ended reasoning. A fine-tuned 7B model can beat a frontier model at its narrow task. It will not become a better general reasoner than a model ten times its size. If the job requires broad world knowledge and flexible thinking, prompting a capable model is the correct answer — and often the cheaper one when you count engineering time.

Also worth knowing: fine-tuned models go stale on their own terms. The base model your adapter sits on gets deprecated, a newer base outperforms yours, and retraining on the new base is a project, not a settings change. Budget for it.

The cost shape of each path

The two approaches bill you differently, and teams routinely misjudge which is cheaper:

  • Prompting has near-zero fixed cost and a recurring variable cost. Every call pays for the system prompt, examples, and schema. At low volume it is trivially cheap; at high volume with a long prompt, context tokens can become the dominant line item — especially when most calls need only a fraction of that boilerplate.
  • Fine-tuning inverts this. Training is a one-time cost you pay per version, plus evaluation time. Per-call cost usually drops, because prompts shrink and you may be able to move from a frontier model to a smaller one. The break-even depends on volume: a task running a thousand times a day will amortize a training run quickly; a feature getting fifty requests a week probably will not.
  • The cost people forget is engineering time. Prompting iterations are cheap and reversible, so you can afford many of them. Fine-tuning runs need data prep, evaluation, and a rollout plan, so each iteration is expensive — which is exactly why you want the prompt to have already told you what to train on.

There is also a latency dimension. A shorter prompt means fewer input tokens to process before the first output token, so a distilled model often feels faster even with identical generation speed. For interactive products, that difference is frequently worth more than a few points of accuracy.

Neither number should be guessed: log prompt tokens per call for a week, price your realistic traffic, and compare it against the fully-loaded cost of one training run plus the maintenance of a model artifact. The answer is rarely dramatic, but it is always knowable.

The third option most comparisons leave out

Prompting and fine-tuning are not the only levers, and RAG sits between them for a large class of problems:

Question you are askingRight lever
“Does the model know my rules?”Prompting
“Does the model know my data?”Retrieval
“Does the model apply my behavior consistently?”Fine-tuning
“Does the model know the world as of today?”Nothing you do locally — pick a current model

Most production systems end up layered: a system prompt for rules, retrieval for facts, and occasionally a fine-tuned model for the one or two behaviors that refused to stabilize any other way.

The order that wastes the least time

  1. Spec the task in writing.
  2. Prompt a capable model, with 3–5 examples and output validation.
  3. Build a golden set, record the number.
  4. Fail cases → better examples → re-prompt. Repeat until the curve flattens.
  5. If the prompt is now enormous and expensive, distill it into a fine-tune.
  6. If the number is still not good enough for a narrow, high-volume task, fine-tune with the failure cases as data.
  7. If the number is not good enough for an open-ended task, reconsider the task or the model — not the training method.

Fine-tuning is a strong tool with a narrow job. The teams that get value from it are the ones who exhausted prompting first — not because prompting is sacred, but because the prompt was going to tell them exactly what to train on.

Frequently asked questions

Is fine-tuning the same as training a model?
No. Fine-tuning starts from an already trained model and updates its weights on a smaller task-specific dataset, usually with parameter-efficient methods like LoRA. It is measured in hours and hundreds of dollars, not months and millions.
Can fine-tuning add new knowledge to a model?
It can inject some facts, but unreliably: models blend new facts with existing ones, forget them under further training, and you cannot cite where a fact came from. For knowledge that must be correct or current, use retrieval instead.
How many examples does fine-tuning need?
Behavior and format changes often show results with a few hundred high-quality examples; genuine domain reasoning benefits from thousands. More data rarely hurts, but noisy data hurts immediately — filtering beats volume.
Will fine-tuning make a weak model as good as a frontier model?
For a narrow, well-defined task, a fine-tuned small model can match or beat a much larger general model at a fraction of the cost and latency. For open-ended reasoning, it will not close the gap — that is a capability ceiling, not a data problem.