Fine-Tuning & LoRA

The ceiling prompting can't fix

A good prompt plus RAG covers most of what a beginner needs. Fine-tuning earns its cost in three narrow cases: the model needs a voice or format no prompt reliably holds (your exact support-ticket schema, every time), it must get much faster/cheaper by shrinking to a smaller fine-tuned model that matches a bigger one on your task, or it needs knowledge too dense to fit in any context window (thousands of product SKUs with quirky naming). If none of those are true, fine-tuning is solving the wrong problem — go fix the prompt or the retrieval first.

Full fine-tune vs LoRA

Full fine-tune: every weight updates. Most capable, most expensive, and risks catastrophic forgetting — the model gets great at your task and quietly worse at everything else, because gradient steps nudge the whole network.

LoRA (Low-Rank Adaptation): freeze the base model, inject small trainable "adapter" matrices into each layer. You're training a few million parameters instead of billions. Cheaper to run, cheaper to store (one base model + many small adapters, swapped per customer or task), and because the base weights never move, general ability survives largely intact.

What you actually need

A clean dataset of input/output pairs (hundreds to low thousands, not millions — this isn't pretraining), a held-out eval set so you can tell "learned the task" from "memorized the training set," and a baseline comparison against the prompted version. If the fine-tune doesn't beat your best prompt on the eval set, you haven't earned the complexity yet.

Remember this

• Fine-tune last, not first — prompting and RAG are cheaper levers.
• LoRA freezes the base, trains small adapters: cheap, swappable, forgetting-resistant.
• No eval set, no fine-tune — "it feels better" isn't a result.