Instruction-Tuning (LoRA)

A base language model was trained on one task: continue the text. Ask "What is the capital of France?" and a raw base model might respond with a list of more exam questions, because that is what tends to follow such a line on the internet. It has the knowledge; it lacks the format of helpfulness.

Supervised fine-tuning (SFT) trains on datasets of instructions paired with ideal responses, often including a system prompt and multi-turn structure. Crucially, the loss is only computed on the response tokens, not the instructions — you do not want to teach the model to write your prompts back to you.

# Chat template with masked prompt tokens
messages = [
    {"role": "system", "content": "You are a concise ML tutor."},
    {"role": "user", "content": "Explain gradient descent in one sentence."},
    {"role": "assistant", "content": "Repeatedly step downhill on the loss surface ..."},
]

ids = tokenizer.apply_chat_template(messages, return_tensors="pt")

# labels = -100 on the prompt/system tokens -> zero loss contribution
labels = ids.masked_fill(prompt_mask, -100)

Worse, every fine-tune produces a full copy of the model. Serving ten customer-specific variants means ten 14 GB checkpoints. This is the practical wall that parameter-efficient fine-tuning (PEFT) was invented to break.

LoRA (Low-Rank Adaptation) starts from an empirical observation: the change a fine-tune makes to a weight matrix has low intrinsic rank. So instead of updating a d×d weight matrix W directly, freeze W and learn a correction ΔW = B·A, where A is r×d and B is d×r with r as small as 8 or 16.

• A is initialised randomly, B is initialised to zero, so ΔW starts at zero and the model begins as the original model.
• Target modules matter: adapting all attention projections and the MLP is the standard recipe; attention-only is a cheaper subset.
• Merging works because the correction is a plain matrix add: W₀ + BA folds LoRA away for zero inference overhead.
• Multiple adapters can share one base model, and switching them costs only a load — the reason multi-tenant fine-tuned serving is practical.

QLoRA quantises the frozen base model to 4-bit and keeps LoRA adapters in higher precision. The base is never trained, so its quantisation error is a fixed cost, not something you must accumulate gradients through. This is how a 7B or even 13B model can be fine-tuned on a single 24 GB GPU.

from peft import LoraConfig, get_peft_model
from transformers import BitsAndBytesConfig

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype="bfloat16",
)

model = AutoModelForCausalLM.from_pretrained(base, quantization_config=bnb)

config = LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj"],
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, config)   # ~0.5% of params now trainable

• Quality over quantity in SFT data: a few thousand curated, consistent examples beat 100k noisy ones.
• Consistent formatting matters more than raw volume — the model imitates your template exactly.
• Learning rate: LoRA tolerates ~10–100× higher rates than full fine-tuning (1e-4 to 2e-4).
• Watch for catastrophic forgetting: heavy fine-tuning on one narrow style degrades general ability. Mix in a little general data, or lower alpha.

• Always start with the cheapest option that could plausibly work: prompting, then RAG, then PEFT.
• A fine-tune cannot add knowledge reliably, and retrieval cannot reliably change style. Use both when you need both.
• Keep an evaluation set of 50–200 real cases. Without it you cannot tell whether any of this helped.
• Version everything: base model, adapter, prompt template and evaluation set are one artefact.