मॉडल को केवल सही ही नहीं, बल्कि लोगों की पसंद भी सिखाएँ।
⚠ यह चरण हिन्दी में 3% अनूदित है; शेष कार्ड अंग्रेज़ी में दिखाए जा रहे हैं। कोई गलती दिखी? अनुवाद सुधार का स्वागत है।
पठन प्रगति0%
Card 1 — the gapकार्ड 1 / 5
Correct is not the same as good
Many different answers are all technically right. Only some are helpful.
After instruction tuning a model can follow formats, but it still writes answers that are verbose, preachy, or subtly unhelpful. The problem is that the training signal is "imitate the reference answer". What we actually want is "produce an answer a human would prefer". That is a fundamentally different objective, and it needs a different data type: comparisons.
Analogy
Rating coffee, not writing recipes
Asking annotators to write the perfect answer is like asking them to invent a coffee recipe. Asking them to pick the better of two cups is easy, fast and reliable. Preference data collects "which of these two is better?" — a question humans answer consistently even when they cannot articulate the ideal answer.
SFT (Stage 9) learns from demonstrations: here is the right answer.
RLHF (this stage) learns from preferences: A is better than B, and here is why it matters.
Preference data can capture subtle qualities: harmlessness, concision, honesty under uncertainty, refusal style.
The trade-off is real: aggressively optimising for "helpful" can reduce accuracy unless the reward model is good.
?
अगले कार्ड से पहले एक छोटी जाँच
What kind of training data does RLHF/DPO require that SFT does not?
5 जाँचें
🔒
प्रयोगशाला बंद है
इस सिमुलेशन को खोलने के लिए सभी 5 कार्ड और उनकी जाँचें पूरी करें।
Read the full stage transcript (all 5 cards, no JavaScript required)
Card 1 — the gap
1. Correct is not the same as good
Many different answers are all technically right. Only some are helpful.
After instruction tuning a model can follow formats, but it still writes answers that are verbose, preachy, or subtly unhelpful. The problem is that the training signal is "imitate the reference answer". What we actually want is "produce an answer a human would prefer". That is a fundamentally different objective, and it needs a different data type: comparisons.
Analogy
Rating coffee, not writing recipes
Asking annotators to write the perfect answer is like asking them to invent a coffee recipe. Asking them to pick the better of two cups is easy, fast and reliable. Preference data collects "which of these two is better?" — a question humans answer consistently even when they cannot articulate the ideal answer.
SFT (Stage 9) learns from demonstrations: here is the right answer.
RLHF (this stage) learns from preferences: A is better than B, and here is why it matters.
Preference data can capture subtle qualities: harmlessness, concision, honesty under uncertainty, refusal style.
The trade-off is real: aggressively optimising for "helpful" can reduce accuracy unless the reward model is good.
Knowledge check: What kind of training data does RLHF/DPO require that SFT does not?
a. Pairs of responses ranked by human preference← correct
b. More raw internet text
c. Larger vocabularies
Comparisons are much cheaper to collect than perfect demonstrations, and they capture desirable qualities that are hard to write down but easy to recognise.
Card 2 — the reward model
2. Compress human taste into one number
A reward model is a scoring function trained on preferences.
The reward model (RM) is usually a copy of the base language model with the output head replaced by a single scalar. Given a prompt and a response it outputs a score. Trained on pairs where response A was preferred to B, it learns to rank responses the way your annotators did.
L = −log σ( r(x, y_w) − r(x, y_l) )
The Bradley–Terry pairwise loss: push the preferred answer (y_w) above the rejected one (y_l).
Analogy
The tasting judge
You cannot ship a human judge with every request. So you train a judge who tastes millions of cups in an afternoon. That judge is the reward model — fast, cheap, and only as fair as the tastings it learned from.
Watch outReward hacking
The policy model is an optimiser, and optimisers find loopholes. Give it a reward for "long, confident, well-formatted answers" and you get padding and flattery. Mitigations: KL penalty against the reference model, reward-model ensembles, length normalisation, and frequent re-collection of fresh preference data.
Knowledge check: What is reward hacking?
a. The policy learns to exploit flaws in the reward model rather than becoming genuinely better← correct
b. Annotators disagreeing with each other
c. The reward model running out of memory
A proxy objective is always exploitable. This is why RLHF pipelines include a KL penalty keeping the policy close to the SFT model, plus constant data refresh.
Card 3 — the classic loop
3. RLHF with PPO, step by step
Four models in memory at once — and only one of them is being trained.
1Sample a batch of prompts from the preference dataset.
2Generate responses with the policy model (the one being trained).
3Score every response with the frozen reward model.
4Compute the advantage: reward minus the KL divergence from the frozen SFT reference, to stop drift.
5Update the policy with PPO, using the frozen critic/value model to estimate the baseline.
Model in memory
Role
Trained?
Policy
Generates responses
Yes — the actual target
Reference (SFT)
Anchors the policy via KL penalty
No
Reward model
Scores responses
No
Critic / value
Estimates expected reward for advantage
Yes (usually)
Analogy
A student with a coach and a referee
The policy is the student. The reward model is the referee scoring each attempt. The reference model is the school syllabus that says "do not wander too far from what you already know". The critic predicts what the referee will say. Standard PPO requires the gym to hold all four at once — which is exactly why it is expensive.
Watch outInstability
RLHF with PPO is notoriously sensitive: reward scale, KL coefficient, clipping range, and the number of PPO epochs all matter. It is also prone to generative reward hacking (endless hedging, excessive disclaimers) when the reward model over-rewards caution.
Knowledge check: What is the role of the frozen reference (SFT) model in RLHF?
a. It provides a KL penalty that keeps the policy from drifting too far from sensible behaviour← correct
b. It generates the training prompts
c. It replaces the reward model after training
Without that anchor the policy will happily wander into degenerate regions that exploit the reward model. The KL term is the leash.
Card 4 — the shortcut
4. DPO: alignment without a reward model
Skip reinforcement learning entirely and optimise preferences directly.
Direct Preference Optimization observes something elegant: the optimal policy under a KL-constrained reward objective has a closed form, and the reward function can be rewritten in terms of the policy itself. That means you can train on preference pairs with a simple classification-style loss — no sampling, no reward model, no critic.
Increase the relative likelihood of preferred over rejected answers, moderated by the reference model.
Approach
Models in memory
Stability
Data appetite
PPO / RLHF
4
Fragile, many knobs
Moderate prompts + RM training
DPO
2 (policy + reference)
Much more forgiving
Needs good preference pairs
Constitutional / AI feedback
2
Stable
Can generate preferences with a model
β controls divergence: small β drifts far from the reference, large β barely moves.
DPO is offline: it cannot explore new responses, so the data distribution bounds what it can learn.
Preference quality beats preference quantity, and chosen answers should still be good answers.
Many production assistants are now SFT → DPO, using RLHF only when an interactive reward signal is genuinely needed.
Knowledge check: What is the main practical advantage of DPO over PPO-based RLHF?
a. It needs no reward model or critic, and is far more stable to train← correct
b. It requires no preference data
c. It works without any base model
DPO turns alignment into a supervised-style loss over pairs, which removes the reinforcement-learning machinery and much of the fragility.
Card 5 — the state of the art
5. Beyond direct human preferences
Annotation is slow and biased. Some of the preference signal can be generated.
Reinforcement learning from AI feedback (RLAIF) replaces some human labelling with model-generated critiques guided by a written constitution or rubric. It scales far beyond what a labelling budget allows and makes criteria explicit and auditable — at the cost of inheriting the judge model's biases.
Constitutional AI: the model critiques and revises its own answer against principles, then the revised pairs train the preference model.
Rubric-based grading: graders score answers against explicit criteria instead of a holistic "which is better".
Process reward models: reward each reasoning step rather than only the final answer — better for maths and code.
Verifiable rewards: for tasks with a checkable outcome (unit tests pass, answer parses, citation resolves), replace human preference with a program.
Watch outThe alignment tax and over-refusal
Over-tuned safety makes a model refuse legitimate requests. Measure both harm and helpfulness on held-out sets; a model that refuses everything scores perfectly on a harm benchmark and is useless.
NoteOne stage left
Your model is trained, tuned and aligned. Stage 11 covers the physics of actually serving it: prefill, decode, sampling knobs and the VRAM arithmetic behind every request.
Knowledge check: When is a verifiable reward better than a human preference model?
a. When the outcome is mechanically checkable, like unit tests passing or a JSON schema validating← correct
b. When the task is creative writing
c. Never — human preference is always superior
Programmatic checks give an exact, unfakeable signal. Where such a signal exists, use it — reserve preference models for qualities you cannot verify.