Introduction
In the last post we covered fine-tuning, including a quick look at preference tuning. This post goes deep on that third stage of the LLM pipeline — RLHF (Reinforcement Learning from Human Feedback) — because it’s the least intuitive and the most-asked-about in interviews.
We’ll build up from scratch: why supervised fine-tuning isn’t enough, what reinforcement learning is, how Q-learning and DQN work (and why they don’t work for LLMs), how PPO fixes that, and how all the pieces combine into RLHF. Then we’ll look at a hands-on example with a tiny model, and finish with DPO — the simpler alternative. DBA analogies are sprinkled throughout.
What you’ll learn:
- Why SFT alone can’t make a model helpful, harmless and honest
- Reinforcement learning basics: agent, environment, action, state, reward
- Q-learning → DQN → PPO, and why PPO is the one used for LLMs
- The “cast of characters” in RLHF: policy, reward, value and reference models
- The complete RLHF pipeline, step by step, with worked numbers
- How DPO, GRPO and ORPO compare
Prerequisites
- [ ] Read Posts 3 and 5 (the 3-stage LLM pipeline and fine-tuning basics)
- [ ] Comfort with the idea of a neural network learning by adjusting weights
- [ ] Optional: a GPU notebook (e.g., Colab) to run the hands-on section
Part 1: Why Do We Need Preference Alignment?
Recall the pipeline: pretraining → supervised fine-tuning (SFT) → preference tuning.
- Pretraining is next-token prediction. The model becomes fluent and knowledgeable, but its training data can contain incorrect or misleading information, and it doesn’t reliably follow instructions. In the words of the class notes: pretraining makes the model knowledgeable, but not controllable and trustworthy.
- SFT teaches how to follow instructions and the expected format of answers.
So what’s missing? Look at what SFT actually shows the model:
User: "What is the capital of France?"
Model is taught: "Paris is the capital of France."
That’s all it sees — only the correct answer. Now in the real world someone asks about Germany, and the model could produce either “Paris is the capital of Germany” (wrong) or “Berlin is the capital of Germany” (right). SFT only taught that “Paris is correct for France.” It never taught that “Paris is wrong for Germany.”
More generally, SFT does not explicitly teach:
- which answers are better than others,
- which answers are unacceptable,
- human values like politeness, safety, tone, or helpfulness.
So during SFT the probability of good answers goes up — but the probability of bad answers can also go up by accident.
Preference alignment is the process of training a model to generate outputs that humans prefer. It bridges the gap between “knows language” and “behaves well.”
A concrete example: for the prompt “How do I prepare for exams?”, one response says “Make a study plan, revise daily, and take proper breaks to stay consistent.” Another says “If you’re not studying 10 hours a day, you’re going to fail.” Both are fluent. One is calm, structured advice (preferred); the other is overly harsh and unsafe (rejected). This matters most where behavior matters as much as correctness: healthcare assistants, finance tools, education platforms, enterprise copilots.
Alignment aims for a model that is helpful, harmless, honest, polite, safe, non-toxic, and consistent with human expectations.
DBA analogy: SFT is like loading only passing test cases into a regression suite. The system learns what correct looks like, but never sees a failing case to avoid. Preference data adds the failing cases, with a verdict on which result is better.
Part 2: Reinforcement Learning in Plain English
The idea
Reinforcement learning (RL) is a learning method where an agent learns through trial and error by interacting with an environment and receiving rewards or penalties.
The loop: State → Action → Environment → Reward → Update → Repeat.
| Component | Meaning | Game example | Self-driving car |
|---|---|---|---|
| Agent | The learner | Player | The car’s driving model |
| Environment | Where the agent acts | The game world | Road traffic |
| State | Current situation | Position in level | Camera view |
| Action | What the agent does | Move left/right | Brake / accelerate / turn |
| Reward | Good or bad feedback | +10 win, −10 lose | Safe driving |
A real-life version: a baby learning to walk. Walks correctly → parents clap (positive reward). Falls → pain (negative reward). Over time the baby learns which actions are good.
How it differs from supervised learning
| Supervised learning | Reinforcement learning | |
|---|---|---|
| Data | Labeled data | No labels |
| Feedback | Immediate (correct/incorrect) | Delayed reward |
| Goal | Predict the correct output | Maximize total reward |
| Learning style | Direct | Trial and error |
| Example | Image classification | Game playing, ChatGPT alignment |
Supervised learning has a teacher present; RL has no teacher, only a reward system.
What does “learning” change in RL?
In ML/DL, learning means updating weights to reduce a loss. In RL, the model has a policy π(a|s): the probability of taking action a in state s. Learning means changing those probabilities — good action → probability goes up; bad action → probability goes down. The intuition behind PPO-style updates: update ∝ reward × log(probability of action). High reward → make that action more likely; low reward → make it less likely.
Deep RL is simply RL powered by neural networks — and it’s the form behind AlphaGo and ChatGPT’s alignment.
A short timeline
1950s: foundations (Markov Decision Processes) → 1980s–90s: classical RL with Q-learning and tables → 2013–15: Deep Q-Networks (DQN) bring neural networks to RL → 2016: policy-gradient and actor-critic methods → 2017: PPO (stable training, widely used) → 2020–22: RL meets LLMs (RLHF).
Part 3: Q-Learning → DQN → PPO
RL algorithms fall in two families: value-based (learn how good each action is — Q-learning, DQN) and policy-based (learn the action probabilities directly — PPO).
Q-learning: a lookup table
Imagine a treasure hunt. You’re at a crossroad with three actions. Q-learning stores the expected future reward for each (state, action) pair in a Q-table:
| State | Action | Q-value |
|---|---|---|
| Crossroad | Go Left | 10 |
| Crossroad | Go Right | 50 |
| Crossroad | Go Straight | 20 |
The agent picks the highest value — Go Right. Simple and exact — but it only works for small, discrete problems.
The Q-table explosion: if the state is a 1000×1000 camera image, the number of possible states is in the billions. You’d need a row for every state and a column for every action. Impossible to store.
DBA analogy: a Q-table is a lookup table keyed by (state, action). It works until the key space explodes — the same reason you can’t index every possible combination of high-cardinality columns.
DQN: replace the table with a neural network
A Deep Q-Network replaces the table with a neural network: input = state, output = Q-values for all actions. It generalizes instead of memorizing.
Worked example from class (a robot in a grid, goal: reach +10):
- Q-network gets state S1 and outputs Q(S1,Left)=1, Right=4, Up=2, Down=0 → picks Right
- Environment gives reward +5 and next state S2
- Store the experience (S1, Right, +5, S2) in a replay buffer
- During training, a separate target network looks at S2 and finds the best future value: max = 6
- Target = reward + γ × best future value = 5 + 0.9 × 6 = 10.4
- Current prediction was 4, so loss = (10.4 − 4)² = 40.96
- Update weights; over training Q(S1,Right) moves 4 → 7 → 9 → 10.4
Note that DQN is not supervised learning — there are no labeled answers; it learns from trial and error with rewards as feedback.
Why DQN doesn’t work for LLMs
DQN predicts a value for every possible action and picks the best. That’s fine when actions are few and discrete (left, right, jump, shoot). But for an LLM, a single “action” is choosing the next token from 100,000+ possibilities — and a sentence is an enormous combination of those. Estimating a Q-value for every option is impractical.
The contrast from the notes:
- DQN asks: “What is the Q-value of every action?”
- PPO asks: “How should I slightly adjust my behavior to get higher rewards?”
PPO: learn the policy directly
PPO (Proximal Policy Optimization) is policy-based and uses an actor-critic design with two networks:
- Policy network (actor): state → probabilities of each action
- Value network (critic): state → expected future reward V(s)
The “proximal” idea: don’t change the policy too much at once. Bad approach: 70% → 100% in one jump. Good PPO approach: 70% → 75%. Small safe updates keep training stable.
Worked example from class (robot in a grid):
- Policy network outputs P(Left)=0.10, P(Right)=0.60, P(Up)=0.20, P(Down)=0.10 → picks Right; store old_prob = 0.60
- Environment gives reward +5
- Value network had predicted V(S1) = 3 (expected)
- Advantage = actual − expected = 5 − 3 = +2 → Right did better than expected
- Policy update: P(Right) goes 0.60 → 0.72. Ratio = new / old = 0.72 / 0.60 = 1.2
- Clipping: if the allowed range is 0.8–1.2, a ratio of 1.2 is fine. If the probability had jumped to 0.90, the ratio would be 1.5 — too aggressive, so it’s clipped
One subtle point from the notes: clipping doesn’t directly stop weights from changing. It shapes the loss function so that once the ratio exceeds the limit, there’s no extra benefit from pushing further; gradients shrink, weight updates shrink, and the probability can’t jump too much in one step.
PPO is not supervised learning: it learns from reward signals, not ground-truth labels.
DBA analogies:
- Advantage = actual − expected is exactly the optimizer’s estimated vs. actual comparison. If actual performance beats the estimate, reinforce what you did.
- Clipping is like throttling a schema migration or rate-limiting a bulk update: make progress in controlled increments instead of one dangerous big-bang change.
Part 4: RLHF — Putting It Together for LLMs
RLHF is a training process that uses reinforcement learning (typically PPO) along with human feedback to align language models with human preferences.
SFT teaches “this is the correct answer.” RLHF teaches “this answer is better than that one” — which matters because in the real world there can be many correct answers. For the prompt “Explain AI,” a simple, clear answer and a dense, jargon-heavy answer can both be technically correct, but humans prefer the first.
The cast of characters
| Character | Role | Analogy from class |
|---|---|---|
| Human annotators | Provide preference judgments | The teachers/judges |
| Reward model | Scores responses | The trained judge |
| Value model | Predicts the expected reward | The coach who predicts how well the student should do |
| Policy model | The LLM being trained | The student |
| PPO | The update rule | The training rule that updates the student safely |
Architecturally: the policy model is the full LLM; the reward model is a smaller LLM with a reward head (outputs a single score); the value model is the policy model with a value head. A fourth model — the reference model — is a frozen copy of the original policy, used to keep the policy from drifting too far (more below).
Step 1: Collect human preferences
Humans see a prompt (“Explain Machine Learning”) and two model responses. The evaluator says “A is better than B.” We store:
Prompt, Chosen = A, Rejected = B
This is the human feedback, and it becomes a preference dataset — the same prompt/chosen/rejected format from Post 5. (A well-known public example is Anthropic’s hh-rlhf dataset on the Hugging Face Hub.)
Step 2: Train a reward model
We don’t start PPO immediately. First we train a reward model: input = prompt + response, output = a single score. The objective is Reward(chosen) > Reward(rejected) — instead of learning “correct vs. incorrect,” it learns “better vs. worse.” In RLHF, the reward model is a proxy for human judgment.
Step 3: Run PPO
For each prompt, the loop is:
- Policy model generates a response (this is the “action”; store its probability as old_prob = 0.60)
- Reward model scores it — say +8
- Value model had predicted an expected reward of 6
- Advantage = 8 − 6 = +2 → better than expected
- PPO updates the policy: probability 0.60 → 0.72, ratio 1.2 — fine
- If it had gone to 0.95 (ratio ≈ 1.58), clip it back to the allowed 1.2
- Repeat across thousands of diverse prompts (“What is Machine Learning?”, “Explain Python”, “Write a polite email”…)
Over time the model becomes more helpful, harmless, and honest.
The interview question
Does PPO directly use human feedback? No. Human feedback is first used to train the reward model. PPO never interacts with humans directly; it only receives reward scores from a reward model that has learned human preferences.
The reference model and KL penalty
Your main trainable model is the policy model. Before training, you keep a frozen copy — the reference model. During PPO, a KL-divergence penalty measures how far the policy has drifted from the reference and penalizes it if it goes too far. Without this, the policy could learn to exploit quirks of the reward model rather than genuinely improve.
DBA analogy: the reference model is a baseline snapshot, and the KL penalty is a drift-detection check against it — you’re allowed to evolve, but not to wander arbitrarily far from a known-good state.
The full pipeline
Humans → Preference Data → Reward Model Training → Reward Model
→ PPO Training → Policy Update → Better LLM
Part 5: Hands-On — A Miniature RLHF Run
The class notebook runs this whole pipeline on a tiny model (HuggingFaceTB/SmolLM2-135M-Instruct) using Hugging Face TRL, so it can run on a free GPU. This is a learning demo — a handful of preference examples and a few PPO episodes won’t produce a meaningfully better model, but it lets you see every moving part.
Version note: the notebook installs
trl>=1.0.0and importsPPOTrainer/PPOConfigfromtrl.experimental.ppo. Experimental APIs can change between releases, so check the TRL docs if an import fails.
1. The preference dataset
The class dataset is JSONL — one prompt/chosen/rejected record per line:
{"prompt": "How should I prepare for exams?",
"chosen": "Make a schedule, revise daily, practice previous questions, and sleep well.",
"rejected": "Do nothing and panic at the end."}
(The sample file has 20 such rows; the notebook’s inline version uses 4.) Each prompt is wrapped with the model’s chat template, and an end-of-sequence token is appended to the responses:
def make_chat_prompt(user_prompt):
messages = [{"role": "user", "content": user_prompt}]
return tok_rm.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
def make_reward_example(example):
return {
"prompt": make_chat_prompt(example["prompt"]),
"chosen": example["chosen"] + tok_rm.eos_token,
"rejected": example["rejected"] + tok_rm.eos_token,
}
2. Train the reward model
The reward model is a sequence-classification model with a single output (num_labels=1) — the scalar score:
from transformers import AutoModelForSequenceClassification
from trl import RewardTrainer, RewardConfig
rm = AutoModelForSequenceClassification.from_pretrained(model_name, num_labels=1)
rm_args = RewardConfig(
output_dir="./rm_out",
per_device_train_batch_size=1,
num_train_epochs=1,
learning_rate=1e-5,
max_length=256,
)
rm_trainer = RewardTrainer(model=rm, args=rm_args,
processing_class=tok_rm, train_dataset=reward_ds)
rm_trainer.train()
rm_trainer.save_model("./rm_out")
3. Set up the four models for PPO
from transformers import AutoModelForCausalLM
# Policy: the model PPO will update
policy = AutoModelForCausalLM.from_pretrained(model_name)
# Reference: frozen copy for the KL penalty
ref_model = AutoModelForCausalLM.from_pretrained(model_name)
for p in ref_model.parameters():
p.requires_grad_(False)
# Reward model: trained above, frozen during PPO
reward_model = AutoModelForSequenceClassification.from_pretrained("./rm_out", num_labels=1)
for p in reward_model.parameters():
p.requires_grad_(False)
# Value model: trainable critic (initialized from the reward model)
value_model = AutoModelForSequenceClassification.from_pretrained("./rm_out", num_labels=1)
Notice the pattern: two trainable models (policy and value) and two frozen ones (reference and reward). This is why RLHF is expensive — you’re holding several models in memory at once.
4. Run PPO and use the result
from trl.experimental.ppo import PPOTrainer, PPOConfig
ppo_args = PPOConfig(
output_dir="./ppo_out",
learning_rate=1e-6,
num_ppo_epochs=1,
total_episodes=6, # tiny, for demonstration only
response_length=16,
)
ppo_trainer = PPOTrainer(
args=ppo_args,
processing_class=tok,
model=policy,
ref_model=ref_model,
reward_model=reward_model,
value_model=value_model,
train_dataset=ppo_train_dataset, # tokenized prompts only
)
ppo_trainer.train()
ppo_trainer.save_model("./ppo_out")
Notice that the PPO dataset contains only prompts — the responses are generated by the policy during training, and scored by the reward model. Finally, load ./ppo_out like any causal LM and call generate() to see the aligned model’s output.
Part 6: The Simpler Alternatives — DPO, GRPO, ORPO
RLHF with PPO works, but it’s complex and expensive: multiple models plus an RL training loop. That’s why simpler methods emerged. The comparison from class:
| Method | Uses RL? | Separate reward model? | Complexity | Notes |
|---|---|---|---|---|
| PPO (RLHF) | Yes | Yes | Very high | The classical method, associated with OpenAI |
| GRPO | Yes | Yes | High | Associated with DeepSeek; drops the value model |
| DPO | No | No (implicit preference objective) | Medium | Directly optimizes chosen > rejected |
| ORPO | No | No | Low | Combines SFT and preference learning in one loss |
DPO (Direct Preference Optimization)
DPO makes the model prefer the chosen answer over the rejected answer — directly, with no separate reward model and no PPO loop.
How it works:
- Data: the same prompt / chosen / rejected format.
- Two models: the policy model (trainable) and a reference model (a frozen version of the starting model).
- For each example, compute how likely each model finds the chosen and rejected responses, and compare.
- The DPO loss pushes the policy to increase the probability of the chosen response and decrease the probability of the rejected one, relative to the reference.
- A hyperparameter (beta) controls how far the policy may move away from the reference.
The class summary line says it best: SFT teaches how to answer. DPO teaches which answer is better.
DPO is attractive precisely because it removes the reward model, the value model, and the RL loop — simpler and more stable training. In Post 5 we saw how Unsloth patches TRL’s DPOTrainer to run it efficiently.
DBA analogy: PPO-based RLHF is a multi-stage ETL with a scoring service, a baseline estimator and a throttled update loop. DPO is a single direct transformation from labeled comparisons to an updated model.
Key Takeaways
✅ SFT shows only correct answers; it never explicitly teaches which answers are worse — preference alignment fills that gap ✅ RL = an agent learns by trial and error from rewards; “learning” means changing action probabilities (the policy) ✅ Q-learning uses a table and DQN uses a neural network, but both struggle with huge action spaces — which is why they aren’t used for LLM token generation ✅ PPO learns the policy directly, uses an actor-critic design, and keeps updates small via clipping ✅ Advantage = actual reward − expected reward (from the value model) ✅ RLHF has five roles: human annotators, reward model, value model, policy model, and PPO — plus a frozen reference model with a KL penalty ✅ PPO never uses human feedback directly — humans train the reward model, and the reward model feeds PPO ✅ DPO and ORPO skip RL entirely; GRPO keeps RL but drops the value model
What’s Next
This post is part of the GenAI Foundations series:
| # | Post | Status |
|---|---|---|
| 1 | How Transformers and Embeddings Actually Work | ✅ Published |
| 2 | Positional Encoding and Self-Attention (Q, K, V) Explained | ✅ Published |
| 3 | Encoder-Only vs. Encoder-Decoder vs. Decoder-Only + LLM Training Pipeline | ✅ Published |
| 4 | How to Access LLMs and Multimodal Models | ✅ Published |
| 5 | Fine-Tuning LLMs: Full vs. PEFT, LoRA/QLoRA, Data Types, HF + Unsloth | ✅ Published |
| 6 | RLHF Explained: PPO, Reward Models, and DPO | 📍 You are here |
| 7 | Fine-Tuning GPT Models with the OpenAI API (chat-format JSONL) | ⬜ Coming next |
👉 Next up: fine-tuning a hosted GPT model through the OpenAI API with a chat-format dataset — no GPU required.
References
- Proximal Policy Optimization Algorithms (PPO) — https://arxiv.org/abs/1707.06347
- Direct Preference Optimization (DPO) — https://arxiv.org/abs/2305.18290
- Hugging Face TRL documentation — https://huggingface.co/docs/trl
- Anthropic hh-rlhf dataset — https://huggingface.co/datasets/Anthropic/hh-rlhf
- SmolLM2 (model used in the demo) — https://huggingface.co/HuggingFaceTB/SmolLM2-135M-Instruct
- Course repo — https://github.com/sunnysavita10/Full-Stack-GenAI-Bootcamp-1.0
Found this helpful? Share it with your team! Questions? Drop them in the comments below.