Introduction
In the last post we learned how to call models. This one is about changing them. Suppose your company has thousands of pages of pharma research and you want a model that actually knows that domain. Training an LLM from scratch is out of reach for almost everyone, so the practical route is: download an open-source model, then fine-tune it on your own data.
Fine-tuning has a lot of moving parts — three training stages, three data formats, full vs. parameter-efficient methods, adapters, quantization, and two competing toolchains. This post lays out the map first and then fills in the details, using a running pharma example and DBA analogies.
What you’ll learn:
- What fine-tuning is, and where it fits in the 3-stage LLM pipeline
- Full fine-tuning vs. PEFT, and what LoRA, QLoRA and DoRA actually do
- The three kinds of training data: raw text, instruction pairs, and preference pairs
- How adapters are attached or merged, and how to chain training stages
- What the Hugging Face ecosystem provides, and where Unsloth fits
- Epochs, batches and steps — what “training” is really doing
Prerequisites
- [ ] Read Post 3 (the 3-stage LLM pipeline: pretraining → SFT → preference alignment)
- [ ] Basic Python and comfort with the idea of model “weights”
- [ ] A GPU environment (e.g., Google Colab) if you want to run training yourself
What Is Fine-Tuning?
Fine-tuning is taking an already-trained model and training it further on domain-specific or task-specific data, so it performs better for a particular use case.
Recall the pipeline from Post 3. Every major lab publishes models at different points along it. Using Meta’s Llama as the example from class:
| Stage | What it is | Example model name |
|---|---|---|
| 1. Pretraining | Trained on internet-scale raw data | meta-llama/Meta-Llama-3-8B (base model) |
| 2. Instruction tuning (SFT) | Taught to follow instructions | meta-llama/Meta-Llama-3-8B-Instruct |
| 3. Preference tuning | Aligned to human preferences | e.g., OpenRLHF/Llama-3-8b-rlhf-100k |
The key insight: you can plug into any of these stages with your own data. You can do your own pretraining-style training on raw documents, your own instruction tuning, or your own preference tuning.
Which starting model? The class use case: a pharma company has documents on molecule studies.
- If you only need the model to generate text in that domain (not chat), fine-tune the base model on your raw pharma text.
- If you want a chatbot / Q&A assistant, start from the Instruct model and fine-tune it on your own instruction data.
DBA analogy: the base model is a freshly provisioned database engine with generic system knowledge. Instruction tuning is the layer that makes it respond to queries in a usable way. Fine-tuning on your data is loading your schema and data into it so it answers questions about your business.
The Core Question: Which Parameters Do You Train?
A model’s parameters are its weights and biases. In a Transformer, these live in the self-attention layers (the Q, K and V weight matrices from Part 2) and the feed-forward layers. Training means adjusting these numbers.
Full fine-tuning
Retrain all parameters. Powerful, but it needs huge GPU power and infrastructure — impractical for most teams with modern, very large LLMs.
Partial fine-tuning
Train only a subset of parameters. There are two flavors:
1. The older approach (used with CNNs like VGG/ResNet and early models like BERT, BART, T5):
- Freeze all layers and retrain only the output layer, or
- Freeze some of the early layers and retrain the remaining later layers
2. The modern approach for LLMs — PEFT (Parameter-Efficient Fine-Tuning): designed for small infrastructure, such as a single GPU with limited VRAM. The main PEFT techniques:
| Technique | Idea |
|---|---|
| LoRA (Low-Rank Adaptation) | Keep the base weights frozen; train small “adapter” matrices instead |
| QLoRA | LoRA applied to a quantized base model, cutting memory further |
| DoRA | Weight-decomposition variant of LoRA |
| (BitFit, IA³) | Other PEFT methods mentioned, not covered in depth |
How LoRA works, in plain words
Instead of changing a huge weight matrix directly, LoRA freezes it and learns a small low-rank update — a pair of much smaller matrices whose product approximates the change. The class illustration: a model with 1 billion parameters might train only on the order of 1 million adapter parameters (illustrative numbers, but that scale of reduction is the point). The result is a compact LoRA adapter you can store, share, and attach to the base model.
What “quantization” means
Weights are normally stored as 32-bit floats. Quantization stores them with fewer bits (e.g., 16-, 8-, or 4-bit), so the model takes far less memory. QLoRA = quantized base model + LoRA adapters, which is why large models can be fine-tuned on a single modest GPU.
DBA analogies:
- Full fine-tuning = rewriting every row of a giant table. PEFT/LoRA = keeping the table read-only and storing a small delta/patch table with your changes.
- Freezing layers = marking part of the data as read-only, like a read-only tablespace.
- Quantization = shrinking column data types (e.g., storing numbers in a smaller type, or compressing) to save storage and memory at a small precision cost.
Three Kinds of Training Data
This is where many beginners get confused. Each stage of the pipeline needs a different shape of data.
1. Non-instruction data (raw text) → custom pretraining
Whenever you have raw documents — PDFs, Word docs, slides, plain text — you use non-instruction fine-tuning. The model learns domain language, terminology and patterns by predicting the next token, exactly like pretraining, just on your data.
Where do real teams get this raw data? SharePoint, Confluence, Google Drive, AWS S3, Azure Blob Storage, or an internal document portal — downloaded as PDF, DOCX, HTML, TXT, or Markdown.
Training examples are built by sliding along the text with a next-token target:
Input: "You need to" → Target: "change"
Input: "You need to change" → Target: "your"
Input: "You need to change your" → Target: "setting"
A typical non-instruction pipeline from class:
- Collect raw data (PDF, DOCX, etc.)
- Read/parse the data
- Clean the data
- Produce the final dataset
- Convert to a Hugging Face–compatible format
- Preprocess (tokenize)
- Load the model (quantized, with LoRA adapters)
- Fine-tune (PEFT)
- Save the adapter / fine-tuned model
- Run inference
Padding and truncation: sentences have different lengths, but the model processes fixed-size batches. So you pick a maximum length (e.g., 512 tokens), pad shorter sequences with special pad tokens, and truncate longer ones. Long documents are split into fixed-size blocks.
2. Instruction data → supervised fine-tuning (SFT)
Here each record is an input/output pair: a question or instruction and an ideal answer. This teaches the model how to respond, not just what the domain text looks like.
Common formats are Alpaca-style (instruction, input, output) and the OpenAI-style chat format (system / user / assistant messages). From the class pharma dataset:
{
"instruction": "Explain the primary mechanism of action of metformin.",
"input": "",
"output": "Metformin primarily acts by activating AMP-activated protein kinase (AMPK)..."
}
Before training, each record is formatted into a single text string (instruction, input, and output stitched together with a template) so the model sees one consistent sequence.
How do real teams get instruction data? Three methods from class:
- Human-created Q&A — domain experts write questions and answers by hand.
- Existing conversations — customer support tickets, chat transcripts, email threads, expert answers; cleaned and converted to instruction format.
- Synthetic generation — a strong LLM generates Q&A pairs from your raw documents, then a human expert reviews them.
Important: in regulated domains like pharma or healthcare, synthetic data must be reviewed by domain experts before training.
3. Preference data → preference tuning
Each record has a prompt, a chosen (better) response, and a rejected (worse) response. The model learns to prefer the chosen style. Example structure from the class dataset:
{
"prompt": "### Instruction:\nExplain the primary mechanism of action of metformin.\n\n### Response:\n",
"chosen": "Metformin primarily acts by activating AMP-activated protein kinase (AMPK)...",
"rejected": "Metformin mainly works by increasing insulin secretion from the pancreas..."
}
(The rejected answer here is deliberately incorrect — that’s what makes it a useful contrast.) Preference pairs come from subject-matter experts, or from AI assistance on top of raw data.
The pharma content used in class is educational material for practicing fine-tuning — not medical advice.
| Data type | Shape | Training stage | Teaches |
|---|---|---|---|
| Non-instruction | Raw text | Custom pretraining | Domain language and knowledge |
| Instruction | input ↔ output | SFT | How to respond to users |
| Preference | prompt, chosen, rejected | Preference alignment | Which responses humans prefer |
DBA analogy: raw text is a bulk load of unstructured documents; instruction data is a curated set of query/expected-result test cases; preference data is user feedback ranking which result set was better.
Preference Alignment Methods
Preference tuning is how models get aligned to human responses (think of the “which response do you prefer?” prompts in chat products). The methods named in class:
| Method | Notes |
|---|---|
| RLHF | The original framework, popularized by OpenAI; uses reinforcement learning, with PPO as the classic algorithm |
| GRPO | An RL-based method associated with DeepSeek |
| DPO | No reinforcement learning — optimizes directly on chosen/rejected pairs |
| ORPO | Another non-RL approach |
If RL feels heavy, DPO and ORPO are the lighter-weight options — a big reason they’re popular for custom tuning. (Class also flagged RLHF/DPO as topics for a later session, so treat this as the overview.)
Adapters: Attach, Merge, and Chain
After LoRA training you don’t have a new full model — you have a small adapter. Two ways to use it:
- Append (attach): keep the base model and the adapter separate and combine them at inference time. This uses runtime memory (VRAM) for both.
- Merge: fold the adapter weights permanently into the base model, producing a standalone model.
from peft import PeftModel
# Load the trained LoRA adapter on top of the base model
model_with_adapter = PeftModel.from_pretrained(base_model, adapter_dir)
# Merge adapter weights into the base model weights permanently
merged_model = model_with_adapter.merge_and_unload()
DBA analogy: attaching is like applying a patch at query time through a view; merging is applying the patch to the table itself and committing.
Chaining the three stages — two approaches
Approach 1 — one final model (sequential merge):
Stage 1: Base model + LoRA adapter 1 (non-instruction data) → merge → M1
Stage 2: M1 + LoRA adapter 2 (instruction data) → merge → M2
Stage 3: M2 + LoRA adapter 3 (preference data) → merge → M3 (final model)
Choose this if you want a single final model.
Approach 2 — separate adapters per task: keep one base model and train three independent adapters (L1, L2, L3) on it. At inference, load the base model plus whichever adapter matches the task. Flexible, but each adapter only reflects its own stage.
One caution from the class discussion: be deliberate about what each adapter was trained on top of. An adapter trained against the merged model from the previous stage expects that model underneath it — mixing adapters with the wrong base is a common source of confusion.
What Does “Training” Actually Do? Epochs, Batches, Steps
The training loop is simple:
- Forward pass (FP): the model makes a prediction.
- Compare to the actual value → compute the loss.
- Backpropagation (BP): adjust weights and biases to reduce the loss.
- An optimizer (AdamW is the example in class) applies the update.
Repeat — loss goes down, quality goes up.
Worked example from class: you have 1,000 training examples and a batch size of 10.
- One batch = 10 rows
- One step = one forward + backward pass on a batch
- One epoch = one pass through all data = 1000 / 10 = 100 steps
DBA analogy: batches and steps are like batched commits — you process the data in chunks, and one epoch is one full pass over the table.
The Hugging Face Ecosystem
Hugging Face is an AI company providing a complete ecosystem for AI development: training, fine-tuning, evaluation, and hosting. For modern LLM fine-tuning, these libraries are the ones to know:
| Library | Role |
|---|---|
| Transformers | Load and run models |
| Datasets | Load and process datasets (built-in and custom) |
| Tokenizers | Turn text into token IDs |
| PEFT | LoRA / QLoRA adapters |
| Accelerate | Distributed and multi-GPU training |
| TRL | Trainers for SFT, RLHF, DPO, PPO, GRPO |
| Bitsandbytes | 4-bit / 8-bit quantization |
| Safetensors | Safe checkpoint format |
| Evaluate / Lighteval | Evaluation |
An interesting point from class: not every tool in the ecosystem is built by Hugging Face. Libraries like Bitsandbytes, timm, Sentence Transformers, Argilla and Distilabel are “outsiders” that are integrated — the ecosystem is integrated even if ownership is different. Hugging Face also offers the Hub (host models and datasets, like GitHub for models) and Spaces (host your apps).
A typical Hugging Face fine-tuning workflow: load a dataset (built-in or custom) → preprocess/clean → tokenize → load the model → apply PEFT/LoRA with quantization → train with a Trainer/TRL → evaluate → save checkpoints → push to the Hub → deploy/inference → monitor (quality, latency, GPU usage, token cost) and iterate (more data, better adapters, hyperparameter tuning).
Evaluation metrics mentioned: accuracy, F1-score, BLEU, ROUGE, perplexity, and benchmark scores.
Datasets: built-in datasets are already on the Hub, uploaded by companies, universities, and the open-source community. A custom dataset is yours — personal or enterprise. You can convert your data (CSV, XLSX, TSV, JSON, or big-data formats like Parquet/ORC) into a Hugging Face–compatible format and even upload it to share.
Exploring the Hub from code
from huggingface_hub import HfApi, list_repo_files
api = HfApi()
# Inspect a model
info = api.model_info(repo_id="bert-base-uncased")
print("Tags:", info.tags)
print("Downloads:", info.downloads)
print("Likes:", info.likes)
# Search models
for m in api.list_models(search="llama", limit=50):
print(m.modelId, "-", m.downloads)
# See what files a model repo contains
print(list_repo_files("google/flan-t5-base"))
# Inspect a dataset
dataset = api.dataset_info("imdb")
print([f.rfilename for f in dataset.siblings])
Use separate read and write tokens (store them as secrets, never in code).
You can also call hosted models through InferenceClient, or through LangChain’s Hugging Face wrappers (as in the last post).
Tokenization in one paragraph
Models don’t read words — they read token IDs. "I love AI" becomes tokens like ["I", "love", "AI"], then IDs like [40, 1842, 9552] (illustrative numbers). Common tokenization algorithms are BPE (byte pair encoding) and WordPiece, and you can even create a custom method. The pipeline: text → tokens → token IDs → word embeddings → positional encoding → self-attention → feed-forward — exactly the flow from Parts 1 and 2.
Side note: model sizes and heads
BERT-Base has 12 encoder layers, 768 hidden units, 12 attention heads (~110M parameters); BERT-Large has 24 layers, 1024 hidden units, 16 heads (~340M). In the Hugging Face AutoModel class, the raw model outputs a vector, and a task-specific head on top turns it into predictions (classification, etc.).
Where Unsloth Fits
The standard Hugging Face workflow is powerful but can be slow, VRAM-hungry, and expensive. Unsloth is an optimized fine-tuning framework built on top of the Hugging Face stack (Transformers + PEFT + TRL), focused on LoRA/QLoRA/RLHF-style training with low VRAM. It doesn’t replace Hugging Face — it’s an optimized engine on top of it.
Unsloth’s stated claims (as covered in class — treat these as vendor claims and benchmark on your own setup):
- Roughly 2–3× faster training
- Roughly 50–80% less VRAM
- Ability to fine-tune large models even on free GPUs
- Much longer context lengths during training before running out of memory
How is it faster? Low-level GPU optimizations: custom CUDA/Triton kernels (CUDA is NVIDIA’s GPU programming layer; Triton is a Python-like language that generates optimized GPU kernels), fused attention, optimized forward/backward passes, smart checkpointing, Flash Attention, manual backpropagation, and automatic sequence packing. Crucially, the class notes stress that Unsloth keeps exact math — it speeds things up without approximation, so accuracy loss is negligible.
Unsloth
↓
Transformers + PEFT + TRL
↓
PyTorch
↓
CUDA / Triton
↓
GPU
For preference tuning, the class showed how Unsloth patches TRL’s DPO trainer:
try:
from unsloth import PatchDPOTrainer
PatchDPOTrainer()
print("DPO patch applied.")
except Exception as e:
print("DPO patch skipped:", repr(e))
from trl import DPOTrainer
# DPO training now runs with Unsloth optimization
DBA analogy: Unsloth is like swapping in a faster storage engine or optimizer under the same SQL layer — same queries, same results, less time and memory.
Hugging Face or Unsloth? Both.
| Stage | Data format | HF route | Unsloth route |
|---|---|---|---|
| Non-instruction (custom pretraining) | {"text": "..."} | Trainer | SFT trainer |
| Instruction tuning | Alpaca-style {instruction, input, output} → text | Trainer | SFT trainer |
| Preference tuning | {prompt, chosen, rejected} | DPOTrainer | Patched DPO trainer |
The data work is the same either way; the difference is the training engine underneath.
Key Takeaways
✅ Fine-tuning = further training a pretrained model on your own domain data; you can plug in at any of the three pipeline stages ✅ Base model for domain text generation; Instruct model for chatbot/Q&A use cases ✅ Full fine-tuning retrains everything and needs heavy GPUs; PEFT (LoRA, QLoRA, DoRA) trains a small set of adapter weights on small infrastructure ✅ QLoRA = LoRA on a quantized (lower-bit) base model, cutting memory further ✅ Three data shapes: raw text (non-instruction), input/output pairs (instruction), and prompt/chosen/rejected (preference) ✅ In regulated domains, have experts review any synthetic training data ✅ Adapters can be attached at runtime or merged permanently; you can chain stages or keep one adapter per task ✅ DPO and ORPO are non-RL preference methods; RLHF (PPO) and GRPO use reinforcement learning ✅ Hugging Face provides the foundation (Transformers, Datasets, PEFT, TRL…); Unsloth is an optimized layer on top for faster, lower-VRAM training
What’s Next
This post is part of the GenAI Foundations series:
| # | Post | Status |
|---|---|---|
| 1 | How Transformers and Embeddings Actually Work | ✅ Published |
| 2 | Positional Encoding and Self-Attention (Q, K, V) Explained | ✅ Published |
| 3 | Encoder-Only vs. Encoder-Decoder vs. Decoder-Only + LLM Training Pipeline | ✅ Published |
| 4 | How to Access LLMs and Multimodal Models | ✅ Published |
| 5 | Fine-Tuning LLMs: Full vs. PEFT, LoRA/QLoRA, Data Types, HF + Unsloth | 📍 You are here |
| 6 | Hands-On: Fine-Tuning on a Pharma Dataset (non-instruction → instruction → preference) | ⬜ Coming next |
👉 Next up: a hands-on walkthrough that runs the three fine-tuning stages end to end on the pharma dataset — and after fine-tuning, we move on to RAG.
References
- Hugging Face Hub — https://huggingface.co/
- Hugging Face tokens — https://huggingface.co/settings/tokens
- Course repo — https://github.com/sunnysavita10/Full-Stack-GenAI-Bootcamp-1.0
- Attention Is All You Need — https://arxiv.org/pdf/1706.03762
- BERT — https://arxiv.org/pdf/1810.04805
Found this helpful? Share it with your team! Questions? Drop them in the comments below.