Generative AI & LLM Engineering · Chapter 11 · ~25 minIntermediate → Advanced

Fine-Tuning & Alignment (LoRA, QLoRA, DPO & RLHF)

Learn when to prompt, when to retrieve with RAG, and when you must fine-tune. Master parameter-efficient adaptation (LoRA & QLoRA) to adapt 70B models on commodity hardware, and understand how modern Direct Preference Optimization (DPO) aligns models with human intent without fragile reinforcement learning loops.

Prompt vs. RAG vs. Fine-Tuning decision rules
Low-Rank Adaptation (LoRA) weight decomposition
4-Bit NormalFloat QLoRA on consumer GPUs
SFT, RLHF & Direct Preference Optimization (DPO)

1The Adaptation Spectrum: Prompting vs. RAG vs. Fine-Tuning

The single most common mistake in AI engineering is reaching for fine-tuning to solve a knowledge problem. LLMs acquire general reasoning during pre-training on trillions of tokens. Trying to teach an LLM recent facts via fine-tuning causes hallucinations and catastrophic forgetting.

Interactive Architectural Decision Matrix

Select your real-world system constraints to see the recommended adaptation pattern.

System & Business Constraints

Optimal ArchitectureKnowledge Retrieval

RAG (Retrieval-Augmented Generation)

Inject relevant chunks dynamically into the context window with exact citations and zero training cost.

Key Advantages:

No GPU training hours or dataset curation required.

Updates instantaneously whenever documents or knowledge change.

Provides rock-solid grounding and audit trails for compliance.

Engineering Tradeoff:

Consumes prompt tokens on every request, increasing TTFT (Time to First Token) and API token costs.

2The Computational Cost of Full Fine-Tuning

During full fine-tuning, every single parameter in the neural network is updated via backpropagation. For a modern 8-billion parameter model, storing the model weights in 16-bit precision takes only 16 GB of VRAM. Why can't we train it on a single 24 GB GPU?

1. Model Weights (FP16)

2 Bytes / param

For an 8B model: 16 GB of pure weights.

2. Gradients (FP16)

2 Bytes / param

First-order derivatives computed during backward pass: +16 GB.

3. AdamW Optimizer (FP32)

8–12 Bytes / param

Maintains 1st and 2nd momentum states in FP32: +64 to 96 GB!

Total VRAM for full fine-tuning an 8B model easily exceeds 96–120 GB, requiring an expensive distributed cluster of NVIDIA A100/H100 GPUs. Furthermore, training every weight risks catastrophic forgetting—the model loses its general knowledge and reasoning skills in exchange for the new niche data.

3Parameter-Efficient Fine-Tuning: LoRA & QLoRA

In 2021, Hu et al. from Microsoft introduced LoRA (Low-Rank Adaptation) based on a profound hypothesis: the weight updates ($Δ W$) during task adaptation have a very low intrinsic rank. Instead of updating the massive weight matrix $W$ directly, LoRA decomposes the update into two small low-rank matrices:

W_adapted = W_frozen + (B × A) × (α / r)

Base weights (W) are completely frozen: No optimizer states or gradients are stored for them.
Matrix A: Initialized with random Gaussian noise with dimensions $(r × d_in)$.
Matrix B: Initialized with zeros with dimensions $(d_out × r)$, ensuring $Δ W = 0$ at the start of training.
Rank (r): Typically chosen between 8 and 64. Reduces trainable parameters by over 99.8%!

LoRA & QLoRA Parameter & VRAM Calculator

See how low-rank decomposition shrinks billions of weights into lightweight megabyte adapter files.

Base Model Size8 Billion Params

LoRA Rank (r)r = 16

r=4 (Extreme low cost)r=16 (Balanced)r=64 (Expressive)

Weight Update Formula:

W_new = W_base + (B × A) × (α / r)

Base weight W_base is frozen. Adapter matrix B (d × r) and A (r × k) learn task delta.

Trainable Parameters

0.12%of total 8B base parameters

Adapter Checkpoint File Size

~19 MBvs 16 GB full base model checkpoint

✓ Swappable at runtime in milliseconds

VRAM Needed to Train (QLoRA)

~9.3 GBFits on 1x Consumer GPU (RTX 4090 / 3090)

Base: 6.0GB · Grads & Buffer: 3.3GB

Full Fine-Tuning (All Weights)

~144 GBRequires 4-8x 80GB A100s in parallel cluster

Risk: High catastrophic forgetting of general reasoning

QLoRA: Quantized LoRA (Dettmers et al., 2023)

QLoRA takes PEFT even further by quantizing the frozen base model weights to 4-bit NormalFloat (NF4)—an information-theoretically optimal quantile distribution for normally distributed neural network weights. With Double Quantization (quantizing the quantization constants) and Paged Optimizers (spilling memory spikes to CPU RAM), a full 70-billion parameter model can be fine-tuned on a single 48 GB GPU!

4Alignment: From Raw Model to Helpful Assistant (SFT, RLHF, DPO)

Pre-training teaches a model statistical correlation across the internet. Alignment teaches the model to be Helpful, Honest, and Harmless (the 3 Hs).

Alignment Lab: SFT vs. RLHF vs. Direct Preference Optimization (DPO)

Compare classic Reinforcement Learning from Human Feedback (PPO) with modern DPO loss.

Direct Preference Optimization (DPO): The Mathematical BreakthroughRafailov et al. (Stanford, 2023) proved that the optimal RLHF policy has an exact closed-form solution. DPO eliminates the Reward Model and PPO entirely. It optimizes the LLM directly on preference pairs using standard binary cross-entropy loss!

Interactive Preference Dataset InspectorPair 1 of 2

User Instruction

Explain how to safely reset a database password in production.

Chosen Response (y_w)

To safely reset a database password: 1. Generate a high-entropy secret in your KMS. 2. Update credentials using zero-downtime dual-password rotation. 3. Verify application health before revoking the legacy credential.

Rejected Response (y_l)

Just run UPDATE users SET password = 'admin123' in psql and restart the backend server during lunch break.

Optimization Target: +2.48 log-prob preference advantage

Why DPO Has Replaced PPO in Modern Open-Source AI

In traditional RLHF with PPO, training is unstable because the policy model and reward model can enter adversarial feedback loops. If the reward model rewards longer answers, the LLM starts babbling useless paragraphs ("verbosity bias").

Traditional RLHF (PPO):

Requires 3-4 models in GPU memory
Hyperparameter sensitive & unstable
Samples generated during training loops

Direct Preference Optimization (DPO):

Only requires base model & frozen reference
Simple binary cross-entropy loss formulation
Offline dataset of (prompt, chosen, rejected) pairs

5Dataset Engineering: Less Is More for Alignment (LIMA)

The famous Meta AI paper LIMA: Less Is More for Alignment (Zhou et al., 2023) demonstrated that fine-tuning on just 1,000 meticulously curated, high-quality instruction pairs outperformed models fine-tuned on 50,000 uncurated pairs.

Standard ChatML Training Template

<|im_start|>system
You are a senior PostgreSQL database engineer. Respond only with valid SQL queries.<|im_end|>
<|im_start|>user
Find all users who registered in the last 30 days and have spent over $500.<|im_end|>
<|im_start|>assistant
SELECT u.id, u.email, SUM(o.amount) AS total_spent
FROM users u
JOIN orders o ON u.id = o.user_id
WHERE u.created_at >= NOW() - INTERVAL '30 days'
GROUP BY u.id, u.email
HAVING SUM(o.amount) > 500;<|im_end|>

Special delimiter tokens (<|im_start|>, <|im_end|>) prevent prompt injection attacks from breaking out of user roles during model execution.

6Instant Challenge

Architect the Medical Summarizer Pipeline

You are the Lead AI Architect at a hospital network. You must build an AI system that ingests confidential, real-time patient lab reports and outputs summaries formatted to strict Epic EHR HL7/FHIR JSON specifications. The system must run on on-premise servers (air-gapped), cite exact lab values, and guarantee schema compliance with minimal latency. Which architecture should you select?

0 / 1 correct

1Choose the architecture that satisfies the hospital's requirements.

7Knowledge Checks

Concept Check 1 of 4

Why does LoRA freeze the base model weights and only train low-rank adapter matrices?

Concept Check 2 of 4

What is the primary technical breakthrough of Direct Preference Optimization (DPO) compared to PPO-based RLHF?

Concept Check 3 of 4

If your company requires an AI agent that knows about product stock levels that change every 5 minutes, what is the correct strategy?

Concept Check 4 of 4

What does the NF4 data type in QLoRA stand for, and why is it special?

8Recap

RAG for facts, Fine-Tuning for form: Never fine-tune to inject frequently changing knowledge. Fine-tune to bake in specialized formatting, tone, or domain syntax.
LoRA eliminates VRAM bottlenecks: Decomposing weight updates into rank matrices ($W + B × A$) cuts trainable parameters by over 99%.
QLoRA democratizes 70B training: 4-bit NormalFloat quantization fits large models on single commodity GPUs.
DPO simplifies alignment: Replaces fragile PPO reinforcement learning loops with direct binary cross-entropy optimization on preference pairs.
Quality over quantity: LIMA proves that 1,000 clean, diverse instruction-response pairs beat 50,000 noisy samples.

Solve the challenge above to complete the chapter, or mark it complete now: