Fine-Tuning LLMs:
Full Fine-Tuning, LoRA & QLoRA

GPT-4 is extraordinarily general.
It knows history, science, poetry, and code.
But it doesn't know your product documentation.
It doesn't speak with your brand voice.
It doesn't know the specialist vocabulary of retinal imaging.

Pre-training gives a model the world.
Fine-tuning gives it a job.

01 — What Is Fine-Tuning and Why Do We Need It?

Pre-training builds a foundation. A model trained on hundreds of billions of tokens from the internet, books, and code develops a deep general understanding of language, facts, and reasoning. But general understanding is not the same as specific usefulness. A doctor who knows everything about medicine still needs orientation training before working at a specific hospital.

Fine-tuning works because of a key insight: the knowledge encoded in a pre-trained model's weights is largely general and reusable. The layers that detect grammar, understand references, reason through logic — all of that transfers. What changes is the model's behaviour: how it formats responses, what terminology it uses, what tasks it prioritises. These behavioural changes require adjusting far fewer parameters than starting from scratch.

When Do You Actually Need Fine-Tuning?

Not always. Many tasks can be solved with good prompting. Fine-tuning is worth the investment when:

02 — Full Fine-Tuning: Change Everything

Full fine-tuning is conceptually the simplest approach: take all the model's weights, run your training data through it, compute gradients with respect to every single parameter, and update everything. No frozen layers, no approximations — every weight is a trainable variable.

How It Works

The training procedure is identical to pre-training — the only difference is the data and the starting point:

The Memory Problem

Full fine-tuning of a 7B parameter model requires storing significantly more than just the model weights. During training you need:

A single A100 80GB GPU barely handles a 7B model. For 13B or 70B models, full fine-tuning requires multiple high-end GPUs with expensive interconnects. This is out of reach for most researchers and companies. It also risks catastrophic forgetting — when a model is trained aggressively on a narrow dataset, it can "forget" its general capabilities. A medical fine-tune trained only on radiology reports might degrade badly at writing normal English.

03 — LoRA: Train Almost Nothing, Get Almost Everything

LoRA (Low-Rank Adaptation — Hu et al., 2021) is the most important fine-tuning innovation in recent years. It starts from a deceptively simple observation about how neural networks change during fine-tuning, and builds an elegant solution that makes fine-tuning a 7B model possible on a single modest GPU.

The Core Insight: Weight Updates Are Low-Rank

The mathematical justification: research found that when you fine-tune a model, the weight update matrix ΔW has very low intrinsic rank. A full fine-tune changes a 4096×4096 weight matrix — that's 16.7 million numbers. But the actual information content of that change can be captured with far fewer numbers, because the update ΔW lies in a low-dimensional subspace.

The LoRA Math

For each weight matrix W in the attention layers (typically W_q, W_k, W_v, W_o), LoRA adds a parallel low-rank branch:

Why This Works: The Numbers

For a 7B model (LLaMA-2 7B), the attention weight matrices have shape 4096 × 4096. Let's compare the trainable parameters:

Training 0.57% of parameters means: gradient computation is 175× cheaper. Optimiser states (the biggest memory consumer in full fine-tuning) shrink by 175×. The frozen W weights don't need gradients — PyTorch skips backpropagation through them entirely. A fine-tune that needed 80GB VRAM now needs ~16GB.

Choosing Rank r

Rank r is the most important LoRA hyperparameter. It controls the expressiveness of the adaptation — higher rank can model more complex changes but uses more memory and risks overfitting on small datasets:

Which Layers to Apply LoRA To?

By default, LoRA is applied to the query (W_q) and value (W_v) attention matrices. Research shows adding key (W_k) and output (W_o) projections gives a small additional boost. The MLP (feed-forward) layers contain most of the model's factual knowledge — applying LoRA there helps with domain-specific facts but risks more forgetting of general knowledge. In practice, target_modules=["q_proj", "k_proj", "v_proj", "o_proj"] is a reliable starting point.

04 — QLoRA: Fine-Tune a 70B Model on One GPU

LoRA solved the gradient/optimizer memory problem. But a 7B model's frozen weights still consume 14GB in float16. For 13B models that's 26GB; for 70B it's 140GB. QLoRA (Dettmers et al., 2023) adds one more trick: quantise the frozen base model to 4 bits.

NF4: A Smarter Way to Use 4 Bits

Standard int4 quantisation (just rounding weights to the nearest 1 of 16 levels) introduces significant precision loss. QLoRA uses a custom NF4 (NormalFloat4) data type, designed specifically for neural network weights:

Double Quantisation

QLoRA adds a second trick called double quantisation: quantise the quantisation constants themselves. When you quantise a weight tensor, you need to store a scaling factor (one per block of ~64 weights) to recover the original scale. These scaling factors are stored in fp32 — and there are many of them. Double quantisation quantises these scaling factors too (from fp32 to fp8), saving an additional ~0.37 bits per parameter. Small, but it adds up across billions of parameters.

The QLoRA Pipeline

Memory Comparison: The Numbers That Matter

05 — Training Data: What You Feed the Model

The training algorithm matters less than the data you train on. Every fine-tuning run is ultimately a question of: what examples do I show the model, and in what format?

Instruction Fine-Tuning Format

The dominant format for chat and instruction models is instruction–response pairs. Every training example consists of an instruction (what to do) and a target response (what the ideal answer looks like). The model is trained to maximise the probability of the response given the instruction — exactly CLM, just on carefully curated pairs.

How Much Data Do You Need?

Quality beats quantity. 500 perfectly curated examples that consistently demonstrate the target behaviour often outperform 50,000 noisy examples. GPT-4-generated synthetic data has become a standard technique: use the large model to create training examples for the smaller one. This is how Alpaca, Vicuna, and most open instruction-following models were built.

06 — Code: LoRA and QLoRA with HuggingFace + PEFT

Complete working code — from loading a model to training a LoRA adapter to saving and merging. Uses HuggingFace transformers, peft, and trl (which provides the SFTTrainer that handles instruction masking automatically).

# ─────────────────────────────────────────────────────────────────────────
# Install: pip install transformers peft trl bitsandbytes accelerate datasets
# ─────────────────────────────────────────────────────────────────────────

import torch
from transformers import (
    AutoModelForCausalLM, AutoTokenizer,
    BitsAndBytesConfig, TrainingArguments
)
from peft import LoraConfig, get_peft_model, TaskType, PeftModel
from trl  import SFTTrainer, DataCollatorForCompletionOnlyLM
from datasets import Dataset

# ─────────────────────────────────────────────────────────────────────────
# Step 1: Prepare training data
# ─────────────────────────────────────────────────────────────────────────
# Example: medical Q&A dataset (replace with your own data)
training_data = [
    {"instruction": "What is diabetic retinopathy?",
     "output": "Diabetic retinopathy is a diabetes complication that damages blood vessels in the retina..."},
    {"instruction": "Describe the stages of diabetic retinopathy.",
     "output": "There are four main stages: mild NPDR, moderate NPDR, severe NPDR, and PDR..."},
    # ... hundreds more examples
]

def format_prompt(example):
    """Convert instruction/output dict to training string."""
    return {
        "text": f"### Instruction:\n{example['instruction']}\n\n### Response:\n{example['output']}"
    }

dataset = Dataset.from_list(training_data).map(format_prompt)


# ─────────────────────────────────────────────────────────────────────────
# Step 2a: LoRA config (fp16 base model, ~18 GB VRAM for 7B)
# ─────────────────────────────────────────────────────────────────────────
model_id = "meta-llama/Llama-2-7b-hf"

lora_config = LoraConfig(
    r=16,                                    # rank — higher = more expressive, more params
    lora_alpha=32,                             # scaling factor (α/r = 2.0 here)
    target_modules=["q_proj", "k_proj",
                    "v_proj", "o_proj"],     # which layers to adapt
    lora_dropout=0.05,                         # regularisation
    bias="none",                              # don't add bias to LoRA layers
    task_type=TaskType.CAUSAL_LM,
)

tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.pad_token = tokenizer.eos_token   # LLaMA has no pad token by default

# Load base model in fp16 (LoRA — no quantisation)
base_model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto"
)
lora_model = get_peft_model(base_model, lora_config)
lora_model.print_trainable_parameters()
# → trainable params: 39,976,960 || all params: 6,778,318,848 || trainable%: 0.5899


# ─────────────────────────────────────────────────────────────────────────
# Step 2b: QLoRA config (4-bit NF4 base, ~10 GB VRAM for 7B)
# ─────────────────────────────────────────────────────────────────────────
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",               # NormalFloat4 — optimal for weight distributions
    bnb_4bit_compute_dtype=torch.bfloat16,  # dequantise to bf16 during forward pass
    bnb_4bit_use_double_quant=True,          # quantise the quantisation constants too
)

qlora_base = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto"
)

# Prepare model for k-bit training (gradient checkpointing + frozen BN layers)
from peft import prepare_model_for_kbit_training
qlora_base = prepare_model_for_kbit_training(qlora_base)
qlora_model = get_peft_model(qlora_base, lora_config)


# ─────────────────────────────────────────────────────────────────────────
# Step 3: Training with SFTTrainer
# ─────────────────────────────────────────────────────────────────────────
training_args = TrainingArguments(
    output_dir="./qlora-medical-llama2",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,          # effective batch size = 4 × 4 = 16
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    fp16=True,                              # mixed precision training
    logging_steps=10,
    save_strategy="epoch",
    report_to="none",
)

# SFTTrainer handles: chat template formatting, loss masking on instructions,
# gradient accumulation, and packing short examples together efficiently
trainer = SFTTrainer(
    model=qlora_model,
    tokenizer=tokenizer,
    args=training_args,
    train_dataset=dataset,
    dataset_text_field="text",
    max_seq_length=512,
    packing=True,    # pack multiple short examples into one sequence for efficiency
)

trainer.train()
trainer.model.save_pretrained("./qlora-adapter")
tokenizer.save_pretrained("./qlora-adapter")


# ─────────────────────────────────────────────────────────────────────────
# Step 4a: Inference with adapter (keep base + adapter separate)
# ─────────────────────────────────────────────────────────────────────────
model_for_inference = PeftModel.from_pretrained(qlora_base, "./qlora-adapter")
model_for_inference.eval()

prompt = "### Instruction:\nExplain macular oedema in simple terms.\n\n### Response:\n"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

with torch.no_grad():
    output = model_for_inference.generate(
        **inputs, max_new_tokens=200, temperature=0.7, do_sample=True
    )
print(tokenizer.decode(output[0], skip_special_tokens=True))


# ─────────────────────────────────────────────────────────────────────────
# Step 4b: Merge LoRA weights into base model (for clean single-file deployment)
# ─────────────────────────────────────────────────────────────────────────
# Load base in fp16 for merging (can't merge into 4-bit)
base_fp16 = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16)
merged_model = PeftModel.from_pretrained(base_fp16, "./qlora-adapter")
merged_model = merged_model.merge_and_unload()   # W_new = W_frozen + B·A
merged_model.save_pretrained("./qlora-merged")
# Result: a single 7B model in fp16 with the fine-tuning baked in

07 — Head-to-Head: Full vs LoRA vs QLoRA

08 — Common Mistakes and How to Avoid Them

09 — Real-World Example: Fine-Tuning for Medical Q&A

This is close to what was done in the Medical VLM project described in this portfolio — fine-tuning Qwen 2.5 7B with LoRA rank-32 on 35,000 retinal imaging Q&A pairs. The approach achieved a 240% accuracy gain (20% → 68% on diabetic retinopathy classification) while reducing VRAM from 17GB to 7GB via 4-bit QLoRA.

10 — Other PEFT Methods Worth Knowing

LoRA and QLoRA are the dominant approaches, but several other Parameter-Efficient Fine-Tuning (PEFT) methods are worth understanding:

11 — A Thought to Sit With

12 — Coming Up Next

Tomorrow — Day 29: RAG — Retrieval-Augmented Generation Pipeline. Fine-tuning teaches a model new habits. RAG gives it long-term memory. Instead of baking knowledge into weights, RAG retrieves relevant documents at inference time and feeds them into the context window — so the model can answer questions about things that happened after its training cutoff, or about private documents it has never seen. We build the full pipeline: document ingestion, chunking strategy, embedding, vector database, retrieval, and grounded generation — all from scratch.

13 — Go Deeper

📄 LoRA: Low-Rank Adaptation of Large Language Models — Hu et al. (2021)
The LoRA paper. Section 4 (the low-rank update formulation) is the core contribution. Table 2 comparing full fine-tune vs LoRA quality on GPT-3 is the key empirical result. Very readable.

📄 QLoRA: Efficient Finetuning of Quantized LLMs — Dettmers et al. (2023)
The QLoRA paper. Sections 2–3 cover NF4, double quantisation, and paged optimisers. Figure 1 showing the memory comparison is the clearest single illustration of why QLoRA matters.

🔗 HuggingFace PEFT Documentation
The reference implementation of LoRA, QLoRA, and all other PEFT methods. The LoRA quickstart guide is the fastest path from zero to a working fine-tune.

📄 DoRA: Weight-Decomposed Low-Rank Adaptation
The DoRA paper — improves on LoRA by separately adapting weight magnitude and direction. Used in LLaMA-3 fine-tuning workflows. Worth reading after understanding standard LoRA.

▶ Fine-tuning LLMs with PEFT and LoRA — Weights & Biases
YouTube · 45 min · End-to-end walkthrough of a real LoRA fine-tune with experiment tracking. Shows hyperparameter sweeps, evaluation, and deployment — the closest thing to a production tutorial.