Fine-Tuning LLMs in Depth: LoRA, QLoRA, Training Data, Evaluation and Serving

Fine-tuning changes how a language model behaves: its format, tone, task skill and vocabulary. It does not reliably change what it knows; retrieval is the better tool for facts (we compare the two in LLM Fine-Tuning vs RAG). This post goes deep on how we fine-tune when it is the right tool: the maths of low-rank adaptation, the GPU memory budget, building the training set, a working supervised fine-tuning pipeline, hyperparameters, preference tuning, evaluation and serving.

Why not just fine-tune all the weights?

Full fine-tuning updates every parameter. With mixed-precision training and the Adam optimiser, a common rule of thumb is about 16 bytes of GPU memory per parameter before activations:

Component	Bytes / parameter
bf16 weights	2
bf16 gradients	2
fp32 master copy of weights	4
Adam first and second moments (fp32)	8
Total	≈ 16

For an 8-billion-parameter model that is about 128 GB, plus activations. That means several data-centre GPUs and sharded training (FSDP or DeepSpeed ZeRO). It also produces a full-size copy of the model for every fine-tune. Parameter-efficient fine-tuning (PEFT) removes most of that cost.

How does LoRA work?

Low-Rank Adaptation (Hu et al., 2021) freezes the pretrained weight matrix $W_0 \in \mathbb{R}^{d \times k}$ and learns a low-rank update:

$$W = W_0 + \Delta W = W_0 + \frac{\alpha}{r} B A, \qquad B \in \mathbb{R}^{d \times r},\; A \in \mathbb{R}^{r \times k},\; r \ll \min(d, k)$$

The forward pass becomes $h = W_0 x + \tfrac{\alpha}{r} B A x$. The details matter:

• Initialisation. $A$ is initialised randomly and $B$ with zeros, so $\Delta W = 0$ at the start. Training begins exactly at the pretrained model.
• Parameter count. A LoRA layer trains $r(d + k)$ parameters instead of $dk$. For a 4096 × 4096 projection with $r = 16$, that is 131,072 instead of about 16.8 million: under 1%.
• Scaling. $\alpha / r$ keeps the update's magnitude roughly stable as you change $r$. rsLoRA proposes scaling by $\alpha / \sqrt{r}$ instead, which behaves better at high ranks.
• Where to apply it. The original paper adapted the attention projections. In practice, adapting all linear layers (attention q/k/v/o and the MLP gate/up/down projections in Llama-style models) usually gives noticeably better results for a modest increase in trainable parameters.
• No inference penalty if merged. After training, $\tfrac{\alpha}{r}BA$ can be added into $W_0$, so the merged model is exactly as fast as the original.

Why does a low-rank update suffice? Empirically, the weight change needed to adapt a pretrained model to a narrow task has low intrinsic rank. The model already has the capabilities; fine-tuning mostly re-weights them.

Variants worth knowing: DoRA decomposes weights into magnitude and direction and applies LoRA to the direction, often closing part of the gap to full fine-tuning. LoRA+ uses a higher learning rate for $B$ than for $A$.

What does QLoRA add?

QLoRA (Dettmers et al., 2023) makes the frozen base model small enough to fine-tune large models on a single GPU:

1. 4-bit NormalFloat (NF4). Pretrained weights are roughly normally distributed. NF4 is a 4-bit data type whose 16 levels are quantiles of a normal distribution, so it spends its precision where the weights actually are. Weights are quantised in small blocks, each with its own scale.
2. Double quantisation. The per-block scales are themselves quantised, saving roughly another 0.4 bits per parameter.
3. Paged optimisers. Optimiser state can page between GPU and CPU memory to survive memory spikes from long sequences.

The base weights are stored in NF4 and de-quantised to bf16 on the fly for each matrix multiplication. Gradients flow only into the LoRA adapters, which are kept in higher precision (PEFT stores adapter weights in fp32 by default). An 8B model's frozen weights take roughly 5 GB in 4-bit instead of 16 GB in bf16. That puts fine-tuning of 7–8B models within reach of a single 24 GB GPU (with modest batch sizes and gradient checkpointing), and much larger models on one 80 GB GPU.

The trade-off: de-quantisation makes each step slower than 16-bit LoRA. If you have the memory, plain LoRA on a bf16 base is faster.

How do we build the training data?

Data quality dominates every other choice. Our process:

1. Define the task and the acceptance metric first. For example: "Extract these 14 fields from a claim email with ≥ 97% field-level accuracy", or "Answer in the house format with the correct policy citation."
2. Collect real inputs. Use production logs, historical tickets and documents, not synthetic inputs alone. Synthetic data generated by a stronger model is useful for coverage, but it should be reviewed and mixed with real examples.
3. Write or correct ideal outputs with domain experts. A few hundred to a few thousand consistent examples usually beat tens of thousands of noisy ones. Inconsistent labels teach inconsistency.
4. Deduplicate and decontaminate. Remove near-duplicates (MinHash or embedding similarity), and make sure no evaluation example, or a paraphrase of one, appears in training.
5. Hold out an evaluation set stratified by input type and difficulty, including the hard cases and "should refuse" cases.
6. Format with the model's own chat template. Train and serve with exactly the same template and special tokens. Template mismatch is a silent, common bug.

We store examples in the conversational prompt–completion format, which lets the trainer compute the loss only on the completion tokens:

{"prompt": [{"role": "system", "content": "You extract claim fields as JSON."},
            {"role": "user", "content": "Email: Hi, my car was hit on 3 March near ..."}],
 "completion": [{"role": "assistant", "content": "{\"incident_date\": \"2026-03-03\", ...}"}]}

Masking the prompt from the loss matters. Without it, the model spends capacity learning to reproduce the inputs, which are often long, instead of learning the response.

A working QLoRA SFT pipeline

The following uses Hugging Face transformers, peft, trl and bitsandbytes. These libraries move quickly; argument names below follow recent TRL releases (for example max_length, which replaced max_seq_length), so check the release notes for the versions you pin.

import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from trl import SFTConfig, SFTTrainer

MODEL_ID = "meta-llama/Llama-3.1-8B-Instruct"   # any causal LM you are licensed to fine-tune

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",             # NormalFloat4
    bnb_4bit_use_double_quant=True,        # quantise the quantisation constants
    bnb_4bit_compute_dtype=torch.bfloat16, # de-quantise to bf16 for matmuls
)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID, quantization_config=bnb, dtype=torch.bfloat16, device_map="auto",
)

peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules="all-linear",           # attention and MLP projections
    task_type="CAUSAL_LM",
)

data = load_dataset("json", data_files={"train": "train.jsonl", "eval": "eval.jsonl"})

args = SFTConfig(
    output_dir="checkpoints/claims-extractor",
    num_train_epochs=2,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,         # effective batch 16 per device
    learning_rate=2e-4,                    # typical for LoRA; full fine-tuning uses ~10x lower
    lr_scheduler_type="cosine",
    warmup_steps=50,
    max_length=2048,
    bf16=True,
    gradient_checkpointing=True,           # trade compute for activation memory
    eval_strategy="steps",
    eval_steps=100,
    save_steps=100,
    logging_steps=10,
    report_to="mlflow",
)

trainer = SFTTrainer(
    model=model,
    args=args,
    train_dataset=data["train"],
    eval_dataset=data["eval"],
    processing_class=tokenizer,
    peft_config=peft_config,               # wraps the 4-bit model with LoRA adapters
)
trainer.train()
trainer.save_model("adapters/claims-extractor-v1")   # saves only the adapter weights (MBs)

Which hyperparameters matter?

• Learning rate. The most sensitive knob. For LoRA, values around 1e-4 to 2e-4 are typical starting points; full fine-tuning uses roughly an order of magnitude less. Use warmup and cosine or linear decay.
• Rank $r$ and $\alpha$. $r$ of 8–64 covers most tasks; set $\alpha$ to $r$ or $2r$ as a starting point. Higher rank helps when the task is far from the model's pretraining distribution. It rarely fixes bad data.
• Epochs. One to three epochs is usual. Watch the evaluation loss and task metrics. Training loss falling while task metrics stall or drop signals overfitting or memorisation.
• Effective batch size and sequence length. Use gradient accumulation to reach a stable effective batch. Packing several short examples into one sequence improves GPU utilisation; make sure your trainer masks attention across packed boundaries if examples must not attend to each other.
• Catastrophic forgetting. Narrow fine-tuning can degrade general abilities. Mitigate with lower learning rates, fewer epochs, LoRA's implicit regularisation, and by mixing in a small share of general instruction data.

When do we add preference tuning (DPO)?

Supervised fine-tuning teaches the model to imitate good answers. Some qualities are easier to express as comparisons: "this answer is more concise and better grounded than that one." Direct Preference Optimisation (Rafailov et al., 2023) trains directly on pairs of a preferred answer $y_w$ and a rejected answer $y_l$, without a separate reward model or reinforcement-learning loop:

$$\mathcal{L}_{\text{DPO}} = -\,\mathbb{E}\left[\log \sigma\!\left(\beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}\right)\right]$$

$\pi_{\text{ref}}$ is the frozen SFT model, and $\beta$ controls how far the policy may drift from it. Our sequence is SFT first, then DPO on pairs collected from expert review of the SFT model's own outputs. TRL's DPOTrainer supports LoRA adapters, so the same memory savings apply.

How do we evaluate a fine-tuned model?

We do not ship a fine-tune because its loss went down. Every candidate is compared with the base model plus the best prompt on the same held-out set:

1. Task metrics. Exact match or field-level F1 for extraction, accuracy and macro-F1 for classification, schema validity for JSON outputs.
2. Rubric-based grading for open-ended outputs: an LLM judge with a written rubric, calibrated against human ratings on a sample, with position bias controlled by randomising order.
3. Regression checks. A fixed suite of general instructions, safety prompts and refusal cases, to catch forgetting.
4. Operational metrics. Latency, tokens per request and cost at the expected load.

All runs, datasets, adapters and evaluation results are tracked in MLflow, so any production adapter can be traced to its exact training data and evaluation report.

How do we serve fine-tuned adapters?

There are two patterns:

• Merge and serve. model.merge_and_unload() folds the adapter into the base weights. Merge into a 16-bit copy of the base model, not the 4-bit training copy, to avoid compounding quantisation error. Then serve it like any model. This is simplest when one adapter is used.
• Multi-adapter serving. Keep one base model in GPU memory and load many small adapters, one per task, customer or language. Serving engines such as vLLM support this natively. You start the server with LoRA enabled and select the adapter per request, which makes dozens of specialised models cost little more than one.

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --enable-lora \
  --lora-modules claims-extractor=adapters/claims-extractor-v1 \
  --max-lora-rank 16
# Clients then request model="claims-extractor" through the OpenAI-compatible API.

Whichever pattern you use, version adapters like models: registry entries with aliases (champion, challenger), staged roll-out, monitoring of output quality and drift, and a one-step rollback.

A checklist before you fine-tune

• Prompting and retrieval were tried and measured, and a behavioural gap remains
• Task, acceptance metric and baseline score are written down
• Training data is real, expert-reviewed, deduplicated and decontaminated
• Evaluation set is held out and covers hard and refusal cases
• Chat template and special tokens are identical in training and serving
• Base model licence and data rights permit fine-tuning and your intended use
• Runs, data versions and evaluations are tracked; rollback is one step

References

• E. J. Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, 2021 (ICLR 2022).
• T. Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs, NeurIPS 2023.
• R. Rafailov et al., Direct Preference Optimization: Your Language Model is Secretly a Reward Model, NeurIPS 2023.
• S.-Y. Liu et al., DoRA: Weight-Decomposed Low-Rank Adaptation, ICML 2024.
• D. Kalajdzievski, A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA, 2023.
• Hugging Face documentation for PEFT and TRL; vLLM documentation on LoRA adapters.