LoRA Fine-Tuning Explained: Customize Any LLM on Consumer Hardware

LoRA fine-tuning lets you adapt any open-source LLM to your specific domain in hours using a single consumer GPU — no A100 cluster required. This guide explains how it works and how to do it.

What Is LoRA Fine-Tuning?

Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning technique that injects small trainable weight matrices into an existing frozen model. Instead of updating all billions of parameters, LoRA trains only ~0.1–1% of them, dramatically cutting VRAM requirements.

A 7B model normally needs 112 GB of VRAM to fine-tune in fp16. With LoRA + 4-bit quantization, the same model fits in 8 GB. That's the difference between needing an A100 and using your gaming RTX 4080.

LoRA vs Full Fine-Tuning vs QLoRA

Method	VRAM (7B)	Quality	Speed
Full Fine-Tune	112 GB	Best	Slowest
LoRA (fp16)	18 GB	Excellent	Fast
QLoRA (4-bit)	8 GB	Very Good	Medium

Hardware Requirements

For QLoRA fine-tuning on a 7B model, you need at minimum an RTX 3060 (12 GB). For 13B models, an RTX 4070 Ti (16 GB) or better. For 70B models, you'll want dual RTX 4090s or a Mac Studio M4 Ultra with 192 GB unified memory.

Before you start, check your GPU's memory bandwidth — training throughput correlates closely with bandwidth. The benchmarks at LLMHardware.io's GPU training guide show per-GPU throughput for common fine-tuning tasks.

Step-by-Step: Fine-Tune Llama 3 on Custom Data

pip install transformers peft trl bitsandbytes datasets accelerate

# train.py
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import get_peft_model, LoraConfig, TaskType
from trl import SFTTrainer
import torch

model_id = "meta-llama/Meta-Llama-3-8B"
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto"
)

lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16,
    lora_alpha=32,
    lora_dropout=0.1,
    target_modules=["q_proj","v_proj"]
)
model = get_peft_model(model, lora_config)

Dataset Preparation

Your dataset should be in conversational format. A minimum of 500 high-quality examples works well for domain adaptation. For instruction following, 1,000–5,000 examples is the sweet spot.

[
  {"messages": [
    {"role": "user", "content": "Summarize this legal clause: ..."},
    {"role": "assistant", "content": "This clause states that..."}
  ]}
]

Merging and Exporting Your Model

After training, merge the LoRA weights back into the base model for deployment:

from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(model_id)
model = PeftModel.from_pretrained(base, "./lora-output")
merged = model.merge_and_unload()
merged.save_pretrained("./merged-model")

You can then convert to GGUF for use with llama.cpp or Ollama. See the hardware requirements for running the merged model at LLMHardware.io.

Hyperparameter Tips

• LoRA rank r=16: good default; use r=64 for complex tasks
• Learning rate: 2e-4 for QLoRA, 1e-4 for full LoRA
• Batch size: maximize to your VRAM limit, use gradient accumulation for effective larger batches
• Epochs: 1–3; more risks overfitting on small datasets