Fine-Tuning LLMs with LoRA Locally: From Zero to Custom Model

Low-Rank Adaptation (LoRA) lets you fine-tune a large language model using a fraction of the compute and memory of full fine-tuning. With an RTX 4090 (24GB), you can fine-tune a 7B model on your own data in under an hour. Here is the complete workflow.

LoRA vs Full Fine-Tuning

Method	VRAM (7B)	Trainable Params	Time (1K examples)
Full fine-tune	~56 GB	7B (all)	~4 hours (A100)
LoRA (r=16)	~14 GB	~3M (0.04%)	~40 min (RTX 4090)
QLoRA (4-bit)	~8 GB	~3M	~60 min (RTX 3080)

Setup with Axolotl (Recommended)

pip install axolotl[flash-attn] deepspeed

# Create config.yml
cat > config.yml << 'EOF'
base_model: Qwen/Qwen2.5-7B-Instruct
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer

load_in_4bit: true  # QLoRA
adapter: lora
lora_r: 16
lora_alpha: 32
lora_dropout: 0.05
lora_target_modules:
  - q_proj
  - v_proj

datasets:
  - path: ./my_training_data.jsonl
    type: alpaca

sequence_len: 2048
sample_packing: true
micro_batch_size: 2
gradient_accumulation_steps: 4
num_epochs: 3
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.0002

output_dir: ./output/my-model
EOF

# Start training
accelerate launch -m axolotl.cli.train config.yml

Data Format

{"instruction": "Explain RTX 5090 bandwidth", "input": "", "output": "The RTX 5090 features 1,792 GB/s memory bandwidth with 32GB GDDR7..."}
{"instruction": "What quantization should I use for 8GB VRAM?", "input": "", "output": "For 8GB VRAM with an 8B model, use Q5_K_M..."}

Merging LoRA and Exporting to GGUF

python -c "
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained('Qwen/Qwen2.5-7B-Instruct')
model = PeftModel.from_pretrained(base, './output/my-model')
merged = model.merge_and_unload()
merged.save_pretrained('./merged-model')
"

# Convert to GGUF for llama.cpp
python llama.cpp/convert_hf_to_gguf.py ./merged-model --outfile my_model_q5.gguf --outtype q5_k_m

For hardware recommendations for fine-tuning at different scales, see LLMHardware.io's fine-tuning hardware guide.