Fine-tune LLMs on Custom Data 2026 — QLoRA Guide with HuggingFace

Introduction

Why This Matters

Prompt engineering has limits. When you need consistent domain-specific behavior, a proprietary writing style, or dramatically reduced hallucinations on specialized topics, fine-tuning is the answer. A fine-tuned 7B model often beats a prompted 70B model on domain-specific tasks — while running 10x faster and costing 10x less per inference.

QLoRA (Quantized Low-Rank Adaptation) made fine-tuning accessible in 2023, and in 2026 it is the standard approach for teams that cannot afford to train from scratch. You can fine-tune a 7B model on a single 16GB GPU for under $5 on cloud GPUs, producing a model that is quantizably better than any system prompt for your specific task.

Understanding when to fine-tune versus when to use RAG or better prompting prevents expensive experiments that do not move the needle.

When to Fine-tune vs Prompt Engineering

Situation	Best Solution
Need specific response format	Prompt engineering (cheaper, faster)
Specialized domain knowledge	Fine-tuning
Consistent tone or persona	Fine-tuning
Very long system prompts (high latency)	Fine-tune to "bake in" the behavior
Reduce hallucinations on your data	Fine-tuning + RAG
Need 10x faster inference	Fine-tune a smaller model
New capability not in training data	Fine-tuning or RAG

QLoRA: Fine-tune on Consumer Hardware

pip install transformers peft datasets trl bitsandbytes accelerate

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer, SFTConfig
from datasets import Dataset
 
# 1. Load base model in 4-bit quantization
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)
 
model_name = "meta-llama/Llama-3.2-8B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token
 
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config,
    device_map="auto",
)
model = prepare_model_for_kbit_training(model)
 
# 2. Configure LoRA adapters
lora_config = LoraConfig(
    r=16,                  # Rank: higher = more params, better quality
    lora_alpha=32,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 4,194,304 || all params: 8,035,162,112 || trainable%: 0.052%

Only 0.05% of the model's parameters are updated during training — this is why QLoRA fits on a 16GB GPU.

Prepare Your Dataset

training_data = [
    {
        "instruction": "What is a Python list comprehension?",
        "response": "A list comprehension creates a new list by applying an expression to each item in an iterable.\n\n```python\nsquares = [x**2 for x in range(10)]\n```\n\nThis is equivalent to a for loop but more concise and faster."
    },
    # ... hundreds more examples
]
 
def format_instruction(sample: dict) -> str:
    return f"""<|begin_of_text|><|start_header_id|>system<|end_header_id|>
You are an expert Python tutor.<|eot_id|><|start_header_id|>user<|end_header_id|>
{sample['instruction']}<|eot_id|><|start_header_id|>assistant<|end_header_id|>
{sample['response']}<|eot_id|>"""
 
dataset = Dataset.from_list(training_data)
dataset = dataset.map(lambda x: {"text": format_instruction(x)})

Training

training_args = SFTConfig(
    output_dir="./fine-tuned-model",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    fp16=True,
    logging_steps=10,
    save_strategy="epoch",
    warmup_ratio=0.03,
    lr_scheduler_type="cosine",
    max_seq_length=2048,
    report_to="none",
)
 
trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    args=training_args,
    dataset_text_field="text",
)
 
trainer.train()
trainer.save_model("./fine-tuned-model")

Inference with Fine-tuned Model

from peft import PeftModel
 
base_model = AutoModelForCausalLM.from_pretrained(
    model_name, quantization_config=bnb_config, device_map="auto"
)
model = PeftModel.from_pretrained(base_model, "./fine-tuned-model")
 
inputs = tokenizer(
    "<|begin_of_text|><|start_header_id|>user<|end_header_id|>\nWhat is a decorator?<|eot_id|><|start_header_id|>assistant<|end_header_id|>",
    return_tensors="pt"
).to("cuda")
 
with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=300,
        temperature=0.1,
        do_sample=True,
        pad_token_id=tokenizer.eos_token_id,
    )
 
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Deploy to HuggingFace Hub

# Merge LoRA weights into base model for deployment
merged_model = model.merge_and_unload()
merged_model.save_pretrained("./merged-model")
tokenizer.save_pretrained("./merged-model")
 
from huggingface_hub import HfApi
api = HfApi()
api.upload_folder(
    folder_path="./merged-model",
    repo_id="your-username/my-python-tutor",
    repo_type="model"
)

Cost Estimate

GPU	Time for 7B model	Cost
Google Colab T4 (free)	2-3 hours	Free
Google Colab A100	30 min	~$3
Runpod A100 80GB	20 min	~$2
Modal.com H100	15 min	~$5

Common Mistakes / Pitfalls

Using quantity over quality — 500 high-quality, diverse examples beat 5,000 repetitive ones
Mismatched chat template — always use the exact chat template of your base model
Training on the wrong format — instruction fine-tuning requires question/answer pairs, not raw text
No evaluation set — always hold out 10% of data to measure improvement, not just training loss
Fine-tuning when prompting would suffice — if a detailed system prompt achieves 90%, fine-tuning for 95% may not be worth the cost

Best Practices

Start with the model's official chat template format — deviating causes unpredictable behavior
Use a diverse dataset covering all your target use cases, including edge cases
Monitor both training loss and eval loss — training loss dropping while eval loss rises = overfitting
Run inference on your eval set manually after each epoch, not just automated metrics
Merge and upload LoRA weights to HuggingFace Hub immediately after training to prevent loss

Key Takeaways

QLoRA fine-tunes only 0.05% of model parameters, enabling 7B model training on a single 16GB GPU
A fine-tuned 7B model frequently outperforms a prompted 70B model on domain-specific tasks
Training 1,000 examples on an A100 GPU takes about 20 minutes and costs approximately $2-5
Chat template format must exactly match the base model — mismatches cause silently wrong outputs
LoRA adapters can be merged into the base model for deployment without adapter overhead at inference
Evaluation loss is a better training signal than training loss — monitor both during every run
HuggingFace Hub handles versioning, hosting, and inference endpoint deployment for fine-tuned models
Fine-tuning and RAG are complementary: RAG provides current facts, fine-tuning provides style and domain behavior

Introduction#

Why This Matters#

When to Fine-tune vs Prompt Engineering#

QLoRA: Fine-tune on Consumer Hardware#

Prepare Your Dataset#

Training#

Inference with Fine-tuned Model#

Deploy to HuggingFace Hub#

Cost Estimate#

Common Mistakes / Pitfalls#

Best Practices#

Key Takeaways#

Related reading