Fine-tuning LLMs — Complete Guide with OpenAI and Open-Source Models 2025

Introduction

Why This Matters

Fine-tuning is the process of adapting a pre-trained LLM to a specific task or domain by training it further on a curated dataset. Done well, it produces models that follow your exact instructions, use your domain terminology, and maintain a consistent style — without the per-request cost of long system prompts or few-shot examples.

In 2025, fine-tuning is more accessible than ever. OpenAI's fine-tuning API works with GPT-4o mini and GPT-3.5 Turbo — you can upload a JSONL dataset, start a training job, and have a custom model deployed in under an hour. For open-source, libraries like Hugging Face TRL and the Unsloth project make fine-tuning LLaMA 3 and Mistral models practical on consumer GPUs.

The critical question is: when should you fine-tune instead of improving your prompts or using RAG? This guide answers that and walks through both the OpenAI and open-source fine-tuning workflows.

Fine-tuning vs Prompt Engineering vs RAG

Choose the right approach before investing time in fine-tuning:

Need	Best Approach
Model needs updated/recent knowledge	RAG
Model should cite sources	RAG
Model should follow a specific response format	Prompt engineering or fine-tuning
Model needs domain-specific vocabulary or style	Fine-tuning
Reduce prompt token cost at scale	Fine-tuning
Improve consistency across thousands of calls	Fine-tuning
Model does a specific task repeatedly	Fine-tuning

Fine-tune when: you have 50+ high-quality examples of the task, your use case is well-defined, and you want consistency without long prompts.

Do NOT fine-tune when: you need current knowledge, you have fewer than 50 examples, or a good system prompt already achieves your goal.

Data Preparation

Fine-tuning data must be in chat format (messages array):

import json
 
# Each training example is a conversation
training_examples = [
    {
        "messages": [
            {"role": "system", "content": "You are a customer support agent for AcmeCorp. Be concise and professional."},
            {"role": "user", "content": "My order hasn't arrived after 2 weeks."},
            {"role": "assistant", "content": "I apologize for the delay. Could you please provide your order number? I'll immediately check its status and arrange a solution for you."}
        ]
    },
    {
        "messages": [
            {"role": "system", "content": "You are a customer support agent for AcmeCorp. Be concise and professional."},
            {"role": "user", "content": "I want a refund."},
            {"role": "assistant", "content": "I'd be happy to process your refund. Our policy covers returns within 30 days. Please share your order number and the reason for the return so I can get this started immediately."}
        ]
    },
]
 
# Write to JSONL format (one JSON object per line)
with open("training_data.jsonl", "w") as f:
    for example in training_examples:
        f.write(json.dumps(example) + "\n")
 
# Validate your data
def validate_training_data(filepath: str) -> dict:
    examples = []
    errors = []
    with open(filepath) as f:
        for i, line in enumerate(f):
            try:
                example = json.loads(line)
                assert "messages" in example, "Missing 'messages' key"
                for msg in example["messages"]:
                    assert "role" in msg and "content" in msg, f"Invalid message at example {i}"
                examples.append(example)
            except Exception as e:
                errors.append(f"Line {i+1}: {e}")
 
    return {"count": len(examples), "errors": errors}
 
print(validate_training_data("training_data.jsonl"))

Data quality guidelines:

Minimum 50 examples; 500–1000 is recommended for meaningful improvement
Consistent system prompt across all examples
Diverse coverage of the task — include edge cases
High-quality responses — fine-tuning amplifies patterns including errors

OpenAI Fine-tuning API

from openai import OpenAI
import time
 
client = OpenAI()
 
# Step 1: Upload training file
with open("training_data.jsonl", "rb") as f:
    upload_response = client.files.create(file=f, purpose="fine-tune")
 
training_file_id = upload_response.id
print(f"Uploaded file: {training_file_id}")
 
# Step 2: Create fine-tuning job
job = client.fine_tuning.jobs.create(
    training_file=training_file_id,
    model="gpt-4o-mini-2024-07-18",  # or "gpt-3.5-turbo"
    hyperparameters={
        "n_epochs": 3,                # 3-5 epochs typical
        "learning_rate_multiplier": 1.0,
        "batch_size": "auto"
    }
)
 
job_id = job.id
print(f"Fine-tuning job: {job_id}")
 
# Step 3: Monitor job progress
while True:
    job_status = client.fine_tuning.jobs.retrieve(job_id)
    print(f"Status: {job_status.status}")
    if job_status.status in ["succeeded", "failed", "cancelled"]:
        break
    time.sleep(30)
 
# Step 4: Get fine-tuned model name
if job_status.status == "succeeded":
    fine_tuned_model = job_status.fine_tuned_model
    print(f"Fine-tuned model: {fine_tuned_model}")
    # e.g., ft:gpt-4o-mini-2024-07-18:your-org::abc123

Using the Fine-tuned Model

from openai import OpenAI
 
client = OpenAI()
 
response = client.chat.completions.create(
    model="ft:gpt-4o-mini-2024-07-18:your-org::abc123",
    messages=[
        {"role": "system", "content": "You are a customer support agent for AcmeCorp."},
        {"role": "user", "content": "My payment was declined but I was still charged."}
    ],
    temperature=0.3
)
 
print(response.choices[0].message.content)

Open-Source Fine-tuning with Hugging Face TRL

For LLaMA 3, Mistral, or Phi-3, use the TRL library:

pip install trl transformers datasets accelerate bitsandbytes peft

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from trl import SFTTrainer, SFTConfig
from peft import LoraConfig
from datasets import Dataset
 
# Load model in 4-bit quantization (fits on 16GB GPU)
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype="bfloat16",
)
 
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3-8B-Instruct",
    quantization_config=bnb_config,
    device_map="auto"
)
 
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")
tokenizer.pad_token = tokenizer.eos_token
 
# LoRA configuration for parameter-efficient fine-tuning
lora_config = LoraConfig(
    r=16,                    # LoRA rank
    lora_alpha=32,           # scaling factor
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    lora_dropout=0.1,
    bias="none",
    task_type="CAUSAL_LM"
)
 
# Training arguments
training_args = SFTConfig(
    output_dir="./fine-tuned-model",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    fp16=True,
    logging_steps=10,
    max_seq_length=1024,
)
 
# Dataset
dataset = Dataset.from_dict({"text": formatted_training_texts})
 
trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    peft_config=lora_config,
    args=training_args,
)
 
trainer.train()
trainer.save_model("./fine-tuned-model")

Evaluating Fine-tuned Models

from openai import OpenAI
 
client = OpenAI()
 
def evaluate_model(model_name: str, test_cases: list[dict]) -> dict:
    correct = 0
    results = []
 
    for case in test_cases:
        response = client.chat.completions.create(
            model=model_name,
            messages=case["messages"],
            temperature=0
        )
        output = response.choices[0].message.content.strip()
        expected = case["expected"]
 
        # Simple exact match; use LLM-as-judge for open-ended tasks
        is_correct = expected.lower() in output.lower()
        if is_correct:
            correct += 1
        results.append({"input": case["messages"][-1]["content"], "output": output, "correct": is_correct})
 
    return {
        "accuracy": correct / len(test_cases),
        "results": results
    }
 
# Compare base vs fine-tuned
base_results = evaluate_model("gpt-4o-mini", test_cases)
ft_results = evaluate_model("ft:gpt-4o-mini-2024-07-18:org::abc123", test_cases)
 
print(f"Base model accuracy: {base_results['accuracy']:.2%}")
print(f"Fine-tuned accuracy: {ft_results['accuracy']:.2%}")

Common Mistakes / Pitfalls

Fine-tuning with fewer than 50 examples — you will overfit, not improve
Inconsistent system prompts across training examples — the model learns conflicting behaviors
Including factual knowledge in fine-tuning data instead of using RAG — the model hallucinates more, not less
Not validating JSONL format before uploading — the job will fail at the validation step
Using temperature 0 during training evaluation and comparing to temperature 0.7 at inference — always test at your production temperature

Best Practices

Start with prompt engineering; only fine-tune once you have validated the task and collected quality data
Use your fine-tuned system prompt in every training example — the model learns what behavior follows it
Include negative examples and edge cases in training data — diversity beats volume
Evaluate using LLM-as-judge (GPT-4o evaluating outputs) for open-ended tasks where exact match is insufficient
Version your fine-tuned models and keep training data — you will need to retrain when the base model is updated

Key Takeaways

Fine-tuning adapts a pre-trained LLM to a specific task, style, or domain using supervised training on curated examples
Minimum 50 training examples for meaningful improvement; 500-1000 examples for production-quality results
OpenAI's fine-tuning API supports GPT-4o mini and GPT-3.5 Turbo — upload JSONL, start a job, use the model via API
Open-source fine-tuning with TRL + LoRA enables training LLaMA 3 or Mistral on a single 16GB GPU
Fine-tuning is NOT a replacement for RAG — fine-tune for style/format/behavior, use RAG for current knowledge
Data quality matters more than quantity — a few hundred perfect examples outperform thousands of noisy ones
Always evaluate against a held-out test set before deploying a fine-tuned model to production
Fine-tuning reduces inference cost by enabling shorter prompts — a 500-token system prompt can be baked into the model weights

Introduction#

Why This Matters#

Fine-tuning vs Prompt Engineering vs RAG#

Data Preparation#

OpenAI Fine-tuning API#

Using the Fine-tuned Model#

Open-Source Fine-tuning with Hugging Face TRL#

Evaluating Fine-tuned Models#

Common Mistakes / Pitfalls#

Best Practices#

Key Takeaways#

Related reading