Microsoft Phi-3 — Small Language Models Complete Guide 2026

Introduction

Why This Matters

Microsoft's Phi-3 family challenges the assumption that bigger models are always better. The Phi-3-Mini (3.8B parameters) achieves performance comparable to Mistral 7B on many benchmarks by training on carefully curated, high-quality data — "textbooks are all you need" taken to its logical extreme. This makes Phi-3 models uniquely suited for resource-constrained environments: laptops, edge devices, mobile phones, and Raspberry Pis.

For developers building applications where latency, memory, or power consumption matters — on-device AI assistants, offline applications, IoT inference, or cost-sensitive APIs — Phi-3 is often the best model choice in 2026. Understanding the Phi-3 family, its chat format, and its deployment options is increasingly important as AI moves to the edge.

Phi-3 Model Family

Model	Parameters	Context	VRAM (fp16)	VRAM (4-bit)	Best for
Phi-3-Mini-4K-Instruct	3.8B	4K tokens	8 GB	2.5 GB	Edge, mobile, laptops
Phi-3-Mini-128K-Instruct	3.8B	128K tokens	8 GB	2.5 GB	Long documents on edge
Phi-3-Small-8K-Instruct	7B	8K tokens	14 GB	4.5 GB	Balanced quality/speed
Phi-3-Medium-4K-Instruct	14B	4K tokens	28 GB	9 GB	High quality, mid GPU
Phi-3.5-Mini-Instruct	3.8B	128K tokens	8 GB	2.5 GB	Improved Mini successor
Phi-3.5-MoE-Instruct	41.9B (6.6B active)	128K tokens	—	—	MoE efficiency at scale

Quick Start with Ollama

# Phi-3 Mini — fastest, smallest
ollama pull phi3
ollama run phi3 "Explain recursion with a Python example."
 
# Phi-3 Medium — better quality
ollama pull phi3:medium
ollama run phi3:medium "What are the tradeoffs between SQL and NoSQL databases?"
 
# Phi-3.5 Mini — improved successor
ollama pull phi3.5

Using Hugging Face Transformers

pip install transformers torch accelerate

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
 
# Phi-3 Mini — no GPU required, runs on CPU
model_id = "microsoft/Phi-3-mini-4k-instruct"
 
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float32,   # Works on CPU without CUDA
    device_map="auto",
    trust_remote_code=True,
)

# Phi-3 Medium — GPU recommended
model_id_med = "microsoft/Phi-3-medium-4k-instruct"
 
model_medium = AutoModelForCausalLM.from_pretrained(
    model_id_med,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

Phi-3 Chat Format

messages = [
    {"role": "system", "content": "You are a helpful Python tutor. Be concise."},
    {"role": "user", "content": "What is the difference between `is` and `==` in Python?"},
]
 
formatted = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
 
inputs = tokenizer(formatted, return_tensors="pt").to(model.device)
 
outputs = model.generate(
    **inputs,
    max_new_tokens=300,
    temperature=0.7,
    do_sample=True,
    pad_token_id=tokenizer.eos_token_id,
)
 
new_tokens = outputs[0][inputs["input_ids"].shape[1]:]
response = tokenizer.decode(new_tokens, skip_special_tokens=True)
print(response)

Running on CPU (No GPU Required)

Phi-3-Mini is unique among capable LLMs: it runs adequately on CPU alone, making it the only model in its quality class that works on standard development laptops without a GPU:

from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
import torch
 
# CPU-only inference
model_id = "microsoft/Phi-3-mini-4k-instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
 
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float32,
    trust_remote_code=True,
)
# No .to("cuda") — stays on CPU
 
pipe = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    max_new_tokens=200,
)
 
messages = [{"role": "user", "content": "What is a hash table?"}]
result = pipe(messages)
print(result[0]["generated_text"][-1]["content"])

Expect 5-15 tokens/sec on a modern CPU — adequate for low-throughput applications.

4-bit Quantization for Maximum Efficiency

from transformers import BitsAndBytesConfig
 
# 4-bit NF4 — Phi-3-Mini fits in ~2.5 GB VRAM
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)
 
model = AutoModelForCausalLM.from_pretrained(
    "microsoft/Phi-3-mini-4k-instruct",
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True,
)

Multi-turn Chat Application

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch
 
class Phi3Chat:
    def __init__(self, model_id: str = "microsoft/Phi-3-mini-4k-instruct"):
        bnb = BitsAndBytesConfig(
            load_in_4bit=True,
            bnb_4bit_quant_type="nf4",
            bnb_4bit_compute_dtype=torch.bfloat16,
        )
        self.tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
        self.model = AutoModelForCausalLM.from_pretrained(
            model_id,
            quantization_config=bnb,
            device_map="auto",
            trust_remote_code=True,
        )
        self.messages: list[dict] = []
 
    def chat(self, user_input: str, max_new_tokens: int = 400) -> str:
        self.messages.append({"role": "user", "content": user_input})
        formatted = self.tokenizer.apply_chat_template(
            self.messages,
            tokenize=False,
            add_generation_prompt=True,
        )
        inputs = self.tokenizer(formatted, return_tensors="pt").to(self.model.device)
        outputs = self.model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            temperature=0.7,
            do_sample=True,
            pad_token_id=self.tokenizer.eos_token_id,
        )
        new_tokens = outputs[0][inputs["input_ids"].shape[1]:]
        response = self.tokenizer.decode(new_tokens, skip_special_tokens=True)
        self.messages.append({"role": "assistant", "content": response})
        return response
 
bot = Phi3Chat()
print(bot.chat("What is a Python context manager?"))
print(bot.chat("Can you show me a custom implementation?"))

Edge Deployment with ONNX Runtime

Phi-3-Mini is officially supported by the ONNX Runtime for deployment on devices without Python or PyTorch:

pip install onnxruntime-genai

import onnxruntime_genai as og
 
# Download ONNX model from: huggingface.co/microsoft/Phi-3-mini-4k-instruct-onnx
model = og.Model("phi-3-mini-4k-instruct-onnx")
tokenizer = og.Tokenizer(model)
 
# Configure generation
params = og.GeneratorParams(model)
params.set_search_options(max_length=300, temperature=0.7)
 
# Tokenize and generate
input_tokens = tokenizer.encode("<|user|>\nWhat is machine learning?<|end|>\n<|assistant|>\n")
params.input_ids = input_tokens
 
generator = og.Generator(model, params)
tokenizer_stream = tokenizer.create_stream()
 
while not generator.is_done():
    generator.compute_logits()
    generator.generate_next_token()
    token = tokenizer_stream.decode(generator.get_next_tokens()[0])
    print(token, end="", flush=True)
print()

ONNX Runtime enables Phi-3 inference on iOS, Android, Raspberry Pi, and Windows ARM with hardware-accelerated NPUs.

Fine-tuning with QLoRA

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, TrainingArguments
from peft import get_peft_model, LoraConfig, TaskType, prepare_model_for_kbit_training
from trl import SFTTrainer
 
model_id = "microsoft/Phi-3-mini-4k-instruct"
 
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)
 
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True,
)
 
model = prepare_model_for_kbit_training(model)
 
peft_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.05,
    bias="none",
)
 
model = get_peft_model(model, peft_config)
model.print_trainable_parameters()
# trainable params: 9,699,328 || all params: 3,831,423,488 || trainable%: 0.253
 
training_args = TrainingArguments(
    output_dir="./phi3-finetuned",
    per_device_train_batch_size=4,     # Phi-3-Mini is small, allows larger batches
    gradient_accumulation_steps=2,
    num_train_epochs=3,
    learning_rate=2e-4,
    bf16=True,
    logging_steps=10,
    save_strategy="epoch",
)

Phi-3 vs Other Small Models

Property	Phi-3-Mini (3.8B)	Gemma 2B	Mistral 7B	LLaMA 3 8B
VRAM (4-bit)	2.5 GB	1.5 GB	5 GB	5 GB
Tokens/sec (RTX 3090)	180	250	120	100
MMLU score (approx)	69%	42%	62%	66%
Long context	128K (Mini-128K)	8K	32K	8K
Edge deployment	ONNX, DirectML	TFLite	Limited	Limited
License	MIT	Gemma Terms	Apache 2.0	Llama 3

Common Mistakes

Forgetting trust_remote_code=True — Phi-3's architecture has custom code; omitting this flag raises a runtime error
Hard-coding the <|user|> format — use tokenizer.apply_chat_template() to be version-safe across Phi-3, Phi-3.5, and future Phi versions
Using float32 on GPU — float32 doubles VRAM usage unnecessarily; use torch.bfloat16 on Ampere/Ada GPUs
Expecting GPT-4-level reasoning from Phi-3-Mini — it punches above its weight but has clear limits on multi-step complex reasoning; use Phi-3-Medium for harder tasks
Using Phi-3 for long-context tasks with the 4K variant — use Phi-3-mini-128k-instruct for documents longer than a few thousand words

Best Practices

Use Phi-3-Mini-128K-Instruct for any task involving long documents — the 128K context is a significant advantage at this model size
On Apple Silicon Macs, use Ollama (ollama pull phi3) for the best performance — it leverages Metal GPU and achieves 80-100 tokens/sec on an M2 Pro
For fine-tuning, Phi-3-Mini's small size allows larger batch sizes (4-8) and faster iteration than 7B+ models
Deploy production workloads via ONNX Runtime for cross-platform support including Windows ARM, iOS, and Android
Monitor hallucination rates more carefully than with larger models — Phi-3-Mini's knowledge gaps are more frequent than in 7B+ models

Key Takeaways

Phi-3-Mini (3.8B) achieves Mistral 7B-level performance by training on high-quality curated data rather than scaling raw parameter count
The model family spans 3.8B (Mini) to 14B (Medium) with variants offering 4K and 128K context windows — pick based on task complexity and memory budget
Phi-3-Mini is the only model in its quality class that runs adequately on CPU-only hardware, making it ideal for developer laptops without a GPU
4-bit NF4 quantization brings Phi-3-Mini to ~2.5 GB VRAM — the smallest footprint of any capable instruction-following LLM in 2026
Official ONNX Runtime support enables Phi-3 deployment on iOS, Android, Raspberry Pi, and Windows ARM devices with hardware NPU acceleration
The MIT license (Phi-3) is more permissive than most open-source LLM licenses, enabling commercial use without attribution requirements
Fine-tuning Phi-3-Mini with QLoRA is very fast — its small size allows 4x larger batch sizes than 7B models on the same hardware
For tasks requiring frontier-level reasoning, Phi-3-Medium (14B) is preferable — Phi-3-Mini has measurable performance gaps on complex multi-step problems

Introduction#

Why This Matters#

Phi-3 Model Family#

Quick Start with Ollama#

Using Hugging Face Transformers#

Phi-3 Chat Format#

Running on CPU (No GPU Required)#

4-bit Quantization for Maximum Efficiency#

Multi-turn Chat Application#

Edge Deployment with ONNX Runtime#

Fine-tuning with QLoRA#

Phi-3 vs Other Small Models#

Common Mistakes#

Best Practices#

Key Takeaways#