LLaMA 3 — Complete Setup, Usage, and Optimization Guide 2026

Introduction

Why This Matters

Meta's LLaMA 3 family is the most widely used open-source LLM architecture in 2026. The 8B variant outperforms older 13B models on most benchmarks while running on consumer GPUs. The 70B variant is competitive with GPT-4-class models on coding, reasoning, and long-context tasks. Because the weights are openly licensed, you can download, inspect, quantize, fine-tune, and deploy LLaMA 3 without any API dependency or usage policy restrictions.

Understanding how to set up and optimize LLaMA 3 is a foundational skill for AI engineers building applications that require predictable costs, offline capability, or data privacy. This guide covers every practical deployment path from Ollama to full Transformers to quantized llama.cpp.

LLaMA 3 Model Variants

Meta released LLaMA 3 in two base sizes, each with a base (pretrained) and an instruct (instruction-tuned) variant:

Model	Parameters	Context	VRAM (fp16)	VRAM (4-bit)
Llama-3-8B	8B	8K tokens	16 GB	5 GB
Llama-3-8B-Instruct	8B	8K tokens	16 GB	5 GB
Llama-3-70B	70B	8K tokens	140 GB	40 GB
Llama-3-70B-Instruct	70B	8K tokens	140 GB	40 GB

Always use the -Instruct variant for chat and Q&A applications. The base variant generates arbitrary text continuations and is intended for further pre-training or specialized fine-tuning.

Quickest Path: Ollama

For most developers, Ollama is the fastest way to run LLaMA 3 locally with no access approval required:

# Pull and run LLaMA 3 8B (4.7 GB download)
ollama pull llama3
ollama run llama3
 
# LLaMA 3 70B (requires ~40 GB RAM)
ollama pull llama3:70b
ollama run llama3:70b
 
# Python usage
pip install ollama

import ollama
 
response = ollama.chat(
    model="llama3",
    messages=[{"role": "user", "content": "Explain transformer attention in plain English."}],
)
print(response["message"]["content"])

Accessing via Hugging Face Transformers

LLaMA 3 on Hugging Face requires a one-time access approval. Visit meta-llama/Meta-Llama-3-8B-Instruct on the Hub, accept the license, and wait for approval (typically a few minutes).

pip install transformers torch accelerate bitsandbytes
huggingface-cli login

from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
import torch
 
model_id = "meta-llama/Meta-Llama-3-8B-Instruct"
 
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,   # bfloat16 is preferred over float16 for LLaMA 3
    device_map="auto",
)
 
# Use the pipeline abstraction for convenience
pipe = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
)

LLaMA 3 Chat Format

LLaMA 3 uses a different chat template than LLaMA 2. The new format uses special tokens and a structured header. The tokenizer's apply_chat_template method handles formatting automatically:

messages = [
    {"role": "system", "content": "You are a helpful AI assistant specialized in Python."},
    {"role": "user", "content": "What is a Python generator and when should I use one?"},
]
 
# Correct way: let the tokenizer apply the template
formatted_prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
 
inputs = tokenizer(formatted_prompt, return_tensors="pt").to(model.device)
 
outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.7,
    do_sample=True,
    pad_token_id=tokenizer.eos_token_id,
)
 
# Decode only the new tokens (skip the prompt)
new_tokens = outputs[0][inputs["input_ids"].shape[1]:]
response = tokenizer.decode(new_tokens, skip_special_tokens=True)
print(response)

Never hard-code the chat format string — it changes between model versions and the tokenizer's apply_chat_template is always authoritative.

Quantization for Consumer Hardware

4-bit quantization with bitsandbytes (NF4) lets the 8B model fit in 5 GB VRAM — within reach of an RTX 3060 or M2 MacBook Air.

from transformers import BitsAndBytesConfig
 
# 4-bit NF4 quantization — best quality-to-size ratio
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,   # Second quantization for extra savings
)
 
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3-8B-Instruct",
    quantization_config=bnb_config,
    device_map="auto",
)
 
# 8-bit quantization — slightly more VRAM, slightly better quality
bnb_config_8bit = BitsAndBytesConfig(load_in_8bit=True)
 
model_8bit = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3-8B-Instruct",
    quantization_config=bnb_config_8bit,
    device_map="auto",
)

Enabling Flash Attention 2 for Speed

Flash Attention 2 dramatically reduces memory and speeds up generation by recomputing attention scores block-by-block rather than materializing the full attention matrix.

pip install flash-attn --no-build-isolation

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3-8B-Instruct",
    attn_implementation="flash_attention_2",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
# Flash Attention 2 requires bfloat16 or float16 — not float32

Flash Attention 2 is transparent: the model produces identical outputs, just faster and with lower VRAM peak usage.

Fine-tuning with QLoRA

QLoRA (Quantized LoRA) enables fine-tuning a 4-bit quantized model on a single consumer GPU. Only the small LoRA adapter layers are updated; the frozen quantized base model stays at 4-bit.

from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments
from peft import get_peft_model, LoraConfig, TaskType, prepare_model_for_kbit_training
from trl import SFTTrainer
from datasets import Dataset
 
# Load quantized base model
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3-8B-Instruct",
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.bfloat16,
    ),
    device_map="auto",
)
 
# Prepare for k-bit training
model = prepare_model_for_kbit_training(model)
 
# LoRA config
peft_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16,                       # Rank — higher = more capacity, more VRAM
    lora_alpha=32,              # Scaling factor
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
)
 
model = get_peft_model(model, peft_config)
model.print_trainable_parameters()
# trainable params: 6,815,744 || all params: 8,037,636,096 || trainable%: 0.0848
 
# Training arguments
training_args = TrainingArguments(
    output_dir="./llama3-finetuned",
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,
    num_train_epochs=3,
    learning_rate=2e-4,
    fp16=True,
    save_strategy="epoch",
    logging_steps=10,
)

Benchmarking Generation Speed

import time
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
 
model_id = "meta-llama/Meta-Llama-3-8B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
 
def benchmark(prompt: str, max_new_tokens: int = 200) -> dict:
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    num_input_tokens = inputs["input_ids"].shape[1]
 
    start = time.perf_counter()
    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=False,  # Greedy for deterministic benchmarks
        )
    elapsed = time.perf_counter() - start
 
    num_output_tokens = outputs.shape[1] - num_input_tokens
    return {
        "input_tokens": num_input_tokens,
        "output_tokens": num_output_tokens,
        "elapsed_s": round(elapsed, 2),
        "tokens_per_sec": round(num_output_tokens / elapsed, 1),
    }
 
result = benchmark("Explain the transformer architecture in detail.")
print(result)
# {'input_tokens': 12, 'output_tokens': 200, 'elapsed_s': 4.1, 'tokens_per_sec': 48.8}

Common Mistakes

Using the wrong chat template — LLaMA 3 uses a completely different format from LLaMA 2; always call tokenizer.apply_chat_template() rather than hard-coding format strings
Loading in float32 by default — Transformers defaults to float32; always pass torch_dtype=torch.bfloat16 explicitly to halve VRAM usage
Not setting pad_token_id — LLaMA 3 has no pad token by default; set pad_token_id=tokenizer.eos_token_id to avoid generation warnings
Forgetting prepare_model_for_kbit_training — quantized models need this call before LoRA application; omitting it causes training instability
Benchmarking with do_sample=True — sampling adds non-deterministic overhead; use greedy decoding (do_sample=False) for reproducible speed measurements

Best Practices

Always load with torch_dtype=torch.bfloat16 — it is faster and uses half the memory of float32 with minimal quality difference
Use device_map="auto" to automatically distribute layers across available GPUs and CPU RAM when the model does not fit on a single device
For production inference serving, use vLLM or Text Generation Inference (TGI) rather than raw Transformers — both provide batching and continuous batching for much higher throughput
Cache tokenized prompts when running the same system prompt repeatedly to eliminate redundant tokenization overhead
Enable Flash Attention 2 whenever hardware supports it — it provides 20-40% speedup on long contexts at no quality cost

Key Takeaways

LLaMA 3 8B-Instruct outperforms LLaMA 2 13B on most benchmarks while using less memory, making it the default choice for consumer GPU deployments
Always use tokenizer.apply_chat_template() to format messages — LLaMA 3's chat format is different from LLaMA 2 and must not be hard-coded
4-bit NF4 quantization via bitsandbytes reduces the 8B model to ~5 GB VRAM with minimal quality degradation
QLoRA enables fine-tuning the 8B model on a single RTX 3090 by only updating small LoRA adapter layers (less than 0.1% of parameters)
Flash Attention 2 (attn_implementation="flash_attention_2") provides 20-40% speed gains and lower peak VRAM with no change in outputs
The easiest local deployment path is Ollama — one command downloads and runs the model with automatic GPU detection
For production serving, use vLLM or TGI — they provide continuous batching, which increases throughput 10x over naive Transformers inference
The 70B variant is competitive with GPT-4-class models but requires at least 40 GB RAM or multi-GPU setup even at 4-bit quantization

Introduction#

Why This Matters#

LLaMA 3 Model Variants#

Quickest Path: Ollama#

Accessing via Hugging Face Transformers#

LLaMA 3 Chat Format#

Quantization for Consumer Hardware#

Enabling Flash Attention 2 for Speed#

Fine-tuning with QLoRA#

Benchmarking Generation Speed#

Common Mistakes#

Best Practices#

Key Takeaways#

Related reading