Hugging Face Transformers — Complete Guide for LLM Inference and Fine-tuning

Introduction

Why This Matters

The Hugging Face Transformers library is the foundation of open-source LLM development. With 130,000+ GitHub stars and downloads in the billions, it provides a unified Python API for loading, running, and fine-tuning virtually every open-source model: LLaMA 3, Mistral, Phi-3, Gemma, Falcon, Qwen, DeepSeek, and hundreds more.

Without Transformers, you would need to implement tokenization, model architecture, and generation loops from scratch for each model family. Transformers provides AutoModel, AutoTokenizer, and pipeline() — three abstractions that work across all architectures through a common interface.

Mastering Hugging Face Transformers means you can run any open-source LLM locally, quantize it to reduce memory, fine-tune it on custom data, and serve it in production — all using the same library.

Installation

pip install transformers torch accelerate bitsandbytes sentencepiece

For full fine-tuning support:

pip install transformers[torch] datasets peft trl evaluate

The pipeline() API

The highest-level abstraction — get results in three lines:

from transformers import pipeline
 
# Text generation
generator = pipeline(
    "text-generation",
    model="microsoft/Phi-3-mini-4k-instruct",
    device_map="auto",
    torch_dtype="auto"
)
 
result = generator(
    "Explain transformer attention in one paragraph:",
    max_new_tokens=200,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
)
 
print(result[0]["generated_text"])

Other common pipeline tasks:

# Classification
classifier = pipeline("zero-shot-classification", model="facebook/bart-large-mnli")
result = classifier("I need to cancel my subscription", candidate_labels=["billing", "technical", "cancellation"])
 
# Summarization
summarizer = pipeline("summarization", model="facebook/bart-large-cnn")
summary = summarizer(long_text, max_length=130, min_length=30)
 
# Embeddings / feature extraction
extractor = pipeline("feature-extraction", model="BAAI/bge-base-en-v1.5")
embedding = extractor("Hello world")[0][0]  # (1, seq_len, hidden_size) -> first token

AutoModel and AutoTokenizer

For more control, use the Auto classes directly:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
 
model_name = "meta-llama/Meta-Llama-3-8B-Instruct"
 
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,  # use bfloat16 for memory efficiency
    device_map="auto"            # auto-distribute across available GPUs
)
 
# Tokenize and generate
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is RAG in LLM applications?"}
]
 
# Apply chat template (model-specific formatting)
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
 
with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=512,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        pad_token_id=tokenizer.eos_token_id
    )
 
# Decode only the generated tokens (exclude input)
new_tokens = outputs[0][inputs.input_ids.shape[1]:]
response = tokenizer.decode(new_tokens, skip_special_tokens=True)
print(response)

Quantization for Memory Efficiency

Run large models on consumer hardware with bitsandbytes quantization:

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
 
# 4-bit NF4 quantization (recommended)
bnb_config_4bit = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype="bfloat16",
    bnb_4bit_use_double_quant=True
)
 
# 8-bit quantization (better quality, more memory)
bnb_config_8bit = BitsAndBytesConfig(load_in_8bit=True)
 
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3-70B-Instruct",
    quantization_config=bnb_config_4bit,  # 70B model in ~42GB VRAM
    device_map="auto"
)
 
print(f"Model footprint: {model.get_memory_footprint() / 1e9:.1f} GB")

Memory comparison for LLaMA 3 70B:

Full precision (fp32): ~280GB — impossible on consumer hardware
Half precision (bf16): ~140GB — requires 4x A100 80GB
8-bit: ~70GB — 2x A100 80GB
4-bit NF4: ~42GB — fits on single A100 80GB

Generation Configuration

Fine-tune text generation behavior:

from transformers import GenerationConfig
 
# Greedy decoding (deterministic)
greedy_config = GenerationConfig(
    max_new_tokens=512,
    do_sample=False,  # greedy
    repetition_penalty=1.1  # penalize repeating tokens
)
 
# Sampling with nucleus filtering
sampling_config = GenerationConfig(
    max_new_tokens=512,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
    top_k=50,
    repetition_penalty=1.1
)
 
# Beam search (best for translation, summarization)
beam_config = GenerationConfig(
    max_new_tokens=256,
    num_beams=4,
    early_stopping=True,
    no_repeat_ngram_size=3
)
 
outputs = model.generate(**inputs, generation_config=sampling_config)

Working with Chat Templates

Each model family has its own prompt format. apply_chat_template() handles this correctly:

from transformers import AutoTokenizer
 
# LLaMA 3 format
llama_tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")
 
messages = [
    {"role": "system", "content": "You are a Python expert."},
    {"role": "user", "content": "How do I read a CSV file?"},
]
 
# Produces the model-specific prompt format
prompt = llama_tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
print(prompt)
# <|begin_of_text|><|start_header_id|>system<|end_header_id|>...

Streaming Generation

Stream tokens as they are generated for responsive UIs:

from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer
import torch
 
model_name = "microsoft/Phi-3-mini-4k-instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto", device_map="auto")
 
streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
 
inputs = tokenizer("Explain the transformer architecture:", return_tensors="pt").to(model.device)
 
with torch.no_grad():
    model.generate(
        **inputs,
        max_new_tokens=300,
        streamer=streamer,  # prints tokens as they are generated
        do_sample=True,
        temperature=0.7
    )

Saving and Loading Models

# Save model and tokenizer locally
model.save_pretrained("./my-model")
tokenizer.save_pretrained("./my-model")
 
# Load from local directory
model = AutoModelForCausalLM.from_pretrained("./my-model", device_map="auto")
 
# Push to Hugging Face Hub
model.push_to_hub("your-username/your-model-name")
tokenizer.push_to_hub("your-username/your-model-name")
 
# Download a specific model revision
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3-8B-Instruct",
    revision="main",  # or specific commit hash
    cache_dir="/path/to/cache"  # custom download location
)

Common Mistakes / Pitfalls

Not using apply_chat_template() — raw prompts fed to instruct models produce poor results
Forgetting pad_token_id=tokenizer.eos_token_id in generate() — causes warning and potential issues with batching
Loading in fp32 by default — always specify torch_dtype=torch.bfloat16 to halve memory usage
Not using torch.no_grad() during inference — wastes memory storing unnecessary gradients
Calling model.generate() on CPU for large models — always use device_map="auto" for GPU placement

Best Practices

Always use apply_chat_template() for instruct/chat models — raw string formatting is error-prone
Load in bfloat16 by default (torch_dtype=torch.bfloat16) — reduces memory by 2x with negligible quality impact
Use device_map="auto" to automatically distribute model layers across available GPUs
Enable use_cache=True (default) for generation — KV-cache dramatically speeds up autoregressive decoding
For batch inference, use the DataLoader pattern with tokenizer.pad_token and attention_mask

Key Takeaways

Hugging Face Transformers provides a unified Python API for 130,000+ models with pipeline(), AutoModel, and AutoTokenizer
apply_chat_template() correctly formats messages for each model's specific prompt structure — never format manually
4-bit NF4 quantization via BitsAndBytesConfig reduces a 70B model from 140GB to 42GB VRAM
device_map="auto" automatically distributes model layers across all available GPUs
TextStreamer enables token-by-token streaming output for responsive UI experiences
Load models in bfloat16 by default — half the memory of float32 with negligible accuracy loss
GenerationConfig controls all sampling parameters — temperature, top_p, top_k, repetition_penalty, beam_search
Models and tokenizers can be saved locally with save_pretrained() and shared on the Hub with push_to_hub()

Introduction#

Why This Matters#

Installation#

The pipeline() API#

AutoModel and AutoTokenizer#

Quantization for Memory Efficiency#

Generation Configuration#

Working with Chat Templates#

Streaming Generation#

Saving and Loading Models#

Common Mistakes / Pitfalls#

Best Practices#

Key Takeaways#

Related reading