Ollama 2026 — Run LLMs Locally for Free: Complete Guide

Introduction#

Why This Matters#

Cloud LLM APIs charge $5-$15 per million tokens. For high-volume applications — code completion, document summarization, local RAG — those costs compound fast. Ollama lets you run Llama 3, Mistral, DeepSeek-R1, Gemma 3, and 100+ other open-source models locally with a single command.

For privacy-sensitive industries (healthcare, legal, finance), local inference means data never leaves your infrastructure. For developers, it means unlimited experimentation with no API bills. For production edge deployments, it means inference without internet dependency.

On Apple Silicon (M1/M2/M3) and NVIDIA GPUs, Ollama achieves near-cloud performance through Metal and CUDA acceleration. A 7B model runs at 30-40 tokens per second on a MacBook Pro — fast enough for real-time chat and autocomplete.

Installation#

# macOS and Linux
curl -fsSL https://ollama.com/install.sh | sh
 
# Windows: download installer from https://ollama.com/download
 
# Verify
ollama --version

Pull and Run Models#

ollama pull llama3.2          # 3B — great for laptops (4GB RAM)
ollama pull llama3.3          # 70B — state of the art (48GB RAM)
ollama pull mistral           # 7B — fast, high quality
ollama pull gemma3            # Google Gemma 3
ollama pull deepseek-r1       # Exceptional reasoning
ollama pull codellama         # Optimized for code
ollama pull phi4              # Microsoft — compact and smart
ollama pull qwen2.5-coder     # Strong coding model
 
# Start a chat session
ollama run llama3.2

REST API#

Ollama exposes a REST API on localhost:11434 that accepts the same request format as OpenAI:

# Generate endpoint
curl http://localhost:11434/api/generate \
  -d '{"model": "llama3.2", "prompt": "Why is the sky blue?", "stream": false}'
 
# Chat endpoint
curl http://localhost:11434/api/chat \
  -d '{
    "model": "llama3.2",
    "messages": [{"role": "user", "content": "Write a Python quicksort"}],
    "stream": false
  }'

Python Integration#

import ollama
 
# Simple generation
response = ollama.generate(model='llama3.2', prompt='Explain closures in JavaScript')
print(response['response'])
 
# Multi-turn chat
messages = [
    {'role': 'system', 'content': 'You are a senior engineer. Be concise.'},
    {'role': 'user', 'content': 'Difference between TCP and UDP?'},
]
response = ollama.chat(model='llama3.2', messages=messages)
print(response['message']['content'])
 
# Streaming
for chunk in ollama.generate(model='llama3.2', prompt='Write a merge sort', stream=True):
    print(chunk['response'], end='', flush=True)

OpenAI-Compatible Mode#

Ollama speaks the OpenAI API format — drop it into any OpenAI app for free local development:

from openai import OpenAI
 
client = OpenAI(
    base_url='http://localhost:11434/v1',
    api_key='ollama'  # any string works
)
 
response = client.chat.completions.create(
    model='llama3.2',
    messages=[{'role': 'user', 'content': 'Write a binary search in Python'}]
)
print(response.choices[0].message.content)

This means zero code changes to switch between Ollama (dev/local) and OpenAI (production).

Local RAG App#

from langchain_ollama import ChatOllama, OllamaEmbeddings
from langchain_community.vectorstores import Chroma
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.document_loaders import PyPDFLoader
 
# 100% local — no API keys, no data leaving your machine
llm = ChatOllama(model="llama3.2", temperature=0)
embeddings = OllamaEmbeddings(model="nomic-embed-text")
 
loader = PyPDFLoader("confidential_report.pdf")
chunks = RecursiveCharacterTextSplitter(chunk_size=500).split_documents(loader.load())
db = Chroma.from_documents(chunks, embeddings)
 
docs = db.as_retriever(search_kwargs={"k": 3}).invoke("What is the main finding?")
context = "\n".join([d.page_content for d in docs])
response = llm.invoke(f"Context: {context}\n\nQuestion: What is the main finding?")
print(response.content)

Custom Modelfiles#

FROM llama3.2
 
SYSTEM """
You are an expert Python developer who writes clean, well-commented code.
Always include type hints. Always handle exceptions. Use f-strings.
"""
 
PARAMETER temperature 0.1
PARAMETER num_ctx 8192

ollama create python-expert -f Modelfile
ollama run python-expert

Performance Guide#

Apple Silicon uses Metal GPU acceleration for 3-5x speedup over CPU. NVIDIA GPUs via CUDA deliver near-cloud performance.

Common Mistakes / Pitfalls#

• Pulling 70B models without enough RAM — they swap to disk and run at 1-2 tokens/s; start with 3B-8B
• Not using the OpenAI compatibility layer — you can use your existing OpenAI code unchanged
• Forgetting to keep Ollama running — it must be active as a background service before API calls
• Using default context length for long documents — increase num_ctx in the Modelfile for RAG tasks
• Not benchmarking on your hardware — token speeds vary significantly between GPU and CPU inference

Best Practices#

• Use nomic-embed-text for local embeddings — it matches the quality of text-embedding-3-small at zero cost
• Keep Ollama updated — new model support and performance improvements ship frequently
• Use quantized 4-bit models (Q4_K_M) for the best quality-to-speed ratio on consumer hardware
• Set OLLAMA_NUM_PARALLEL=4 to handle multiple concurrent requests in server mode
• Integrate with the Continue VS Code extension for free, private AI code completion

Key Takeaways#

• Ollama runs Llama, Mistral, Gemma, DeepSeek, and 100+ open-source models locally with one command
• The REST API is OpenAI-compatible — zero code changes needed to switch from cloud to local inference
• Local inference costs $0 per token — unlimited usage once the model is downloaded
• Data never leaves your machine — critical for HIPAA, GDPR, and confidential enterprise workloads
• Apple Silicon (M1/M2/M3) achieves 30-80 tokens/second via Metal GPU acceleration
• Custom Modelfiles let you bake in system prompts, personalities, and parameter settings permanently
• A 7B model with 8GB RAM delivers very good quality for most coding and writing tasks
• Ollama integrates with LangChain, LlamaIndex, Continue, and the entire LLM ecosystem