Google Gemma — Open Source LLM Complete Guide 2026

Introduction

Why This Matters

Google's Gemma models represent the open-source distillation of techniques powering Gemini. Released under a permissive license that allows commercial use, Gemma models are built on the same research infrastructure as Google's frontier models but sized for accessibility. Gemma 2 (released mid-2024) significantly improved on the original: the 9B variant beats LLaMA 3 8B on several benchmarks, and the 27B variant competes with 70B-class models.

For developers, Gemma offers Google-quality training signals, robust safety filtering, and efficient architecture — particularly the sliding window attention in the 2B and 9B variants that improves throughput on long contexts. Understanding how to deploy and fine-tune Gemma is valuable for teams that want the quality of Google research without the cost of the Gemini API.

Gemma Model Family

Model	Parameters	Context	VRAM (fp16)	VRAM (4-bit)	Notes
Gemma-2B	2B	8K	5 GB	1.5 GB	Ultra-compact
Gemma-2B-IT	2B	8K	5 GB	1.5 GB	Instruction-tuned
Gemma-7B	7B	8K	14 GB	4.5 GB	Balanced
Gemma-7B-IT	7B	8K	14 GB	4.5 GB	Instruction-tuned
Gemma-2-9B-IT	9B	8K	18 GB	5.5 GB	Best 9B open-source
Gemma-2-27B-IT	27B	8K	54 GB	16 GB	Near-70B quality

Always use the -IT (instruction-tuned) variant for chat applications. Base variants are for pre-training continuation and specialized fine-tuning.

Quick Start with Ollama

# Gemma 2 9B — best default choice
ollama pull gemma2
ollama run gemma2 "Explain attention mechanism in transformers."
 
# Gemma 2B — minimal footprint
ollama pull gemma:2b
ollama run gemma:2b "Write a Python hello world program."
 
# Gemma 2 27B — near-70B quality (requires ~16 GB VRAM quantized)
ollama pull gemma2:27b

Using Hugging Face Transformers

Gemma models on Hugging Face require a one-time license acceptance at the model page. After approval:

pip install transformers torch accelerate
huggingface-cli login

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
 
model_id = "google/gemma-2-9b-it"   # Gemma 2 9B instruction-tuned
 
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

Gemma Chat Format

Gemma uses <start_of_turn> and <end_of_turn> tokens. Gemma 2 updated the format slightly from Gemma 1. Always use apply_chat_template:

messages = [
    {"role": "user", "content": "What is gradient descent and how does it work?"},
]
 
formatted = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
 
# The template produces something like:
# <bos><start_of_turn>user
# What is gradient descent and how does it work?<end_of_turn>
# <start_of_turn>model
#
 
inputs = tokenizer(formatted, return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=400,
    temperature=0.7,
    do_sample=True,
)
 
new_tokens = outputs[0][inputs["input_ids"].shape[1]:]
response = tokenizer.decode(new_tokens, skip_special_tokens=True)
print(response)

Multi-turn Conversation

messages = [
    {"role": "user", "content": "What is a Python decorator?"},
]
 
# First turn
formatted = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(formatted, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=300, temperature=0.7, do_sample=True)
new_tokens = outputs[0][inputs["input_ids"].shape[1]:]
response1 = tokenizer.decode(new_tokens, skip_special_tokens=True)
print("Turn 1:", response1)
 
# Second turn — append assistant response and new user message
messages.append({"role": "model", "content": response1})
messages.append({"role": "user", "content": "Can you show me an example that caches function results?"})
 
formatted2 = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs2 = tokenizer(formatted2, return_tensors="pt").to(model.device)
outputs2 = model.generate(**inputs2, max_new_tokens=400, temperature=0.7, do_sample=True)
new_tokens2 = outputs2[0][inputs2["input_ids"].shape[1]:]
response2 = tokenizer.decode(new_tokens2, skip_special_tokens=True)
print("Turn 2:", response2)

Note: Gemma uses "model" (not "assistant") as the assistant role key in the message dict.

Quantization for Consumer Hardware

from transformers import BitsAndBytesConfig
 
# 4-bit NF4 — Gemma 2 9B fits in ~5.5 GB VRAM
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)
 
model = AutoModelForCausalLM.from_pretrained(
    "google/gemma-2-9b-it",
    quantization_config=bnb_config,
    device_map="auto",
)

Using the Pipeline API

from transformers import pipeline
 
pipe = pipeline(
    "text-generation",
    model="google/gemma-2-9b-it",
    model_kwargs={"torch_dtype": torch.bfloat16},
    device_map="auto",
)
 
messages = [{"role": "user", "content": "List 5 Python best practices."}]
outputs = pipe(messages, max_new_tokens=300)
print(outputs[0]["generated_text"][-1]["content"])

Gemma Safety Features

Gemma models include safety training that causes them to decline requests for harmful content. This is by design and cannot be fully removed (unlike models like Mistral or LLaMA which have separate uncensored community variants):

def safe_generate(messages: list[dict], max_retries: int = 2) -> str:
    """Generate with graceful handling of safety refusals."""
    for attempt in range(max_retries):
        formatted = tokenizer.apply_chat_template(
            messages,
            tokenize=False,
            add_generation_prompt=True,
        )
        inputs = tokenizer(formatted, return_tensors="pt").to(model.device)
        outputs = model.generate(
            **inputs,
            max_new_tokens=300,
            temperature=0.7,
            do_sample=True,
        )
        new_tokens = outputs[0][inputs["input_ids"].shape[1]:]
        response = tokenizer.decode(new_tokens, skip_special_tokens=True).strip()
 
        if response:
            return response
 
        # If empty, rephrase as a factual request
        messages[-1]["content"] = f"Provide factual information about: {messages[-1]['content']}"
 
    return "Unable to generate a response for this request."

Fine-tuning Gemma with QLoRA

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, TrainingArguments
from peft import get_peft_model, LoraConfig, TaskType, prepare_model_for_kbit_training
from trl import SFTTrainer
 
model_id = "google/gemma-2-9b-it"
 
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)
 
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto",
)
 
model = prepare_model_for_kbit_training(model)
 
peft_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.05,
    bias="none",
)
 
model = get_peft_model(model, peft_config)
model.print_trainable_parameters()
# trainable params: 20,480,000 || all params: 9,241,927,680 || trainable%: 0.222
 
training_args = TrainingArguments(
    output_dir="./gemma-finetuned",
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,
    num_train_epochs=3,
    learning_rate=2e-4,
    bf16=True,
    logging_steps=10,
    save_strategy="epoch",
    gradient_checkpointing=True,
)

Production Deployment with FastAPI

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch
 
app = FastAPI(title="Gemma Inference API")
 
tokenizer = AutoTokenizer.from_pretrained("google/gemma-2-9b-it")
model = AutoModelForCausalLM.from_pretrained(
    "google/gemma-2-9b-it",
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.bfloat16,
    ),
    device_map="auto",
)
 
class ChatRequest(BaseModel):
    messages: list[dict]
    max_tokens: int = 512
 
@app.post("/chat")
async def chat(req: ChatRequest):
    try:
        formatted = tokenizer.apply_chat_template(
            req.messages, tokenize=False, add_generation_prompt=True
        )
        inputs = tokenizer(formatted, return_tensors="pt").to(model.device)
        outputs = model.generate(
            **inputs, max_new_tokens=req.max_tokens, temperature=0.7, do_sample=True
        )
        new_tokens = outputs[0][inputs["input_ids"].shape[1]:]
        return {"response": tokenizer.decode(new_tokens, skip_special_tokens=True)}
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

Common Mistakes

Using "assistant" as the role key — Gemma uses "model" not "assistant" for the assistant role in message dicts; using the wrong key causes chat template formatting errors
Skipping the license agreement on Hugging Face — Gemma requires accepting the terms at the model page before weights can be downloaded; this causes a 401 error that is easy to misdiagnose
Not using apply_chat_template — Gemma 2 changed the format from Gemma 1; hard-coded format strings will break with model upgrades
Expecting Gemma to fulfill harmful requests — Gemma's safety training is more conservative than Mistral or LLaMA; design your application knowing some edge-case prompts will be declined
Using float16 instead of bfloat16 — Gemma's training used bfloat16; float16 can cause numerical instability on some layers

Best Practices

Use Gemma 2-9B-IT as the default open-source model when quality is the priority and you have a GPU with 6+ GB VRAM
Apply 4-bit NF4 quantization consistently — Gemma 2 retains quality well under quantization due to its distillation-focused training
Note the "model" role key difference and consider wrapping message construction in a helper to avoid cross-model bugs
For production workloads, use vLLM which supports Gemma natively and provides continuous batching for much higher throughput
Gemma 2-27B-IT is a compelling alternative to 70B models when you have 2 GPUs with 16 GB each — it approaches 70B quality at lower hardware cost

Key Takeaways

Google's Gemma 2 family represents open-source access to techniques from the Gemini research pipeline, with the 9B variant beating LLaMA 3 8B on several benchmarks
Gemma uses "model" (not "assistant") as the assistant role key in chat message dicts — a common source of formatting bugs when switching from other models
The -IT suffix indicates instruction-tuned variants, which are required for chat applications; base models generate text continuations without following instructions
4-bit NF4 quantization fits Gemma 2-9B in ~5.5 GB VRAM, making it accessible on consumer GPUs like the RTX 3080 or 4070
Gemma includes conservative safety filtering by design; this cannot be fully disabled and should be factored into application design from the start
The Gemma license permits commercial use subject to usage policies — check the current policy at ai.google.dev/gemma/terms before commercial deployment
QLoRA fine-tuning targets all linear projection modules and updates less than 0.25% of total parameters, enabling domain adaptation on a single consumer GPU
For serving, vLLM is the recommended production engine — it provides PagedAttention-based continuous batching that increases throughput 10x over Transformers pipeline inference

Introduction#

Why This Matters#

Gemma Model Family#

Quick Start with Ollama#

Using Hugging Face Transformers#

Gemma Chat Format#

Multi-turn Conversation#

Quantization for Consumer Hardware#

Using the Pipeline API#

Gemma Safety Features#

Fine-tuning Gemma with QLoRA#

Production Deployment with FastAPI#

Common Mistakes#

Best Practices#

Key Takeaways#

Related reading