Groq API — Fastest LLM Inference Complete Guide 2026

Introduction

Why This Matters

Groq's Language Processing Unit (LPU) delivers a qualitatively different inference experience compared to GPU-based APIs. Where OpenAI or Anthropic might stream at 50-100 tokens per second with 1-2 seconds to first token, Groq delivers 500-2000 tokens per second with first-token latency below 300 milliseconds. This difference matters enormously for user experience: a chatbot that "types" 10x faster feels fundamentally different to interact with.

For developers, Groq is uniquely valuable for latency-sensitive applications: voice assistants where delay breaks the conversation flow, real-time code completion, interactive tutoring systems, and high-frequency classification pipelines. The API is OpenAI-compatible, making adoption from existing code trivial. Understanding Groq's capabilities and constraints is essential for building the next generation of responsive AI applications.

How Groq's LPU Achieves Speed

Standard GPU inference bottlenecks on memory bandwidth — moving model weights from HBM to compute cores on every forward pass. Groq's LPU uses a fundamentally different architecture:

Statically compiled execution graph — the model is compiled once; inference is deterministic execution of the compiled graph with no dynamic dispatch overhead
On-chip SRAM instead of HBM — weights are stored in fast on-chip memory rather than off-chip HBM, eliminating the memory bandwidth bottleneck
Massive parallelism without communication overhead — 230+ processing elements execute in perfect lock-step without the inter-PE communication overhead of GPU warps

The result: Groq achieves 500-2000 tokens/second at single-digit millisecond first-token latency on supported models.

Setup and Authentication

pip install groq
export GROQ_API_KEY="gsk_your_key_here"

Get a free API key at console.groq.com. The free tier provides 14,400 requests per day with generous token limits.

from groq import Groq
import os
 
client = Groq(api_key=os.environ["GROQ_API_KEY"])
 
# List available models
models = client.models.list()
for model in models.data:
    print(f"{model.id}: context={model.context_window} tokens")

Available Models (2026)

Model ID	Parameters	Context	Speed (tok/s)	Best for
llama-3.3-70b-versatile	70B	128K	275	Best quality
llama-3.1-8b-instant	8B	128K	800	Speed + quality balance
llama-3.1-70b-versatile	70B	128K	250	Long-context tasks
mixtral-8x7b-32768	56B (MoE)	32K	500	Fast, high quality
gemma2-9b-it	9B	8K	600	Efficient, capable
llama-guard-3-8b	8B	8K	—	Content moderation

Basic Chat Completion

from groq import Groq
 
client = Groq()
 
response = client.chat.completions.create(
    model="llama-3.3-70b-versatile",
    messages=[
        {"role": "system", "content": "You are a concise technical assistant."},
        {"role": "user", "content": "Explain the difference between TCP and UDP."},
    ],
    max_tokens=400,
    temperature=0.7,
)
 
print(response.choices[0].message.content)
print(f"\nUsage: {response.usage.prompt_tokens} prompt + {response.usage.completion_tokens} completion tokens")

Streaming for Real-time Output

Groq's streaming is exceptionally fast — even streaming feels near-instantaneous at 500+ tokens/second:

import time
 
def stream_with_latency_measurement(prompt: str, model: str = "llama-3.1-8b-instant") -> str:
    """Stream response and measure time to first token."""
    start = time.perf_counter()
    first_token_time = None
    full_response = ""
 
    stream = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        max_tokens=500,
        stream=True,
    )
 
    for chunk in stream:
        delta = chunk.choices[0].delta.content
        if delta:
            if first_token_time is None:
                first_token_time = time.perf_counter()
                print(f"[First token: {(first_token_time - start) * 1000:.0f}ms]")
            print(delta, end="", flush=True)
            full_response += delta
 
    elapsed = time.perf_counter() - start
    tokens = len(full_response.split())
    print(f"\n[Total: {elapsed:.2f}s, ~{tokens / elapsed:.0f} words/sec]")
    return full_response
 
stream_with_latency_measurement("Write a Python function that implements quicksort.")

Multi-turn Conversation

from groq import Groq
 
class GroqChat:
    def __init__(
        self,
        model: str = "llama-3.3-70b-versatile",
        system: str = "You are a helpful assistant.",
    ):
        self.client = Groq()
        self.model = model
        self.messages = [{"role": "system", "content": system}]
 
    def chat(self, user_input: str, max_tokens: int = 500) -> str:
        self.messages.append({"role": "user", "content": user_input})
 
        response = self.client.chat.completions.create(
            model=self.model,
            messages=self.messages,
            max_tokens=max_tokens,
            temperature=0.7,
        )
 
        assistant_content = response.choices[0].message.content
        self.messages.append({"role": "assistant", "content": assistant_content})
        return assistant_content
 
    def stream_chat(self, user_input: str, max_tokens: int = 500) -> str:
        self.messages.append({"role": "user", "content": user_input})
        full_response = ""
 
        stream = self.client.chat.completions.create(
            model=self.model,
            messages=self.messages,
            max_tokens=max_tokens,
            stream=True,
        )
 
        for chunk in stream:
            delta = chunk.choices[0].delta.content
            if delta:
                print(delta, end="", flush=True)
                full_response += delta
        print()
 
        self.messages.append({"role": "assistant", "content": full_response})
        return full_response
 
    def reset(self):
        self.messages = [self.messages[0]]
 
# Usage
bot = GroqChat(system="You are an expert Python engineer.")
bot.stream_chat("What is the GIL and why does it matter?")
bot.stream_chat("What are the alternatives to working around it?")

LangChain Integration

pip install langchain langchain-groq

from langchain_groq import ChatGroq
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
 
llm = ChatGroq(
    model="llama-3.3-70b-versatile",
    temperature=0.7,
    max_tokens=500,
)
 
# Simple chain
prompt = ChatPromptTemplate.from_template(
    "You are a {role}. Explain {concept} clearly and concisely."
)
chain = prompt | llm | StrOutputParser()
 
result = chain.invoke({"role": "senior engineer", "concept": "microservices architecture"})
print(result)

# RAG chain with Groq for fast answer generation
from langchain_community.vectorstores import Chroma
from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain.chains import RetrievalQA
from langchain_core.documents import Document
 
embeddings = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2")
 
docs = [
    Document(page_content="Python is a high-level, interpreted programming language."),
    Document(page_content="Python uses dynamic typing and garbage collection."),
    Document(page_content="Python is popular for data science, web development, and automation."),
]
 
vectorstore = Chroma.from_documents(docs, embedding=embeddings)
qa = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=vectorstore.as_retriever(search_kwargs={"k": 2}),
)
 
answer = qa.run("What is Python used for?")
print(answer)

JSON Mode and Structured Output

import json
 
response = client.chat.completions.create(
    model="llama-3.1-8b-instant",
    messages=[
        {
            "role": "system",
            "content": "You are a data extractor. Always respond with valid JSON.",
        },
        {
            "role": "user",
            "content": "Extract the name, age, and skills from: 'Alice is a 28-year-old Python and Go developer.'",
        },
    ],
    max_tokens=200,
    temperature=0.1,
    response_format={"type": "json_object"},
)
 
data = json.loads(response.choices[0].message.content)
print(data)
# {"name": "Alice", "age": 28, "skills": ["Python", "Go"]}

Building a Real-time FastAPI Endpoint

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from pydantic import BaseModel
from groq import Groq
 
app = FastAPI(title="Groq Real-time Chat API")
client = Groq()
 
class ChatRequest(BaseModel):
    messages: list[dict]
    model: str = "llama-3.1-8b-instant"
    max_tokens: int = 500
 
@app.post("/chat")
async def chat(req: ChatRequest):
    response = client.chat.completions.create(
        model=req.model,
        messages=req.messages,
        max_tokens=req.max_tokens,
        temperature=0.7,
    )
    return {"response": response.choices[0].message.content}
 
@app.post("/stream")
async def stream(req: ChatRequest):
    def generate():
        stream = client.chat.completions.create(
            model=req.model,
            messages=req.messages,
            max_tokens=req.max_tokens,
            stream=True,
        )
        for chunk in stream:
            delta = chunk.choices[0].delta.content
            if delta:
                yield delta
 
    return StreamingResponse(generate(), media_type="text/plain")
 
# Run: uvicorn app:app --host 0.0.0.0 --port 8000
# Stream: curl -X POST http://localhost:8000/stream -H "Content-Type: application/json" \
#         -d '{"messages": [{"role": "user", "content": "Hello!"}]}'

Rate Limits and Caching Strategy

import time
from functools import lru_cache
from groq import Groq, RateLimitError
 
client = Groq()
 
@lru_cache(maxsize=1000)
def cached_completion(prompt: str, model: str = "llama-3.1-8b-instant") -> str:
    """Cache identical prompts to avoid hitting rate limits."""
    response = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        max_tokens=200,
    )
    return response.choices[0].message.content
 
def completion_with_backoff(prompt: str, max_retries: int = 3) -> str:
    """Retry on rate limit with exponential backoff."""
    for attempt in range(max_retries):
        try:
            return cached_completion(prompt)
        except RateLimitError:
            wait = 2 ** attempt
            print(f"Rate limited. Waiting {wait}s... (attempt {attempt + 1})")
            time.sleep(wait)
    raise RuntimeError("Max retries exceeded")

Groq free tier limits (2026): 14,400 requests/day, 500,000 tokens/minute for most models.

Groq vs Other Inference APIs

Provider	Latency (first token)	Speed (tok/s)	GPT-4 class?	Free tier
Groq	~200ms	500-2000	No (uses open models)	14,400 req/day
OpenAI	~800ms	80-150	Yes (GPT-4o)	No
Anthropic	~1000ms	50-100	Yes (Claude 3.5)	No
Together AI	~600ms	100-200	No (open models)	$1 credit
DeepSeek	~500ms	60-100	Yes (DeepSeek-V3)	Limited

Common Mistakes

Sending very large context for speed-sensitive tasks — Groq is fastest on short-to-medium contexts; very long prompts increase latency even on LPU hardware
Not using response_format={"type": "json_object"} for structured output — without it, JSON parsing can fail on malformed model output
Hitting rate limits by not caching — deterministic queries (FAQ answers, fixed-prompt classification) should be cached; the same question asked 1000 times consumes 1000 requests unnecessarily
Using max_tokens too high for fast-response use cases — if you need a short answer, set max_tokens accordingly; the model will not stop sooner unless you tell it to
Ignoring model context limits — mixtral-8x7b-32768 supports 32K tokens, not unlimited; exceeding the limit causes an API error

Best Practices

Use llama-3.1-8b-instant for latency-critical paths and llama-3.3-70b-versatile for quality-critical paths — the 8B model is 3x faster at only moderate quality reduction
Always stream for interactive applications — even though Groq is fast, streaming makes the UI feel responsive from the first token
Cache frequently repeated prompts using an in-memory LRU cache or Redis to reduce both API usage and cost
Use JSON mode (response_format={"type": "json_object"}) for any structured data extraction pipeline — it eliminates JSON parsing failures
Monitor your usage at console.groq.com to avoid surprises; set up alerts if you have automated pipelines that could consume quota unexpectedly

Key Takeaways

Groq's LPU achieves 500-2000 tokens/second with sub-300ms first-token latency by storing model weights in on-chip SRAM and using a statically compiled execution graph
The Groq API is fully OpenAI-compatible — switching from OpenAI requires changing only the base URL and API key in the client initialization
llama-3.3-70b-versatile is the best quality model on Groq; llama-3.1-8b-instant is the fastest for latency-critical applications
JSON mode (response_format={"type": "json_object"}) ensures reliable structured output for data extraction pipelines
The free tier provides 14,400 requests per day — sufficient for development, staging, and low-traffic production
LangChain's ChatGroq class integrates Groq natively with all LangChain chains, agents, and RAG pipelines
Groq does not offer GPT-4-class model quality — it runs open-source models (LLaMA 3, Mixtral, Gemma); for frontier quality with speed, combine Groq for drafts with GPT-4o for final outputs
Streaming on Groq delivers the complete response so fast that for short answers, non-streaming and streaming completion times are nearly indistinguishable

Introduction#

Why This Matters#

How Groq's LPU Achieves Speed#

Setup and Authentication#

Available Models (2026)#

Basic Chat Completion#

Streaming for Real-time Output#

Multi-turn Conversation#

LangChain Integration#

JSON Mode and Structured Output#

Building a Real-time FastAPI Endpoint#

Rate Limits and Caching Strategy#

Groq vs Other Inference APIs#

Common Mistakes#

Best Practices#

Key Takeaways#