Together AI — Run Open-Source LLMs in the Cloud Complete Guide 2026

Introduction

Why This Matters

Together AI occupies a unique position in the LLM infrastructure landscape: it hosts over 200 open-source models behind a unified OpenAI-compatible API, offers the lowest per-token pricing for quality inference among managed services, and provides managed fine-tuning and custom model deployment without requiring you to manage GPUs. For teams that want the flexibility of open-source models with the convenience of a managed API — and without OpenAI's restrictive terms — Together AI is the natural platform in 2026.

For ML engineers building production systems, Together AI provides a cost-effective path to running Llama 3, Mistral, Mixtral, Qwen, DeepSeek, and hundreds of other models through a single SDK. Understanding the platform's capabilities — from embeddings to fine-tuning to serverless inference — is essential for architects designing cost-efficient AI pipelines.

Getting Started

pip install together
export TOGETHER_API_KEY="your-api-key-here"

Get an API key and $1 of free credits at api.together.ai.

from together import Together
import os
 
client = Together(api_key=os.environ["TOGETHER_API_KEY"])
 
# List available models
models = client.models.list()
for model in models[:10]:
    print(f"{model.id}: {model.type}")

Popular Models Available on Together AI

Model	Parameters	Input ($/1M)	Output ($/1M)	Best for
meta-llama/Llama-3.3-70B-Instruct-Turbo	70B	$0.88	$0.88	Best open-source quality
meta-llama/Llama-3.1-8B-Instruct-Turbo	8B	$0.18	$0.18	Fast, cheap general tasks
mistralai/Mistral-7B-Instruct-v0.3	7B	$0.20	$0.20	Efficient instruction following
mistralai/Mixtral-8x7B-Instruct-v0.1	56B	$0.60	$0.60	High quality, efficient MoE
Qwen/Qwen2.5-72B-Instruct-Turbo	72B	$1.20	$1.20	Multilingual, coding
deepseek-ai/DeepSeek-R1	671B	$3.00	$7.00	Frontier reasoning
google/gemma-2-27b-it	27B	$0.80	$0.80	Google quality, balanced
BAAI/bge-large-en-v1.5	—	$0.02	—	Embeddings

Chat Completions (OpenAI-compatible)

Together AI's API is fully OpenAI-compatible. You can use the OpenAI Python SDK directly:

from openai import OpenAI
 
# Together AI via OpenAI SDK
client = OpenAI(
    api_key=os.environ["TOGETHER_API_KEY"],
    base_url="https://api.together.xyz/v1",
)
 
response = client.chat.completions.create(
    model="meta-llama/Llama-3.3-70B-Instruct-Turbo",
    messages=[
        {"role": "system", "content": "You are a senior software engineer."},
        {"role": "user", "content": "Explain the SOLID principles with Python examples."},
    ],
    max_tokens=800,
    temperature=0.7,
)
 
print(response.choices[0].message.content)
print(f"\nCost: ${response.usage.total_tokens * 0.88 / 1_000_000:.6f}")

Or use the native Together SDK:

from together import Together
 
client = Together()
 
response = client.chat.completions.create(
    model="meta-llama/Llama-3.3-70B-Instruct-Turbo",
    messages=[
        {"role": "user", "content": "What are the key differences between REST and GraphQL?"},
    ],
    max_tokens=500,
    temperature=0.7,
    stop=["<|eot_id|>"],
)
 
print(response.choices[0].message.content)

Streaming Responses

from together import Together
 
client = Together()
 
def stream_response(prompt: str, model: str = "meta-llama/Llama-3.1-8B-Instruct-Turbo") -> str:
    """Stream tokens and return full response."""
    full_text = ""
 
    stream = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        max_tokens=600,
        stream=True,
    )
 
    for chunk in stream:
        delta = chunk.choices[0].delta.content
        if delta:
            print(delta, end="", flush=True)
            full_text += delta
    print()
    return full_text
 
stream_response("Write a Python class implementing a binary search tree with insert, search, and delete.")

Generating Embeddings

Together AI hosts several embedding models. The BAAI/bge-large-en-v1.5 model is a popular choice for RAG applications:

from together import Together
 
client = Together()
 
def get_embeddings(texts: list[str], model: str = "BAAI/bge-large-en-v1.5") -> list[list[float]]:
    """Generate embeddings for a list of texts."""
    response = client.embeddings.create(
        model=model,
        input=texts,
    )
    return [item.embedding for item in response.data]
 
texts = [
    "Python is a high-level programming language.",
    "JavaScript runs in the browser and on servers.",
    "Rust provides memory safety without garbage collection.",
]
 
embeddings = get_embeddings(texts)
print(f"Embedding dimension: {len(embeddings[0])}")  # 1024 for bge-large-en

Building a RAG Pipeline

import numpy as np
from together import Together
 
client = Together()
 
class TogetherRAG:
    def __init__(
        self,
        embed_model: str = "BAAI/bge-large-en-v1.5",
        chat_model: str = "meta-llama/Llama-3.1-8B-Instruct-Turbo",
    ):
        self.client = Together()
        self.embed_model = embed_model
        self.chat_model = chat_model
        self.documents: list[str] = []
        self.embeddings: list[list[float]] = []
 
    def add_documents(self, docs: list[str]):
        self.documents.extend(docs)
        new_embeddings = self._embed(docs)
        self.embeddings.extend(new_embeddings)
 
    def _embed(self, texts: list[str]) -> list[list[float]]:
        response = self.client.embeddings.create(model=self.embed_model, input=texts)
        return [item.embedding for item in response.data]
 
    def _cosine_similarity(self, a: list[float], b: list[float]) -> float:
        a_arr = np.array(a)
        b_arr = np.array(b)
        return float(np.dot(a_arr, b_arr) / (np.linalg.norm(a_arr) * np.linalg.norm(b_arr)))
 
    def query(self, question: str, top_k: int = 3) -> str:
        query_emb = self._embed([question])[0]
        scores = [self._cosine_similarity(query_emb, emb) for emb in self.embeddings]
        top_indices = sorted(range(len(scores)), key=lambda i: scores[i], reverse=True)[:top_k]
        context = "\n".join(self.documents[i] for i in top_indices)
 
        response = self.client.chat.completions.create(
            model=self.chat_model,
            messages=[
                {"role": "system", "content": "Answer based on the provided context only."},
                {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"},
            ],
            max_tokens=400,
        )
        return response.choices[0].message.content
 
# Usage
rag = TogetherRAG()
rag.add_documents([
    "Together AI hosts 200+ open-source models via a unified API.",
    "Together AI supports fine-tuning of open-source models.",
    "Together AI pricing is generally lower than OpenAI for equivalent open-source quality.",
])
print(rag.query("What models does Together AI support?"))

LangChain Integration

pip install langchain langchain-together

from langchain_together import Together as TogetherLLM
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
 
llm = TogetherLLM(
    model="meta-llama/Llama-3.3-70B-Instruct-Turbo",
    max_tokens=500,
    temperature=0.7,
)
 
prompt = ChatPromptTemplate.from_template(
    "You are an expert {domain} engineer. Explain {topic} clearly with a code example."
)
 
chain = prompt | llm | StrOutputParser()
 
result = chain.invoke({"domain": "backend", "topic": "database connection pooling"})
print(result)

Fine-tuning on Together AI

Together AI provides managed fine-tuning — upload your data, configure the job, and Together handles the GPU cluster:

from together import Together
 
client = Together()
 
# Step 1: Upload training data (JSONL format)
with open("training_data.jsonl", "w") as f:
    import json
    examples = [
        {
            "messages": [
                {"role": "user", "content": "What is gradient descent?"},
                {"role": "assistant", "content": "Gradient descent is an optimization algorithm..."},
            ]
        },
    ]
    for ex in examples:
        f.write(json.dumps(ex) + "\n")
 
file_response = client.files.upload(file=("training_data.jsonl", open("training_data.jsonl", "rb")))
print(f"File ID: {file_response.id}")
 
# Step 2: Create fine-tuning job
ft_job = client.fine_tuning.create(
    training_file=file_response.id,
    model="meta-llama/Llama-3.1-8B-Instruct-Turbo",
    n_epochs=3,
    learning_rate=1e-5,
    suffix="my-domain-tuned",
)
print(f"Fine-tuning job: {ft_job.id}")

# Step 3: Monitor job progress
import time
 
while True:
    job = client.fine_tuning.retrieve(ft_job.id)
    print(f"Status: {job.status}")
    if job.status in ("completed", "failed"):
        break
    time.sleep(30)
 
# Step 4: Use the fine-tuned model
if job.status == "completed":
    response = client.chat.completions.create(
        model=job.output_name,
        messages=[{"role": "user", "content": "Test question for fine-tuned model"}],
        max_tokens=200,
    )
    print(response.choices[0].message.content)

Batch Processing for Cost Optimization

from together import Together
import asyncio
import aiohttp
 
client = Together()
 
async def async_completion(session: aiohttp.ClientSession, prompt: str, model: str) -> str:
    """Async completion for parallel batch processing."""
    payload = {
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "max_tokens": 200,
    }
    headers = {
        "Authorization": f"Bearer {os.environ['TOGETHER_API_KEY']}",
        "Content-Type": "application/json",
    }
    async with session.post(
        "https://api.together.xyz/v1/chat/completions",
        json=payload,
        headers=headers,
    ) as response:
        data = await response.json()
        return data["choices"][0]["message"]["content"]
 
async def batch_process(prompts: list[str], model: str) -> list[str]:
    async with aiohttp.ClientSession() as session:
        tasks = [async_completion(session, p, model) for p in prompts]
        return await asyncio.gather(*tasks)
 
prompts = [
    "What is Python?",
    "What is JavaScript?",
    "What is Rust?",
    "What is Go?",
]
 
results = asyncio.run(batch_process(prompts, "meta-llama/Llama-3.1-8B-Instruct-Turbo"))
for prompt, result in zip(prompts, results):
    print(f"Q: {prompt}\nA: {result[:100]}...\n")

Together AI vs Alternatives

Feature	Together AI	OpenAI	Groq	Hugging Face
Model variety	200+ open-source	5-10 proprietary	8-10 open-source	1000+ (varies)
Pricing	Very low	High	Medium	Free / Low
Speed	Good	Good	Very fast	Slow (free)
Fine-tuning	Yes	Yes	No	No (Inference API)
Custom deployment	Yes	No	No	Pro plan
OpenAI-compatible	Yes	Native	Yes	Partial

Common Mistakes

Using model IDs without checking current availability — Together AI adds and removes models; always verify the model ID is active at api.together.ai/models before hardcoding it
Not setting stop sequences for Llama models — Llama 3 needs stop=["<|eot_id|>"] to stop at the right place; without it, the model may generate extra turns
Using the Together SDK for embeddings but forgetting the per-character billing — embedding costs are low but accumulate with large document corpora; cache embeddings aggressively
Submitting fine-tuning jobs with poorly formatted JSONL — Together validates the format but silent formatting errors can cause jobs to fail; validate your JSONL with jsonlines library before uploading
Requesting fine-tuning on very large base models — fine-tuning a 70B model requires extended compute time and cost; start with 8B for experiments

Best Practices

Use meta-llama/Llama-3.1-8B-Instruct-Turbo for cost-sensitive pipelines — at $0.18 per million tokens, it is among the cheapest capable models available
Cache embeddings in a vector database (Chroma, Pinecone, Weaviate) rather than regenerating them — embedding costs accumulate quickly for large document stores
Use async HTTP requests for batch processing — Together AI supports high concurrent request rates and async parallelism dramatically reduces pipeline latency
For fine-tuning, always start with 3 epochs and a learning rate between 1e-5 and 2e-5 — lower LR for small datasets, higher for large datasets
Monitor your monthly spend at api.together.ai; set budget alerts to avoid surprises from automated pipelines that scale unexpectedly

Key Takeaways

Together AI hosts 200+ open-source models (LLaMA, Mistral, Mixtral, Qwen, DeepSeek, Gemma) behind a unified OpenAI-compatible API at among the lowest per-token pricing in the industry
The API is fully OpenAI SDK-compatible — switching from OpenAI requires only changing base_url and api_key in the client initialization
Together AI offers managed fine-tuning: upload JSONL data, configure the job, and the platform handles GPU provisioning and training
Embedding models (BAAI/bge-large-en-v1.5, etc.) are available for $0.02 per million tokens, enabling cost-efficient RAG pipelines
Async HTTP parallelism with aiohttp is the most effective way to batch-process large document collections at minimum latency
For Llama 3 models, always set stop=["<|eot_id|>"] to prevent generation of extra conversation turns beyond the first response
Together AI supports custom model deployment — fine-tuned adapters or custom weights can be hosted for private dedicated inference
Start fine-tuning experiments on 8B base models before scaling to 70B — the cost and time difference is 5-10x and 8B is sufficient for most domain adaptation tasks

Introduction#

Why This Matters#

Getting Started#

Popular Models Available on Together AI#

Chat Completions (OpenAI-compatible)#

Streaming Responses#

Generating Embeddings#

Building a RAG Pipeline#

LangChain Integration#

Fine-tuning on Together AI#

Batch Processing for Cost Optimization#

Together AI vs Alternatives#

Common Mistakes#

Best Practices#

Key Takeaways#

Related reading