Intelligent LLM Model Routing — Sending the Right Query to the Right Model

Introduction

Not every query needs the most expensive model. Routing a "what is Python?" question to GPT-4o costs 30x more than routing it to a smaller model that answers just as well. Intelligent model routing classifies query complexity, matches it to a cost-tier, and escalates only when necessary — cutting LLM spend by 40–70% without degrading output quality.

Why Model Routing Matters

LLM pricing scales nonlinearly with capability. A GPT-4o call for a simple factual question costs the same as a complex multi-step reasoning task. At scale, the waste compounds fast:

10,000 requests/day × $0.03/request (GPT-4o) =$ 9,000/month
With routing: 70% go to $0.001 model, 30% to$ 0.03 model = $3,030/month
Savings: $5,970/month with no user-visible quality change

The key insight is that most production traffic is simple — greetings, FAQs, short summaries — and only a minority requires deep reasoning.

Query Complexity Classification

The first step is classifying complexity before routing. Use heuristics that run in microseconds:

import re
 
def classify_complexity(query: str) -> tuple[str, int]:
    """Returns (complexity, score) where complexity is 'simple'|'medium'|'complex'."""
    score = 0
    word_count = len(query.split())
 
    # Length signals complexity
    if word_count < 10:
        score += 10
    elif word_count < 50:
        score += 30
    elif word_count < 150:
        score += 55
    else:
        score += 80
 
    # Code blocks are expensive to reason about
    code_blocks = len(re.findall(r"```", query)) // 2
    score += code_blocks * 25
 
    # Reasoning keywords indicate multi-step work
    reasoning_pattern = re.compile(
        r"\b(explain|why|debug|refactor|optimize|analyze|compare|design)\b", re.I
    )
    if reasoning_pattern.search(query):
        score += 20
 
    # Multiple questions compound complexity
    question_count = query.count("?")
    score += question_count * 10
 
    # Math/formula signals hard reasoning
    if re.search(r"(integral|derivative|matrix|\beigen\b|proof|theorem)", query, re.I):
        score += 30
 
    if score < 30:
        return "simple", score
    elif score < 65:
        return "medium", score
    else:
        return "complex", score

Model Tier Registry

Define tiers with cost, latency, and capability metadata:

from dataclasses import dataclass
 
@dataclass
class ModelTier:
    name: str
    models: list[str]
    cost_per_1k_input: float   # USD
    cost_per_1k_output: float  # USD
    p95_latency_ms: int
    capability_score: int      # 1–10
 
TIERS = {
    "nano": ModelTier(
        name="nano",
        models=["claude-3-haiku-20240307"],
        cost_per_1k_input=0.00025,
        cost_per_1k_output=0.00125,
        p95_latency_ms=400,
        capability_score=5,
    ),
    "standard": ModelTier(
        name="standard",
        models=["claude-3-5-sonnet-20241022"],
        cost_per_1k_input=0.003,
        cost_per_1k_output=0.015,
        p95_latency_ms=1500,
        capability_score=8,
    ),
    "premium": ModelTier(
        name="premium",
        models=["claude-opus-4-5"],
        cost_per_1k_input=0.015,
        cost_per_1k_output=0.075,
        p95_latency_ms=3000,
        capability_score=10,
    ),
}
 
COMPLEXITY_TO_TIER = {
    "simple":  "nano",
    "medium":  "standard",
    "complex": "premium",
}
 
def select_model(query: str) -> tuple[str, str]:
    """Returns (model_name, tier_name)."""
    complexity, _ = classify_complexity(query)
    tier_name = COMPLEXITY_TO_TIER[complexity]
    tier = TIERS[tier_name]
    return tier.models[0], tier_name

Intent-Based Routing

Beyond complexity, the intent of a query determines which model excels. Code generation benefits from models trained heavily on code; creative writing benefits from different strengths:

import re
from typing import Optional
 
INTENT_PATTERNS = {
    "coding": re.compile(
        r"\b(code|function|class|api|implement|debug|refactor|script)\b", re.I
    ),
    "creative": re.compile(
        r"\b(write|story|poem|essay|blog|creative|imagine)\b", re.I
    ),
    "analysis": re.compile(
        r"\b(analyze|research|explain|summarize|compare|trend)\b", re.I
    ),
    "support": re.compile(
        r"\b(help|issue|problem|error|broken|how to|how do)\b", re.I
    ),
}
 
# Map intent to the best-fit model for that task
INTENT_MODEL_MAP = {
    "coding":   "claude-3-5-sonnet-20241022",  # Strong at code
    "creative": "claude-opus-4-5",             # Strong at writing
    "analysis": "claude-3-5-sonnet-20241022",  # Strong at reasoning
    "support":  "claude-3-haiku-20240307",     # Fast, follows instructions
    "general":  "claude-3-haiku-20240307",     # Default
}
 
def detect_intent(query: str) -> str:
    scores = {intent: 0 for intent in INTENT_PATTERNS}
    for intent, pattern in INTENT_PATTERNS.items():
        scores[intent] = len(pattern.findall(query))
    best = max(scores, key=lambda k: scores[k])
    return best if scores[best] > 0 else "general"

Confidence-Based Escalation

Try the cheap model first. If the response signals uncertainty, escalate to a more capable one:

import re
import anthropic
 
UNCERTAINTY_PHRASES = re.compile(
    r"\b(i'm not sure|i don't know|i cannot|unclear|uncertain|might be|possibly)\b",
    re.I
)
 
def estimate_confidence(response_text: str) -> float:
    base = 0.75
    if UNCERTAINTY_PHRASES.search(response_text):
        base -= 0.25
    if len(response_text) < 50:
        base -= 0.20
    if re.search(r"\b(specifically|exactly|definitely)\b", response_text, re.I):
        base += 0.15
    return max(0.0, min(1.0, base))
 
def respond_with_escalation(
    query: str,
    escalation_threshold: float = 0.6,
) -> dict:
    client = anthropic.Anthropic()
    escalation_chain = [
        "claude-3-haiku-20240307",
        "claude-3-5-sonnet-20241022",
        "claude-opus-4-5",
    ]
    for model in escalation_chain:
        resp = client.messages.create(
            model=model,
            max_tokens=1024,
            messages=[{"role": "user", "content": query}]
        )
        text = resp.content[0].text
        confidence = estimate_confidence(text)
        if confidence >= escalation_threshold:
            return {"response": text, "model": model, "confidence": confidence}
    # Fall through with best available
    return {"response": text, "model": model, "confidence": confidence}

Latency-SLA-Aware Routing

For endpoints with strict latency budgets, only consider models whose p95 latency fits the SLA:

def select_model_for_sla(max_latency_ms: int) -> Optional[str]:
    """Return the most capable model that fits within the latency SLA."""
    candidates = [
        (tier.capability_score, tier.models[0])
        for tier in TIERS.values()
        if tier.p95_latency_ms <= max_latency_ms
    ]
    if not candidates:
        return None
    # Pick highest capability score within SLA
    candidates.sort(reverse=True)
    return candidates[0][1]
 
# Example: interactive chat with 1s p95 requirement
model = select_model_for_sla(max_latency_ms=1000)
# Returns "claude-3-haiku-20240307" (400ms p95), skips Sonnet (1500ms)

Routing Metrics and Observability

Track routing decisions to measure effectiveness and catch regressions:

from collections import defaultdict
from datetime import datetime
 
class RoutingMetricsCollector:
    def __init__(self):
        self._records = []
 
    def record(self, model: str, tier: str, cost_usd: float, latency_ms: int):
        self._records.append({
            "model": model,
            "tier": tier,
            "cost_usd": cost_usd,
            "latency_ms": latency_ms,
            "ts": datetime.utcnow().isoformat(),
        })
 
    def summary(self) -> dict:
        if not self._records:
            return {}
        total_cost = sum(r["cost_usd"] for r in self._records)
        by_tier = defaultdict(int)
        for r in self._records:
            by_tier[r["tier"]] += 1
        # Cost if all traffic had gone to premium
        premium_cost_per_req = 0.045  # rough estimate
        premium_baseline = len(self._records) * premium_cost_per_req
        return {
            "total_requests": len(self._records),
            "total_cost_usd": round(total_cost, 4),
            "cost_saved_vs_premium": round(premium_baseline - total_cost, 4),
            "tier_distribution": dict(by_tier),
        }

A/B Testing Routing Decisions

Before committing to a new routing rule, validate it with live traffic:

import random
 
class RoutingABTest:
    def __init__(self, control_fn, variant_fn, variant_percent: float = 0.1):
        self.control_fn = control_fn
        self.variant_fn = variant_fn
        self.variant_percent = variant_percent
        self.results = {"control": [], "variant": []}
 
    def route(self, query: str) -> tuple[str, str]:
        """Returns (model, variant_label)."""
        if random.random() < self.variant_percent:
            model = self.variant_fn(query)
            return model, "variant"
        else:
            model = self.control_fn(query)
            return model, "control"
 
    def record_outcome(self, variant: str, quality_score: float, cost: float):
        self.results[variant].append({"quality": quality_score, "cost": cost})
 
    def should_promote_variant(self, min_samples: int = 500) -> bool:
        ctrl = self.results["control"]
        var  = self.results["variant"]
        if len(ctrl) < min_samples or len(var) < min_samples:
            return False
        ctrl_avg_quality = sum(r["quality"] for r in ctrl) / len(ctrl)
        var_avg_quality  = sum(r["quality"] for r in var)  / len(var)
        ctrl_avg_cost    = sum(r["cost"]    for r in ctrl) / len(ctrl)
        var_avg_cost     = sum(r["cost"]    for r in var)  / len(var)
        # Promote if quality within 5% and cost lower
        return var_avg_quality >= ctrl_avg_quality * 0.95 and var_avg_cost < ctrl_avg_cost

Key Takeaways

Heuristic classification (word count, code blocks, reasoning keywords) routes 80%+ of traffic correctly with zero latency overhead.
Three tiers — nano/standard/premium — are sufficient for most production systems; more tiers add complexity without proportional benefit.
Confidence-based escalation catches cases where the cheap model is genuinely inadequate, preventing silent quality degradation.
Latency-SLA-aware routing prevents interactive endpoints from accidentally routing to slow premium models.
A/B test routing rule changes with at least 500 samples per variant before full rollout.
Routing metrics should track cost saved versus a premium-only baseline to quantify the ROI of the routing layer.
Most production traffic is 60–80% simple queries — routing these to nano-tier alone covers the majority of cost savings.
Monitor escalation rate weekly; a rising rate signals the classifier is mislabeling queries.

Introduction#

Why Model Routing Matters#

Query Complexity Classification#

Model Tier Registry#

Intent-Based Routing#

Confidence-Based Escalation#

Latency-SLA-Aware Routing#

Routing Metrics and Observability#

A/B Testing Routing Decisions#

Key Takeaways#