AI Rate Limiting and Cost Quotas — Protecting Your LLM Budget From Runaway Usage

Introduction

LLM API costs scale with usage in a way that no other infrastructure cost does — a single runaway script can generate a $10,000 bill in hours. Every production AI system needs multi-layer rate limiting: per-user request limits, per-tenant token quotas, global budget caps, and real-time alerts. This post covers the complete architecture for protecting your LLM budget without degrading the user experience.

Why This Matters

Unlike traditional compute costs that scale linearly, LLM costs have a dual dimension: request count and token count. A single user generating 100,000-token responses can consume more budget than 1,000 users making short requests. Rate limiting on requests alone is insufficient — you must also limit on token consumption.

Teams that deploy LLM features without quota systems routinely discover $50,000+ surprise bills at month end. The fix always costs less than the incident.

Token-Based Rate Limiter

Implement a sliding-window rate limiter that tracks both requests and tokens:

interface TokenBucket {
  userId: string;
  requestCount: number;
  tokenCount: number;
  windowStart: number;
  lastUpdated: number;
}
 
interface RateLimitConfig {
  maxRequestsPerMinute: number;
  maxTokensPerMinute: number;
  maxTokensPerDay: number;
}
 
class TokenRateLimiter {
  private buckets = new Map<string, TokenBucket>();
  private dailyUsage = new Map<string, number>();
 
  constructor(private config: RateLimitConfig) {}
 
  check(userId: string): { allowed: boolean; reason?: string; retryAfterMs?: number } {
    const now = Date.now();
    const bucket = this.getOrCreateBucket(userId, now);
 
    // Reset minute window if expired
    if (now - bucket.windowStart > 60_000) {
      bucket.requestCount = 0;
      bucket.tokenCount = 0;
      bucket.windowStart = now;
    }
 
    if (bucket.requestCount >= this.config.maxRequestsPerMinute) {
      const retryAfterMs = 60_000 - (now - bucket.windowStart);
      return { allowed: false, reason: 'Request rate limit exceeded', retryAfterMs };
    }
 
    if (bucket.tokenCount >= this.config.maxTokensPerMinute) {
      const retryAfterMs = 60_000 - (now - bucket.windowStart);
      return { allowed: false, reason: 'Token rate limit exceeded', retryAfterMs };
    }
 
    const dailyTokens = this.dailyUsage.get(userId) ?? 0;
    if (dailyTokens >= this.config.maxTokensPerDay) {
      return { allowed: false, reason: 'Daily token quota exhausted' };
    }
 
    return { allowed: true };
  }
 
  record(userId: string, tokensUsed: number): void {
    const bucket = this.buckets.get(userId);
    if (bucket) {
      bucket.requestCount++;
      bucket.tokenCount += tokensUsed;
    }
 
    const daily = this.dailyUsage.get(userId) ?? 0;
    this.dailyUsage.set(userId, daily + tokensUsed);
  }
 
  private getOrCreateBucket(userId: string, now: number): TokenBucket {
    if (!this.buckets.has(userId)) {
      this.buckets.set(userId, {
        userId,
        requestCount: 0,
        tokenCount: 0,
        windowStart: now,
        lastUpdated: now,
      });
    }
    return this.buckets.get(userId)!;
  }
 
  resetDailyUsage(): void {
    this.dailyUsage.clear();
  }
}

Cost Budget Manager

Track spending per user and tenant against configurable budgets:

interface BudgetConfig {
  userId: string;
  monthlyBudgetUsd: number;
  alertThresholds: number[]; // e.g. [0.5, 0.8, 0.95]
}
 
interface CostRecord {
  userId: string;
  month: string; // YYYY-MM
  spentUsd: number;
  totalInputTokens: number;
  totalOutputTokens: number;
}
 
const MODEL_PRICING_USD_PER_TOKEN: Record<string, { input: number; output: number }> = {
  'claude-3-5-sonnet-20241022': { input: 0.000003, output: 0.000015 },
  'claude-3-5-haiku-20241022': { input: 0.0000008, output: 0.000004 },
  'claude-opus-4-5': { input: 0.000015, output: 0.000075 },
};
 
class BudgetManager {
  private records = new Map<string, CostRecord>();
  private budgets = new Map<string, BudgetConfig>();
  private alertsSent = new Map<string, Set<number>>();
 
  configure(config: BudgetConfig): void {
    this.budgets.set(config.userId, config);
  }
 
  recordUsage(
    userId: string,
    model: string,
    inputTokens: number,
    outputTokens: number,
    onAlert?: (userId: string, threshold: number, spentUsd: number) => void,
  ): void {
    const pricing = MODEL_PRICING_USD_PER_TOKEN[model];
    if (!pricing) throw new Error(`Unknown model pricing: ${model}`);
 
    const costUsd = inputTokens * pricing.input + outputTokens * pricing.output;
    const month = new Date().toISOString().slice(0, 7);
    const key = `${userId}:${month}`;
 
    const record = this.records.get(key) ?? {
      userId,
      month,
      spentUsd: 0,
      totalInputTokens: 0,
      totalOutputTokens: 0,
    };
 
    record.spentUsd += costUsd;
    record.totalInputTokens += inputTokens;
    record.totalOutputTokens += outputTokens;
    this.records.set(key, record);
 
    // Check budget thresholds
    const budget = this.budgets.get(userId);
    if (budget && onAlert) {
      const ratio = record.spentUsd / budget.monthlyBudgetUsd;
      const sentSet = this.alertsSent.get(userId) ?? new Set();
 
      for (const threshold of budget.alertThresholds) {
        if (ratio >= threshold && !sentSet.has(threshold)) {
          sentSet.add(threshold);
          this.alertsSent.set(userId, sentSet);
          onAlert(userId, threshold, record.spentUsd);
        }
      }
    }
  }
 
  checkBudget(userId: string): { withinBudget: boolean; remainingUsd: number; percentUsed: number } {
    const budget = this.budgets.get(userId);
    if (!budget) return { withinBudget: true, remainingUsd: Infinity, percentUsed: 0 };
 
    const month = new Date().toISOString().slice(0, 7);
    const record = this.records.get(`${userId}:${month}`);
    const spentUsd = record?.spentUsd ?? 0;
 
    return {
      withinBudget: spentUsd < budget.monthlyBudgetUsd,
      remainingUsd: Math.max(0, budget.monthlyBudgetUsd - spentUsd),
      percentUsed: (spentUsd / budget.monthlyBudgetUsd) * 100,
    };
  }
}

Middleware Integration

Wire rate limiting and budget checking into an Express middleware:

import type { Request, Response, NextFunction } from 'express';
import Anthropic from '@anthropic-ai/sdk';
 
const client = new Anthropic();
 
function createAIRateLimitMiddleware(
  rateLimiter: TokenRateLimiter,
  budgetManager: BudgetManager,
) {
  return async (req: Request, res: Response, next: NextFunction) => {
    const userId = req.headers['x-user-id'] as string;
    if (!userId) {
      res.status(401).json({ error: 'Missing user ID' });
      return;
    }
 
    // Check request/token rate limit
    const rateCheck = rateLimiter.check(userId);
    if (!rateCheck.allowed) {
      res.setHeader('Retry-After', Math.ceil((rateCheck.retryAfterMs ?? 60_000) / 1000));
      res.status(429).json({ error: rateCheck.reason });
      return;
    }
 
    // Check monthly budget
    const budgetCheck = budgetManager.checkBudget(userId);
    if (!budgetCheck.withinBudget) {
      res.status(402).json({ error: 'Monthly AI budget exhausted', remainingUsd: 0 });
      return;
    }
 
    next();
  };
}
 
// Usage in route handler
async function callLLM(userId: string, message: string, model = 'claude-3-5-haiku-20241022') {
  const response = await client.messages.create({
    model,
    max_tokens: 1024,
    messages: [{ role: 'user', content: message }],
  });
 
  const usage = response.usage;
  rateLimiter.record(userId, usage.input_tokens + usage.output_tokens);
  budgetManager.recordUsage(
    userId,
    model,
    usage.input_tokens,
    usage.output_tokens,
    (uid, threshold, spent) => {
      console.warn(`Budget alert: user ${uid} reached ${threshold * 100}% ($${spent.toFixed(2)})`);
    },
  );
 
  return response;
}

Graceful Degradation

When limits are hit, degrade gracefully instead of returning errors:

type DegradationStrategy = 'error' | 'cached_response' | 'smaller_model' | 'queue';
 
async function callWithDegradation(
  userId: string,
  message: string,
  strategy: DegradationStrategy,
  cache: Map<string, string>,
): Promise<{ response: string; degraded: boolean }> {
  const rateCheck = rateLimiter.check(userId);
 
  if (rateCheck.allowed) {
    const response = await callLLM(userId, message);
    const text = response.content[0].type === 'text' ? response.content[0].text : '';
    cache.set(message, text);
    return { response: text, degraded: false };
  }
 
  switch (strategy) {
    case 'cached_response': {
      const cached = cache.get(message);
      if (cached) return { response: cached, degraded: true };
      return { response: 'Service temporarily busy. Please try again shortly.', degraded: true };
    }
 
    case 'smaller_model': {
      // Fall back to a cheaper, faster model
      const response = await client.messages.create({
        model: 'claude-3-5-haiku-20241022',
        max_tokens: 512,
        messages: [{ role: 'user', content: message }],
      });
      const text = response.content[0].type === 'text' ? response.content[0].text : '';
      return { response: text, degraded: true };
    }
 
    case 'error':
    default:
      throw new Error('Rate limit exceeded');
  }
}

Common Mistakes

Rate limiting only on request count: Token count is the cost driver, not request count. A single 100k-token request costs more than 100 short requests.
Resetting daily quotas at midnight UTC only: Users in different time zones exhaust their quota at inconvenient times. Use per-user rolling 24-hour windows instead.
No budget alerts until 100% used: By the time the budget is exhausted, the damage is done. Alert at 50%, 80%, and 95% to give time to react.
Hard-coding model pricing: LLM providers change pricing regularly. Store pricing in configuration, not in code, so you can update it without a deploy.
Applying rate limits in-process only: In-process maps don't work across multiple server instances. Use Redis for distributed rate limiting.

Best Practices

Use Redis with Lua scripts for atomic sliding-window rate limiting across multiple server instances.
Track both requests-per-minute and tokens-per-minute as independent limits — they catch different abuse patterns.
Implement a graceful degradation path (smaller model, cached response) so rate-limited users get something useful instead of a 429 error.
Alert at 50%, 80%, and 95% of monthly budget so engineers have time to investigate before the limit is hit.
Log every rate-limit event with user ID, time, and limit type for abuse investigation and capacity planning.

Key Takeaways

LLM cost is driven by token volume, not request count — rate limiting on requests alone leaves you exposed to expensive long-context abuse.
A sliding-window algorithm (per-minute, per-day) is more effective than fixed-window because it prevents burst abuse at window boundaries.
Budget thresholds at 50%, 80%, and 95% give engineering teams time to act before the monthly limit is exhausted.
Graceful degradation to a smaller model (e.g. Haiku instead of Sonnet) preserves user experience while reducing cost under pressure.
Distributed rate limiting requires Redis or a shared store — in-process maps only work for single-instance deployments.
Model pricing must be stored in configuration, not hard-coded, because providers change prices every few months.
A right-sized monthly budget per user tier (free, pro, enterprise) allows you to offer AI features profitably without per-user margin risk.
Log all rate-limit and budget-alert events to a centralized system so you can identify abuse patterns and improve the quota model over time.

Introduction#

Why This Matters#

Token-Based Rate Limiter#

Cost Budget Manager#

Middleware Integration#

Graceful Degradation#

Common Mistakes#

Best Practices#

Key Takeaways#

Related reading