Published on

The Complete Backend Checklist for Shipping an AI Product to Production

Authors
  • Name
    Twitter

Introduction

Shipping an AI product is more complex than shipping traditional software. LLM APIs add dependencies, costs, and risks. This checklist ensures nothing is forgotten. Use it before launch day.

Authentication and Multi-Tenancy

  • </> JWT tokens with tenant ID claim
  • </> Tenant ID required in all API requests (header or path)
  • </> Row-level security in database (filter all queries by tenantId)
  • </> Vector DB queries filtered by tenantId
  • </> API responses never include other tenants' data
  • </> Data leakage test in CI/CD (query as tenant B, verify no tenant A data)
  • </> User-level isolation (users can''t see other users'' data within same tenant)
  • </> Subscription model defines what each tier can access

LLM Provider Selection and Fallback

  • </> Primary provider selected (OpenAI, Anthropic, etc.)
  • </> Fallback provider configured (different company)
  • </> Fallback logic tested (primary fails, routes to fallback)
  • </> Costs tracked per provider
  • </> Rate limits known for both providers
  • </> API keys rotated monthly (security)
  • </> Environment variables used, not hardcoded
  • </> Provider SLA documented and monitored
  • </> Cost per request calculated inline

Rate Limiting and Cost Controls

  • </> Token-based rate limiting (not request-based)
  • </> Per-user rate limits by tier (free: 10K/day, pro: 100K/day)
  • </> Per-tenant rate limits
  • </> Hard cost limits that disable feature when exceeded
  • </> Soft cost warnings (email at 80% of limit)
  • </> Rate limit response includes Retry-After header
  • </> Rate limit errors are HTTP 429
  • </> Cost circuit breakers for expensive operations
  • </> Backoff strategy: exponential backoff on 429s

Observability (Traces, Metrics, Logs)

Metrics:

  • </> llm.tokens_input histogram (by model, user tier)
  • </> llm.tokens_output histogram
  • </> llm.latency_ms histogram (by model)
  • </> llm.cost_usd histogram (by model, user tier)
  • </> llm.errors_total counter (by error code)
  • </> cache.hit_rate gauge
  • </> vector.search_latency_ms histogram

Logs:

  • </> Every LLM call logged with model, tokens, latency, cost
  • </> Every error logged with full context (user, tenant, operation)
  • </> Structured JSON logs (not text)
  • </> Request ID in every log line

Traces:

  • </> Distributed tracing for LLM calls
  • </> Span for each LLM call: prompt, model, tokens, latency
  • </> Span for each database query
  • </> Span for each vector search
  • </> Trace context propagated across services

Dashboards:

  • </> Real-time cost per hour (alert if spiking)
  • </> LLM latency percentiles (p50, p95, p99)
  • </> Error rate by error code
  • </> Token usage per user tier
  • </> Cost per feature

Data Privacy and PII Handling

  • </> PII fields identified (email, phone, SSN, etc.)
  • </> PII never sent to LLM unless necessary
  • </> PII masked in logs (email@example.com, not user''s real email)
  • </> Encryption at rest for sensitive data (database, S3)
  • </> Encryption in transit (TLS everywhere)
  • </> Data retention policy documented (keep X days, then delete)
  • </> Delete API implemented (user can request data deletion)
  • </> GDPR compliance checklist (Article 22 for automated decisions)
  • </> Right to be forgotten implemented
  • </> Privacy policy clearly explains what''s stored and sent to LLMs

Content Moderation

  • </> Input moderation: block harmful prompts
  • </> Output moderation: block harmful LLM responses
  • </> Moderation API integrated (OpenAI Moderation, etc.)
  • </> Manual review queue for edge cases
  • </> Policy: what content is blocked? (explicit, illegal, etc.)
  • </> User feedback loop (users report problematic outputs)
  • </> Blocked content logged for audit
  • </> Escalation process for high-risk cases

Conversation Storage and Retention

  • </> Conversations stored in database with user/tenant tags
  • </> Each message tagged with timestamp, model version
  • </> Conversations searchable by user
  • </> Retention policy: delete conversations > 90 days
  • </> Export API: users can export their conversations
  • </> Conversation history never shared across tenants
  • </> User can delete specific conversations
  • </> Admin audit log for compliance

Prompt Versioning

  • </> System prompts stored in database, not hardcoded
  • </> Each prompt has version number
  • </> Prompt changes tracked in changelog
  • </> Rollback to previous prompt version in seconds
  • </> Conversation tracks which prompt version was used
  • </> Test suite for prompts (expected behavior for test cases)
  • </> A/B tests for new prompts before rollout
  • </> Prompt injection attack test cases

Evaluation and Regression Testing

  • </> Evaluation dataset of < 50 test cases per feature
  • </> Automated tests run before every model update
  • </> Test cases check for output quality (not exact match)
  • </> Regression test: new model vs old model on test set
  • </> Pass rate threshold: > 80% for rollout
  • </> Manual review of failures: why did this test case fail?
  • </> Edge case coverage (empty input, very long input, special characters)
  • </> Cost per test case measured

Load Testing LLM Integrations

  • </> Load test: 100 concurrent users making LLM requests
  • </> Load test: 1000 concurrent users
  • </> Test with realistic prompt distribution (mix short & long)
  • </> Test provider rate limits (what happens at 10K req/min?)
  • </> Test fallback switching (primary provider throttles, fallback engages)
  • </> Measure latency under load (p95 latency acceptable?)
  • </> Measure costs under load
  • </> Test queue behavior (requests queued, not dropped)

Incident Response Playbook

Create a document with:

Cost Spike Incident:

  • </> Trigger: hourly cost > $X
  • </> Action: disable expensive features (feature flag)
  • </> Notify: on-call engineer
  • </> Investigate: which user/tenant caused spike?
  • </> Recover: manually re-enable after fix

Quality Regression Incident:

  • </> Trigger: error rate > 5%, or satisfaction score < 3/5
  • </> Action: rollback to previous model version
  • </> Notify: product and engineering leads
  • </> Investigate: what changed?
  • </> RCA: root cause analysis doc

Data Leak Incident:

  • </> Trigger: unauthorized access detected
  • </> Action: disable service immediately
  • </> Notify: security, legal, compliance
  • </> Contain: revoke leaked credentials
  • </> Investigate: scope of leak

Provider Outage Incident:

  • </> Trigger: primary provider down
  • </> Action: route to fallback provider automatically
  • </> Notify: users of temporary degradation
  • </> RCA: what happened with provider?

Compliance Documentation

  • </> Security policy document
  • </> Privacy policy
  • </> Data processing agreement (DPA) for EU customers
  • </> GDPR compliance documentation (Article 22, data rights)
  • </> SOC2 Type II report (if claiming SOC2)
  • </> HIPAA compliance checklist (if handling health data)
  • </> Model card: model training data, limitations, biases
  • </> Terms of service mentioning AI usage
  • </> Disclosure: data sent to third-party LLM APIs

Launch Day Runbook

Pre-Launch Checks (24 hours before):

  • </> All checklist items above completed
  • </> Load test passed
  • </> On-call rotation confirmed
  • </> Incident response runbook reviewed
  • </> Rollback plan tested
  • </> Monitoring dashboards created
  • </> Feature flags created and tested
  • </> Cost alerts configured

Launch Day (morning of):

  • </> Enable feature for 1% of users
  • </> Monitor dashboards for 1 hour
    • Cost OK?
    • Latency OK?
    • Error rate OK?
    • User satisfaction feedback OK?
  • </> If issues: rollback immediately with feature flag
  • </> If OK: increase to 5% of users
  • </> Monitor for 2 hours
  • </> If OK: increase to 25%
  • </> Monitor for 4 hours
  • </> If OK: go to 100%

Post-Launch (first week):

  • </> Monitor dashboards every 4 hours
  • </> Daily standup reviewing metrics
  • </> Customer support alert: watch for user complaints
  • </> Weekly: analyze usage patterns, optimize
  • </> Weekly: review cost trends, adjust rate limits if needed

Post-Launch Metrics to Track

  • </> Daily active users
  • </> Cost per user
  • </> Feature adoption rate
  • </> User satisfaction (NPS, ratings)
  • </> Error rate
  • </> Latency (p95)
  • </> LLM token usage trends
  • </> Cache hit rate
  • </> Provider reliability (uptime %)
  • </> Support tickets related to AI

Checklist (Summary)

Architecture:

  • </> Multi-tenancy with row-level security
  • </> LLM provider + fallback configured
  • </> Token-based rate limiting
  • </> Cost controls and kill switches
  • </> Async job queues for expensive ops
  • </> Comprehensive observability
  • </> Caching for expensive operations
  • </> Idempotent endpoints (Idempotency-Key header)

Security & Compliance:

  • </> PII handling policy
  • </> Encryption at rest and in transit
  • </> Data retention policy
  • </> GDPR/HIPAA compliance (if required)
  • </> Content moderation
  • </> Incident response runbook
  • </> Security policy and privacy policy

Quality:

  • </> Evaluation test cases
  • </> Regression tests
  • </> Load testing
  • </> Feature flags for gradual rollout
  • </> A/B testing for prompts

Ops:

  • </> Monitoring dashboards
  • </> On-call rotation
  • </> Rollback procedure tested
  • </> Launch day runbook
  • </> Post-launch metrics tracked

Conclusion

This checklist is long. That's intentional. AI products are complex. Missing one item can cause outages, cost spikes, compliance violations, or data leaks. Work through it methodically. Check boxes. Test each item. Only then launch.

The cost of launching unprepared is higher than the cost of launching prepared.