AI Output Validation — Schema Checking, Business Rules, and Safety Nets

Introduction

LLMs do not always return what you ask for. They hallucinate fields, omit required keys, return malformed JSON, and occasionally produce unsafe content. In production AI systems, every model response must pass through a validation layer before it reaches users or downstream services. This post covers the full stack of AI output validation: schema enforcement, content safety, retry with feedback, and graceful degradation.

Why This Matters

A single malformed LLM response propagated to a frontend can crash a React app. An unvalidated SQL query generated by an LLM can destroy a database. Content safety failures in consumer products trigger legal exposure. Validation is not optional — it is the contract between your AI subsystem and the rest of your application.

Production teams that skip output validation spend 3-5x more engineering time firefighting incidents than teams that build validation in from day one.

JSON Schema Validation with Retry

Enforce a schema and retry with error feedback on failure:

import Anthropic from '@anthropic-ai/sdk';
import Ajv, { JSONSchemaType } from 'ajv';
 
const ajv = new Ajv();
const client = new Anthropic();
 
interface ProductRecommendation {
  productId: string;
  name: string;
  price: number;
  confidence: number;
  reasoning: string;
}
 
const schema: JSONSchemaType<ProductRecommendation> = {
  type: 'object',
  properties: {
    productId: { type: 'string' },
    name: { type: 'string' },
    price: { type: 'number', minimum: 0 },
    confidence: { type: 'number', minimum: 0, maximum: 1 },
    reasoning: { type: 'string', minLength: 10 },
  },
  required: ['productId', 'name', 'price', 'confidence', 'reasoning'],
  additionalProperties: false,
};
 
const validate = ajv.compile(schema);
 
async function getValidatedRecommendation(query: string): Promise<ProductRecommendation> {
  const messages: Anthropic.MessageParam[] = [{ role: 'user', content: query }];
 
  for (let attempt = 0; attempt < 3; attempt++) {
    const response = await client.messages.create({
      model: 'claude-3-5-sonnet-20241022',
      max_tokens: 512,
      system: 'Return only a valid JSON object with fields: productId, name, price, confidence, reasoning. No markdown.',
      messages,
    });
 
    const text = response.content[0].type === 'text' ? response.content[0].text.trim() : '';
    messages.push({ role: 'assistant', content: text });
 
    try {
      const parsed = JSON.parse(text.replace(/^```json\n?/, '').replace(/\n?```$/, ''));
      if (validate(parsed)) return parsed as ProductRecommendation;
      const errors = validate.errors?.map((e) => e.message).join('; ') ?? 'unknown';
      messages.push({ role: 'user', content: `Validation failed: ${errors}. Return corrected JSON.` });
    } catch {
      messages.push({ role: 'user', content: 'Invalid JSON. Return only a raw JSON object with no markdown fences.' });
    }
  }
 
  throw new Error('LLM failed to return valid structured output after 3 attempts');
}

Structured Output via Tool Use

Using the Anthropic tool-use API forces valid structured output at the protocol level:

import Anthropic from '@anthropic-ai/sdk';
 
const client = new Anthropic();
 
const extractionTool: Anthropic.Tool = {
  name: 'extract_order_details',
  description: 'Extract structured order details from a user message',
  input_schema: {
    type: 'object' as const,
    properties: {
      items: {
        type: 'array',
        items: {
          type: 'object',
          properties: {
            name: { type: 'string' },
            quantity: { type: 'integer', minimum: 1 },
            unitPrice: { type: 'number', minimum: 0 },
          },
          required: ['name', 'quantity', 'unitPrice'],
        },
      },
      deliveryAddress: { type: 'string' },
      urgency: { type: 'string', enum: ['standard', 'express', 'overnight'] },
    },
    required: ['items', 'deliveryAddress', 'urgency'],
  },
};
 
async function extractOrder(message: string) {
  const response = await client.messages.create({
    model: 'claude-3-5-sonnet-20241022',
    max_tokens: 1024,
    tools: [extractionTool],
    tool_choice: { type: 'tool', name: 'extract_order_details' },
    messages: [{ role: 'user', content: message }],
  });
 
  if (response.stop_reason === 'max_tokens') {
    throw new Error('Response truncated — increase max_tokens or simplify request');
  }
 
  const toolUse = response.content.find((c) => c.type === 'tool_use');
  if (!toolUse || toolUse.type !== 'tool_use') {
    throw new Error('Model did not call the extraction tool');
  }
 
  return toolUse.input;
}

Business Rule Validation

Schema validation alone is not enough — enforce domain rules too:

interface InvoiceOutput {
  invoiceId: string;
  amount: number;
  currency: string;
  dueDate: string;
  lineItems: Array<{ description: string; amount: number }>;
}
 
interface ValidationResult {
  valid: boolean;
  errors: string[];
}
 
function validateInvoice(invoice: InvoiceOutput): ValidationResult {
  const errors: string[] = [];
 
  // Business rule: line items must sum to total amount
  const lineItemTotal = invoice.lineItems.reduce((sum, item) => sum + item.amount, 0);
  const TOLERANCE = 0.01;
  if (Math.abs(lineItemTotal - invoice.amount) > TOLERANCE) {
    errors.push(
      `Line items sum (${lineItemTotal}) does not match total amount (${invoice.amount})`
    );
  }
 
  // Business rule: due date must be in the future
  if (new Date(invoice.dueDate) <= new Date()) {
    errors.push('Due date must be in the future');
  }
 
  // Business rule: supported currencies only
  const SUPPORTED = ['USD', 'EUR', 'GBP', 'JPY'];
  if (!SUPPORTED.includes(invoice.currency)) {
    errors.push(`Unsupported currency: ${invoice.currency}`);
  }
 
  // Business rule: invoice ID format
  if (!/^INV-\d{6}$/.test(invoice.invoiceId)) {
    errors.push('Invoice ID must match format INV-XXXXXX');
  }
 
  return { valid: errors.length === 0, errors };
}

Content Safety Validation

Check outputs for harmful content before returning to users:

interface SafetyResult {
  safe: boolean;
  categories: Record<string, boolean>;
  flaggedSnippet?: string;
}
 
class ContentSafetyValidator {
  private readonly RULES: Array<{ name: string; pattern: RegExp }> = [
    { name: 'selfHarm', pattern: /\b(suicide|self.harm|cut myself)\b/i },
    { name: 'pii_ssn', pattern: /\b\d{3}-\d{2}-\d{4}\b/ },
    { name: 'pii_email', pattern: /\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z]{2,}\b/i },
    { name: 'pii_cc', pattern: /\b4[0-9]{12}(?:[0-9]{3})?\b/ },
  ];
 
  validate(text: string): SafetyResult {
    const categories: Record<string, boolean> = {};
    let flaggedSnippet: string | undefined;
 
    for (const rule of this.RULES) {
      const match = rule.pattern.exec(text);
      categories[rule.name] = Boolean(match);
      if (match && !flaggedSnippet) {
        const start = Math.max(0, match.index - 20);
        flaggedSnippet = text.slice(start, start + 60);
      }
    }
 
    const safe = !Object.values(categories).some(Boolean);
    return { safe, categories, flaggedSnippet };
  }
 
  assertSafe(text: string): void {
    const result = this.validate(text);
    if (!result.safe) {
      throw new Error(`Unsafe content detected: ${JSON.stringify(result.categories)}`);
    }
  }
}

Output Length and Format Enforcement

Prevent truncated or excessively long responses:

interface OutputConstraints {
  minLength: number;
  maxLength: number;
  requiredSections?: string[];
  forbiddenPhrases?: string[];
}
 
function enforceConstraints(text: string, constraints: OutputConstraints): string[] {
  const errors: string[] = [];
 
  if (text.length < constraints.minLength) {
    errors.push(`Too short: ${text.length} chars (min ${constraints.minLength})`);
  }
  if (text.length > constraints.maxLength) {
    errors.push(`Too long: ${text.length} chars (max ${constraints.maxLength})`);
  }
  for (const section of constraints.requiredSections ?? []) {
    if (!text.toLowerCase().includes(section.toLowerCase())) {
      errors.push(`Missing required section: "${section}"`);
    }
  }
  for (const phrase of constraints.forbiddenPhrases ?? []) {
    if (text.toLowerCase().includes(phrase.toLowerCase())) {
      errors.push(`Forbidden phrase found: "${phrase}"`);
    }
  }
 
  return errors;
}
 
// Example: blog post summary must be 100-300 chars with no "I cannot"
const summaryErrors = enforceConstraints(modelOutput, {
  minLength: 100,
  maxLength: 300,
  forbiddenPhrases: ["I cannot", "As an AI", "I don't have access"],
});

Common Mistakes

Parsing JSON without stripping markdown fences: Models often wrap JSON in ```json blocks. Always strip before calling JSON.parse.
Retrying without feedback: Sending the same prompt again produces the same invalid output. Include the validation error message in the retry.
Validating only at the HTTP boundary: Internal service-to-service calls bypass API-level validation. Validate at the data consumption layer too.
Ignoring stop_reason: max_tokens: A truncated response from tool use has a malformed input object. Always check stop_reason before parsing.
No fallback for validation failure: Letting a parse exception propagate to the user is a terrible experience. Define a safe default for every failure path.

Best Practices

Prefer tool-use with tool_choice: { type: 'tool' } over prompt-engineering for JSON — it enforces structure at the protocol level.
Strip markdown fences from all model responses before attempting JSON parse.
Apply content safety checks as a separate pipeline stage so they can be upgraded or swapped independently.
Log every validation failure with the raw model output for offline prompt analysis.
Set hard max_tokens limits to prevent runaway output that overflows your schema's field length constraints.
Use Zod for runtime schema validation — it generates TypeScript types automatically and produces human-readable error messages.

Key Takeaways

Every LLM response must pass schema validation before being used by downstream code; treating model output as trusted input is a critical production mistake.
Tool-use with tool_choice is more reliable than prompting for JSON because the schema is enforced at the API protocol level, not just by instruction.
Feeding validation errors back to the model in a multi-turn conversation dramatically improves validity rates on retries versus a blind retry.
Business rule validation (totals matching, dates in the future, valid enums) must run separately from JSON schema validation — they catch different classes of error.
Content safety validation should be a standalone replaceable stage, not baked into schema checking, so it can be upgraded without touching extraction logic.
A validation failure rate above 5% is a signal of a prompt engineering problem, not just noise — investigate the root cause and fix the system prompt.
Log raw model output for every failure so you can analyze patterns offline without waiting for production incidents to recur.
Define explicit fallback responses for every validation failure path so users never see raw JSON errors or exception stack traces.

Introduction#

Why This Matters#

JSON Schema Validation with Retry#

Structured Output via Tool Use#

Business Rule Validation#

Content Safety Validation#

Output Length and Format Enforcement#

Common Mistakes#

Best Practices#

Key Takeaways#

Related reading