Chaos Engineering in Practice — From Game Days to Automated Resilience Testing

Introduction

Why This Matters

Every production system will eventually face hardware failure, network partitions, dependency timeouts, and traffic spikes. The question is not whether these failures will happen, but whether you will discover your system's weaknesses during a controlled experiment — or during a 3 AM incident when the business impact is real.

Chaos engineering is the discipline of deliberately injecting failures in a controlled way, with clear hypotheses and rollback plans, to build confidence in your system's resilience. Netflix famously pioneered this with Chaos Monkey. In 2026, it is table stakes for any team operating at scale.

The Chaos Engineering Principles

Before running any experiment, follow these four principles:

Define a steady state — a measurable baseline of normal behavior (p99 latency, error rate, throughput)
Form a hypothesis — "We hypothesize that if the payment service becomes unavailable, order creation degrades gracefully with a 503 and the user sees a friendly error"
Introduce failure — inject the failure in a controlled, scoped way
Observe and learn — compare actual behavior to hypothesis; fix gaps before the experiment repeats

Experiment 1: Dependency Failure Injection with AWS FIS

AWS Fault Injection Service (FIS) injects failures into EC2, ECS, RDS, and more:

import {
  FISClient,
  CreateExperimentTemplateCommand,
  StartExperimentCommand,
} from '@aws-sdk/client-fis';
 
const fis = new FISClient({ region: 'us-east-1' });
 
// Create a template: kill 25% of ECS tasks running the payment service
const template = await fis.send(new CreateExperimentTemplateCommand({
  description: 'Kill 25% of payment service tasks',
  roleArn: process.env.FIS_ROLE_ARN!,
  stopConditions: [
    {
      source: 'aws:cloudwatch:alarm',
      value: process.env.ERROR_RATE_ALARM_ARN!, // Stop if error rate > 5%
    },
  ],
  targets: {
    'payment-tasks': {
      resourceType: 'aws:ecs:task',
      resourceTags: { Service: 'payment' },
      selectionMode: 'PERCENT(25)',
    },
  },
  actions: {
    'stop-tasks': {
      actionId: 'aws:ecs:stop-task',
      targets: { Tasks: 'payment-tasks' },
    },
  },
  tags: { Environment: 'staging', Experiment: 'payment-kill-25pct' },
}));
 
// Start the experiment
const experiment = await fis.send(new StartExperimentCommand({
  experimentTemplateId: template.experimentTemplate!.id!,
}));
 
console.log(`Experiment started: ${experiment.experiment!.id}`);

Experiment 2: Network Latency Injection

Inject artificial latency into service-to-service calls to test timeout and circuit breaker configurations:

import { exec } from 'child_process';
import { promisify } from 'util';
 
const execAsync = promisify(exec);
 
// Linux tc (traffic control) for network delay injection
// Use in staging/canary environments only
 
async function injectNetworkDelay(
  interfaceName: string,
  targetHost: string,
  delayMs: number,
  jitterMs: number
): Promise<() => Promise<void>> {
  // Add delay to traffic destined for target host
  await execAsync(`
    tc qdisc add dev ${interfaceName} root handle 1: prio &&
    tc filter add dev ${interfaceName} parent 1: protocol ip u32
      match ip dst ${targetHost}/32
      flowid 1:3 &&
    tc qdisc add dev ${interfaceName} parent 1:3 handle 30:
      netem delay ${delayMs}ms ${jitterMs}ms
  `);
 
  console.log(`Injected ${delayMs}ms ±${jitterMs}ms delay to ${targetHost}`);
 
  // Return cleanup function
  return async () => {
    await execAsync(`tc qdisc del dev ${interfaceName} root`);
    console.log('Network delay removed');
  };
}
 
// Example usage
const cleanup = await injectNetworkDelay('eth0', '10.0.1.50', 500, 100);
 
// Run your test assertions here
await runLoadTest({ duration: 60, rps: 100 });
await checkMetrics({
  maxP99LatencyMs: 600,
  maxErrorRate: 0.01,
});
 
// Clean up
await cleanup();

Experiment 3: Database Connection Pool Exhaustion

Test what happens when all database connections are in use:

// chaos-experiment-db-pool.ts
import { Pool } from 'pg';
 
interface ExperimentResult {
  hypothesis: string;
  passed: boolean;
  observations: string[];
}
 
async function runPoolExhaustionExperiment(): Promise<ExperimentResult> {
  const pool = new Pool({ max: 10, connectionTimeoutMillis: 5000 });
  const observations: string[] = [];
 
  // Steady state: measure baseline p99 latency
  const baseline = await measureLatency(pool, 100);
  observations.push(`Baseline p99: ${baseline.p99}ms`);
 
  // Hypothesis: App returns 503 with helpful error within 5s when pool exhausted
  // Inject: hold all 10 connections for 30 seconds
  const heldConnections = await Promise.all(
    Array.from({ length: 10 }, () => pool.connect())
  );
 
  const start = Date.now();
  try {
    const result = await pool.query('SELECT 1');
    observations.push(`ERROR: Query succeeded when pool should be exhausted`);
  } catch (err: unknown) {
    const elapsed = Date.now() - start;
    const message = err instanceof Error ? err.message : String(err);
    observations.push(`Query failed after ${elapsed}ms: ${message}`);
 
    const passed = elapsed < 5500 && message.includes('timeout');
    if (!passed) {
      observations.push('FAIL: Timeout not surfaced correctly to application');
    }
  } finally {
    heldConnections.forEach(c => c.release());
    await pool.end();
  }
 
  return {
    hypothesis: 'Pool exhaustion returns 503 within 5 seconds',
    passed: observations.some(o => o.includes('timeout')),
    observations,
  };
}

Building an Automated Chaos Pipeline

Run chaos experiments as part of your CI/CD pipeline on staging before every major release:

# .github/workflows/chaos.yml
name: Chaos Engineering Tests
 
on:
  schedule:
    - cron: '0 2 * * 1' # Every Monday at 2 AM
  workflow_dispatch:
 
jobs:
  chaos-tests:
    runs-on: ubuntu-latest
    environment: staging
 
    steps:
      - uses: actions/checkout@v4
 
      - name: Deploy to staging
        run: ./scripts/deploy-staging.sh
 
      - name: Wait for stable state
        run: ./scripts/wait-for-healthy.sh --timeout=300
 
      - name: Record baseline metrics
        run: |
          ./scripts/record-metrics.sh \
            --duration=60 \
            --output=baseline.json
 
      - name: Run chaos experiment - dependency failure
        run: |
          ./scripts/run-chaos-experiment.sh \
            --experiment=payment-service-down \
            --duration=120 \
            --rollback-on-error
 
      - name: Compare metrics to baseline
        run: |
          ./scripts/compare-metrics.sh \
            --baseline=baseline.json \
            --max-error-rate=0.05 \
            --max-latency-increase-pct=50
 
      - name: Generate chaos report
        if: always()
        run: ./scripts/generate-chaos-report.sh

Game Days: Structured Team Resilience Testing

A game day is a scheduled practice exercise where the team deliberately triggers real failure scenarios:

Game Day Structure (4 hours):
 
09:00 — Briefing
  - Scenario: "Primary database becomes unavailable"
  - Hypothesis: Reads fail over to replica within 30s, writes queue
  - Observers: SRE team monitors dashboards
  - Rollback: Restore primary, verify replication
 
09:30 — Execute
  - Kill primary DB instance
  - Start timer
  - Record: time to detect, time to alert, time to failover
 
10:00 — Debrief
  - Actual failover time: 4m 12s (hypothesis: under 1 minute)
  - Missing alert: connection pool exhaustion not in runbooks
  - Action items: reduce failover target time, add connection pool alert
 
11:00 — Fix and retest
  - Implement connection pool alert
  - Rerun experiment: 47s failover time
 
12:00 — Retrospective and documentation

Common Mistakes

Running chaos in production without stop conditions. Always define a CloudWatch alarm or SLO breach as an automatic stop condition. Never run open-ended experiments without an automated kill switch.

No steady-state baseline. Without measuring before injecting failure, you cannot tell whether your observations represent a problem or existing baseline behavior.

Skipping staging. Always run new experiments in staging first. Never run an untested chaos experiment in production.

Chaos without on-call team awareness. Always notify your on-call team before running experiments. A chaos experiment that pages the wrong team is a trust-destroying false alarm.

Not documenting findings. The value of chaos engineering is in the learning. Every experiment should produce a written summary of hypothesis, observations, and action items.

Best Practices

Always define measurable stop conditions before starting any experiment
Start with the lowest-impact experiments (latency injection) before high-impact ones (instance termination)
Use automated chaos pipelines on staging to catch regressions before production releases
Document every experiment with hypothesis, results, and action items in your runbooks
Run quarterly game days with the full engineering team to practice incident response
Treat chaos findings as P1 engineering work — fix weaknesses discovered before running the next experiment

Key Takeaways

Chaos engineering discovers system weaknesses during controlled experiments rather than during production incidents when business impact is highest.
Every chaos experiment requires a steady-state baseline, a clear hypothesis, a defined stop condition, and a rollback plan — never ad-hoc failure injection.
AWS Fault Injection Service (FIS) supports structured failure injection into ECS tasks, RDS instances, EC2 nodes, and network paths with automatic stop conditions via CloudWatch alarms.
Network latency injection with Linux tc (traffic control) tests whether timeout configurations and circuit breakers are tuned correctly for real-world latency scenarios.
Pool exhaustion experiments verify that your application returns a graceful 503 within a defined timeout rather than hanging indefinitely when all database connections are held.
Automated chaos pipelines integrated into CI/CD run experiments on staging before every major release, catching resilience regressions early.
Game days are structured team exercises that practice real incident response, exposing gaps in runbooks, alerting, and team coordination that unit tests cannot find.
Every chaos finding should be treated as P1 engineering work — the weakness must be fixed before running the next experiment to avoid compounding risk.

Introduction#

Why This Matters#

The Chaos Engineering Principles#

Experiment 1: Dependency Failure Injection with AWS FIS#

Experiment 2: Network Latency Injection#

Experiment 3: Database Connection Pool Exhaustion#

Building an Automated Chaos Pipeline#

Game Days: Structured Team Resilience Testing#

Common Mistakes#

Best Practices#

Key Takeaways#