Idempotency in Distributed Systems — Making Any Operation Safe to Retry

Introduction

Network failures are inevitable. A client retries a request, which succeeds, but the response is lost. Without idempotency, that retry creates a duplicate charge, duplicate order, or duplicate message. Idempotency is your shield: the same request executed multiple times produces the same result as executing once. We'll explore idempotency key design, storage strategies, and how to verify duplicates at scale.

Idempotency Key Design
Idempotency Store with TTL
Database-Level Upsert Patterns
Idempotent Webhook Processing
CQRS Command Deduplication
Testing Idempotency with Concurrent Requests
HTTP Status Codes for Duplicates
Checklist
Conclusion

Idempotency Key Design

An idempotency key uniquely identifies a request intent. It can be client-generated or deterministically derived.

Client-generated with UUID:

interface CreateChargeRequest {
  amount: number;
  currency: string;
  customerId: string;
  idempotencyKey: string; // Client provides UUID
}

class PaymentAPI {
  async createCharge(req: CreateChargeRequest): Promise<ChargeResponse> {
    const normalized = {
      method: 'POST',
      path: '/charges',
      body: req,
      idempotencyKey: req.idempotencyKey,
    };

    // Check if we've seen this idempotency key
    const cached = await this.idempotencyStore.get(req.idempotencyKey);
    if (cached) {
      return cached.response; // Return previous response
    }

    const charge = await this.stripe.charges.create({
      amount: req.amount,
      currency: req.currency,
      customer: req.customerId,
    });

    const response: ChargeResponse = {
      chargeId: charge.id,
      amount: charge.amount,
      status: charge.status,
    };

    // Store result with TTL (24 hours)
    await this.idempotencyStore.set(req.idempotencyKey, { response, timestamp: Date.now() }, 86400);

    return response;
  }
}

interface ChargeResponse {
  chargeId: string;
  amount: number;
  status: 'succeeded' | 'pending' | 'failed';
}

Deterministic hashing from request content:

import crypto from 'crypto';

class DeterministicIdempotency {
  generateKey(method: string, path: string, body: any): string {
    const normalized = JSON.stringify({
      method,
      path,
      body: this.normalizeBody(body),
    });

    return crypto.createHash('sha256').update(normalized).digest('hex');
  }

  private normalizeBody(body: any): any {
    // Ensure consistent ordering and structure
    if (Array.isArray(body)) {
      return body.map(item => this.normalizeBody(item));
    }
    if (typeof body === 'object' && body !== null) {
      return Object.keys(body)
        .sort()
        .reduce((acc, key) => {
          acc[key] = this.normalizeBody(body[key]);
          return acc;
        }, {} as Record<string, any>);
    }
    return body;
  }

  async createTransfer(req: any): Promise<any> {
    const idempotencyKey = this.generateKey('POST', '/transfers', req);

    const cached = await this.idempotencyStore.get(idempotencyKey);
    if (cached) {
      return cached;
    }

    const result = await this.processTransfer(req);
    await this.idempotencyStore.set(idempotencyKey, result, 86400);
    return result;
  }

  private async processTransfer(req: any): Promise<any> {
    // Implementation
    return {};
  }
}

Idempotency Store with TTL

Redis is ideal for idempotency stores: fast, supports TTL, and atomic operations.

class RedisIdempotencyStore {
  constructor(private redis: Redis) {}

  async get(key: string): Promise<any> {
    const value = await this.redis.get(`idempotent:${key}`);
    if (!value) return null;
    try {
      return JSON.parse(value);
    } catch {
      return null;
    }
  }

  async set(key: string, value: any, ttlSeconds: number): Promise<void> {
    const serialized = JSON.stringify({
      response: value,
      storedAt: Date.now(),
    });
    await this.redis.setex(`idempotent:${key}`, ttlSeconds, serialized);
  }

  async exists(key: string): Promise<boolean> {
    return (await this.redis.exists(`idempotent:${key}`)) === 1;
  }

  async delete(key: string): Promise<void> {
    await this.redis.del(`idempotent:${key}`);
  }

  // For cleanup: scan and remove expired keys
  async cleanup(): Promise<number> {
    const cursor = '0';
    let cleaned = 0;
    const scan = async (c: string): Promise<string> => {
      const [newCursor, keys] = await this.redis.scan(c, 'MATCH', 'idempotent:*', 'COUNT', 100);
      for (const key of keys) {
        const ttl = await this.redis.ttl(key);
        if (ttl === -1) {
          // No TTL set, delete it
          await this.redis.del(key);
          cleaned++;
        }
      }
      return newCursor;
    };

    let currentCursor = cursor;
    do {
      currentCursor = await scan(currentCursor);
    } while (currentCursor !== '0');

    return cleaned;
  }
}

Database-Level Upsert Patterns

For operations that must update state (not just read), use database upserts with unique constraints.

class OrderService {
  async createOrder(req: CreateOrderRequest): Promise<Order> {
    const idempotencyKey = req.idempotencyKey;

    // Upsert with ON CONFLICT
    const result = await this.db.query(
      `INSERT INTO orders (idempotency_key, customer_id, total, status, created_at)
       VALUES ($1, $2, $3, 'pending', NOW())
       ON CONFLICT (idempotency_key)
       DO UPDATE SET updated_at = NOW()
       RETURNING id, customer_id, total, status, created_at`,
      [idempotencyKey, req.customerId, req.totalAmount]
    );

    const order = result.rows[0];

    // If this is a retry, the INSERT will be skipped and we return existing order
    // Ensure the order is not already processed
    if (order.status !== 'pending') {
      return order;
    }

    // Proceed with order processing
    await this.db.query(
      `UPDATE orders SET status = 'processing' WHERE id = $1`,
      [order.id]
    );

    return order;
  }

  async updateOrderStatus(orderId: string, idempotencyKey: string, status: string): Promise<void> {
    const result = await this.db.query(
      `INSERT INTO order_status_changes (order_id, idempotency_key, new_status, changed_at)
       VALUES ($1, $2, $3, NOW())
       ON CONFLICT (order_id, idempotency_key)
       DO NOTHING
       RETURNING id`,
      [orderId, idempotencyKey, status]
    );

    // Only update if this is the first time we've seen this idempotency key
    if (result.rows.length > 0) {
      await this.db.query(`UPDATE orders SET status = $1 WHERE id = $2`, [status, orderId]);
    }
  }
}

Idempotent Webhook Processing

Webhooks are retried on failure. Process them idempotently.

interface WebhookEvent {
  eventId: string; // Unique across all events
  eventType: string;
  timestamp: number;
  data: any;
}

class WebhookProcessor {
  async handleWebhook(event: WebhookEvent): Promise<void> {
    // Use eventId as idempotency key
    const cached = await this.idempotencyStore.get(`webhook:${event.eventId}`);
    if (cached) {
      return; // Already processed
    }

    try {
      await this.processEvent(event);
      await this.idempotencyStore.set(`webhook:${event.eventId}`, { processed: true }, 86400);
    } catch (error) {
      // Don't cache errors; allow retry
      throw error;
    }
  }

  private async processEvent(event: WebhookEvent): Promise<void> {
    switch (event.eventType) {
      case 'payment.completed':
        await this.handlePaymentCompleted(event.data);
        break;
      case 'payment.failed':
        await this.handlePaymentFailed(event.data);
        break;
      default:
        throw new Error(`Unknown event type: ${event.eventType}`);
    }
  }

  private async handlePaymentCompleted(data: any): Promise<void> {
    // Safe to call multiple times
    await this.db.query(
      `UPDATE orders SET status = 'paid', paid_at = NOW()
       WHERE payment_id = $1 AND status != 'paid'`,
      [data.paymentId]
    );
  }

  private async handlePaymentFailed(data: any): Promise<void> {
    await this.db.query(
      `UPDATE orders SET status = 'payment_failed' WHERE payment_id = $1`,
      [data.paymentId]
    );
  }
}

CQRS Command Deduplication

In CQRS, commands are deduplicated at the command handler level.

interface Command {
  commandId: string; // Unique command identifier
  type: string;
  payload: any;
  timestamp: number;
}

class CommandHandler {
  private commandStore = new Map<string, any>(); // or Redis

  async execute<T>(command: Command): Promise<T> {
    // Check if command was already executed
    const result = await this.getCommandResult(command.commandId);
    if (result !== undefined) {
      return result as T;
    }

    // Execute the command
    const commandResult = await this.executeCommand(command);

    // Store result
    await this.storeCommandResult(command.commandId, commandResult);

    return commandResult as T;
  }

  private async executeCommand(command: Command): Promise<any> {
    switch (command.type) {
      case 'CreateAccount':
        return await this.createAccount(command.payload);
      case 'DepositFunds':
        return await this.depositFunds(command.payload);
      case 'WithdrawFunds':
        return await this.withdrawFunds(command.payload);
      default:
        throw new Error(`Unknown command: ${command.type}`);
    }
  }

  private async createAccount(payload: any): Promise<string> {
    const accountId = await this.db.query(
      `INSERT INTO accounts (owner_id, balance, created_at)
       VALUES ($1, $2, NOW())
       RETURNING id`,
      [payload.ownerId, payload.initialBalance]
    );
    return accountId.rows[0].id;
  }

  private async depositFunds(payload: any): Promise<void> {
    await this.db.query(
      `UPDATE accounts SET balance = balance + $1 WHERE id = $2`,
      [payload.amount, payload.accountId]
    );
  }

  private async withdrawFunds(payload: any): Promise<void> {
    const result = await this.db.query(
      `UPDATE accounts SET balance = balance - $1
       WHERE id = $2 AND balance >= $1
       RETURNING balance`,
      [payload.amount, payload.accountId]
    );
    if (result.rows.length === 0) {
      throw new Error('Insufficient funds');
    }
  }

  private async getCommandResult(commandId: string): Promise<any> {
    const result = await this.db.query(
      `SELECT result FROM command_results WHERE command_id = $1`,
      [commandId]
    );
    return result.rows[0]?.result;
  }

  private async storeCommandResult(commandId: string, result: any): Promise<void> {
    await this.db.query(
      `INSERT INTO command_results (command_id, result, executed_at)
       VALUES ($1, $2, NOW())
       ON CONFLICT (command_id) DO NOTHING`,
      [commandId, JSON.stringify(result)]
    );
  }
}

Testing Idempotency with Concurrent Requests

describe('Idempotency', () => {
  it('should deduplicate concurrent identical requests', async () => {
    const service = new PaymentService(store, gateway);
    const idempotencyKey = uuid();

    const promises = Array(5)
      .fill(null)
      .map(() =>
        service.createCharge({
          amount: 1000,
          currency: 'USD',
          customerId: 'cust-123',
          idempotencyKey,
        })
      );

    const results = await Promise.all(promises);

    // All should return same charge ID
    const chargeIds = new Set(results.map(r => r.chargeId));
    expect(chargeIds.size).toBe(1);

    // Only one charge should exist in system
    const charges = await gateway.listCharges({ customerId: 'cust-123' });
    expect(charges.filter(c => c.amount === 1000)).toHaveLength(1);
  });

  it('should return 409 Conflict when retry with same key completes later', async () => {
    const service = new PaymentService(store, gateway);
    const idempotencyKey = uuid();

    const first = service.createCharge({
      amount: 2000,
      currency: 'USD',
      customerId: 'cust-456',
      idempotencyKey,
    });

    const second = service.createCharge({
      amount: 2000,
      currency: 'USD',
      customerId: 'cust-456',
      idempotencyKey,
    });

    const [result1, result2] = await Promise.all([first, second]);
    expect(result1.chargeId).toBe(result2.chargeId);
  });
});

HTTP Status Codes for Duplicates

200 OK: Retry of successful request; return cached response
202 Accepted: Request still processing; advise client to retry
409 Conflict: Different request with same idempotency key; reject with details

class HTTPHandler {
  async post(req: Request): Promise<Response> {
    const idempotencyKey = req.headers['idempotency-key'];
    if (!idempotencyKey) {
      return Response.json({ error: 'Missing idempotency-key' }, { status: 400 });
    }

    const inProgress = await this.store.isInProgress(idempotencyKey);
    if (inProgress) {
      return Response.json({ error: 'Request in progress' }, { status: 202 });
    }

    const existing = await this.store.get(idempotencyKey);
    if (existing) {
      if (existing.requestHash === this.hashRequest(req)) {
        return Response.json(existing.response, { status: 200 });
      } else {
        return Response.json(
          { error: 'Different request with same idempotency key' },
          { status: 409 }
        );
      }
    }

    const response = await this.processRequest(req);
    await this.store.set(idempotencyKey, { response, requestHash: this.hashRequest(req) });
    return Response.json(response, { status: 201 });
  }

  private hashRequest(req: Request): string {
    return crypto
      .createHash('sha256')
      .update(JSON.stringify({ body: req.body }))
      .digest('hex');
  }
}

Checklist

Require idempotency keys in all API requests that mutate state
Store idempotency results with TTL (24+ hours)
Use unique database constraints for insert operations
Verify all operations are truly idempotent (no side effects on retry)
Test with concurrent retries to catch race conditions
Return appropriate HTTP status codes (200, 202, 409)
Monitor idempotency cache hit rate
Document which endpoints require idempotency keys

Conclusion

Idempotency transforms retries from a liability into a safety net. Design your keys thoughtfully, store results durably, and use database constraints to prevent accidental duplicates. Test concurrency aggressively—the production network will find your assumptions wrong.