Duplicate Event Processing — When Your Queue Delivers the Same Message Twice

Introduction

Every major message queue — Kafka, RabbitMQ, SQS — makes an at-least-once delivery guarantee. Under failure conditions, they will deliver the same message more than once. If your consumers are not idempotent, duplicate events cause real damage: orders ship twice, emails send twice, payments charge twice. This post covers five production-tested strategies for building consumers that handle duplicates safely.

Why Queues Deliver Duplicates

The root cause is the gap between processing completion and acknowledgement. A consumer picks up a message, starts processing, the visibility timeout expires, and a second consumer picks up the same message. Both eventually acknowledge it. Both processed it.

SQS default visibility timeout is 30 seconds. If your handler takes 45 seconds, SQS assumes the first consumer failed and redelivers. Kafka has the same problem: if the consumer crashes after processing but before committing the offset, the message is redelivered on restart.

This is not a bug. At-least-once delivery is the price of durability and availability. Exactly-once delivery requires distributed transactions, which are expensive. The correct answer is building consumers that tolerate duplicates.

Strategy 1 — Redis SET NX Idempotency Check

The simplest approach: before processing, try to claim the event ID in Redis using an atomic SET NX (set if not exists). If the key already exists, the event was already processed.

const Redis = require('ioredis')
 
class IdempotentConsumer {
  constructor(redis, ttlSeconds = 86400 * 7) {
    this.redis = redis
    this.ttlSeconds = ttlSeconds
  }
 
  async processOnce(eventId, handler) {
    const key = `processed:${eventId}`
 
    // Atomic: only sets if key does not exist
    const acquired = await this.redis.set(key, '1', 'EX', this.ttlSeconds, 'NX')
 
    if (!acquired) {
      console.log(`Skipping duplicate event: ${eventId}`)
      return { processed: false }
    }
 
    try {
      const result = await handler()
      return { processed: true, result }
    } catch (err) {
      // Remove key so the event can be retried on genuine failures
      await this.redis.del(key)
      throw err
    }
  }
}
 
// Usage
const consumer = new IdempotentConsumer(redis)
 
async function handleOrderPlaced(event) {
  const { processed } = await consumer.processOnce(event.eventId, async () => {
    await db.order.create(event.order)
    await emailService.sendConfirmation(event.order.email)
    await inventoryService.reserve(event.order.items)
  })
 
  if (!processed) {
    console.log(`Duplicate event ${event.eventId} skipped`)
  }
}

The TTL prevents the Redis keyspace from growing unbounded. Seven days is appropriate for most queue systems — duplicates rarely arrive after that window.

Strategy 2 — Database Unique Constraint

Redis can fail or be evicted under memory pressure. A stronger guarantee: insert a row into a processed_events table inside the same transaction as your business logic. A unique constraint on event_id makes the entire operation idempotent at the database level.

async function processWithDbDedup(eventId, handler, handlerName) {
  return db.transaction(async (trx) => {
    // Insert into dedup table — unique constraint prevents double processing
    const inserted = await trx.raw(`
      INSERT INTO processed_events (event_id, handler)
      VALUES (?, ?)
      ON CONFLICT (event_id) DO NOTHING
      RETURNING event_id
    `, [eventId, handlerName])
 
    if (inserted.rows.length === 0) {
      console.log(`Duplicate event ${eventId} ignored`)
      return
    }
 
    // Business logic runs in the same transaction
    await handler(trx)
    // If handler throws, the INSERT also rolls back — safe to retry
  })
}
 
// Kafka consumer
consumer.on('message', async (message) => {
  const event = JSON.parse(message.value)
 
  await processWithDbDedup(event.eventId, 'order-fulfillment', async (trx) => {
    await fulfillOrder(event.orderId, trx)
  })
 
  await message.commit()
})

The key insight: if the handler fails, the entire transaction rolls back including the dedup row. The event can be retried. If the handler succeeds, the dedup row is committed atomically — a second attempt will find the row and skip cleanly.

Strategy 3 — Idempotent Operations by Design

Some operations can be made naturally idempotent without tracking event IDs at all.

// NOT idempotent — inserts a new row each time
await db.query('INSERT INTO emails_sent (order_id, email) VALUES (?, ?)', [orderId, email])
 
// Idempotent — ON CONFLICT DO NOTHING
await db.query(`
  INSERT INTO emails_sent (order_id, email)
  VALUES (?, ?)
  ON CONFLICT (order_id) DO NOTHING
`, [orderId, email])
 
// NOT idempotent — increments balance on each call
await db.query('UPDATE wallets SET balance = balance + ? WHERE user_id = ?', [amount, userId])
 
// Idempotent — check idempotency_key before incrementing
await db.transaction(async (trx) => {
  const alreadyApplied = await trx('transactions')
    .where({ idempotency_key: txKey })
    .first()
 
  if (!alreadyApplied) {
    await trx('wallets').where({ user_id: userId }).increment('balance', amount)
    await trx('transactions').insert({ idempotency_key: txKey, amount, user_id: userId })
  }
})

Designing for idempotency at the operation level is the most robust approach. There is no external state to manage and no risk of the dedup store going out of sync.

Strategy 4 — Kafka Exactly-Once with Transactional Producers

Kafka 0.11+ supports exactly-once semantics by atomically committing the output event and the consumer offset in a single transaction. If the transaction aborts, the offset does not advance and the message is redelivered — but the output is also rolled back, so there is no duplicate.

const { Kafka } = require('kafkajs')
 
const kafka = new Kafka({ brokers: ['localhost:9092'] })
const consumer = kafka.consumer({ groupId: 'order-processor' })
 
const producer = kafka.producer({
  transactionalId: 'order-processor-txn',
  maxInFlightRequests: 1,
  idempotent: true,
})
 
await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    const event = JSON.parse(message.value.toString())
 
    await producer.transaction(async (tx) => {
      // Business logic
      await processOrder(event)
 
      // Emit output event
      await tx.send({
        topic: 'order-confirmed',
        messages: [{ value: JSON.stringify({ orderId: event.orderId }) }],
      })
 
      // Commit input offset in same transaction
      await tx.sendOffsets({
        consumerGroupId: 'order-processor',
        topics: [{
          topic,
          partitions: [{ partition, offset: (BigInt(message.offset) + 1n).toString() }],
        }],
      })
    })
  },
})

This is the strongest guarantee available in Kafka but requires the transactional producer configuration and limits throughput due to maxInFlightRequests: 1.

Strategy 5 — Extend Visibility Timeout for Long Processing

For SQS, the most common cause of duplicates is processing that outlasts the visibility timeout. Extend the timeout while processing rather than letting the message become visible again.

const { SQSClient, ChangeMessageVisibilityCommand, DeleteMessageCommand } = require('@aws-sdk/client-sqs')
 
const sqs = new SQSClient({})
 
async function processWithExtension(message, handler) {
  const EXTENSION_INTERVAL_MS = 60_000  // Extend every 60 seconds
  const EXTENSION_AMOUNT_SECS = 90      // Add 90 seconds each time
 
  const extender = setInterval(async () => {
    await sqs.send(new ChangeMessageVisibilityCommand({
      QueueUrl: process.env.QUEUE_URL,
      ReceiptHandle: message.ReceiptHandle,
      VisibilityTimeout: EXTENSION_AMOUNT_SECS,
    }))
  }, EXTENSION_INTERVAL_MS)
 
  try {
    await handler()
 
    await sqs.send(new DeleteMessageCommand({
      QueueUrl: process.env.QUEUE_URL,
      ReceiptHandle: message.ReceiptHandle,
    }))
  } finally {
    clearInterval(extender)
  }
}

Monitoring Duplicate Rates in Production

A sudden spike in duplicates indicates either infrastructure problems or consumers crashing before acknowledging.

const duplicateCounter = { total: 0, duplicates: 0 }
 
async function handleEvent(event) {
  duplicateCounter.total++
 
  const { processed } = await consumer.processOnce(event.eventId, async () => {
    await processEvent(event)
  })
 
  if (!processed) {
    duplicateCounter.duplicates++
    const rate = duplicateCounter.duplicates / duplicateCounter.total
    if (rate > 0.01) {
      // Alert: duplicate rate above 1%
      console.warn(`High duplicate rate: ${(rate * 100).toFixed(2)}%`)
    }
  }
}

A duplicate rate above 1% is worth investigating. Typical causes: visibility timeout too short, consumers crashing under load, or a redeployment that reset consumer group offsets.

Key Takeaways

At-least-once delivery is the default for every production message queue — duplicates will happen under failure conditions.
Redis SET NX on the event ID is the simplest idempotency check; pair it with a 7-day TTL to bound keyspace growth.
Database unique constraints inside the same transaction as business logic provide the strongest consistency guarantee.
Designing operations to be naturally idempotent (ON CONFLICT DO NOTHING, idempotency keys) eliminates the need for external dedup state.
Kafka transactional producers achieve exactly-once by atomically committing output events and consumer offsets.
SQS visibility timeout extension prevents the most common cause of duplicates in long-running handlers.
Monitor duplicate rates — a rate above 1% indicates infrastructure or consumer lifecycle problems worth fixing.
Never delete the dedup key on success; only delete on failure so genuine retries can proceed.

Introduction#

Why Queues Deliver Duplicates#

Strategy 1 — Redis SET NX Idempotency Check#

Strategy 2 — Database Unique Constraint#

Strategy 3 — Idempotent Operations by Design#

Strategy 4 — Kafka Exactly-Once with Transactional Producers#

Strategy 5 — Extend Visibility Timeout for Long Processing#

Monitoring Duplicate Rates in Production#

Key Takeaways#

Related reading