Apache Kafka with Node.js and TypeScript — Production Guide 2026

Introduction

Why Kafka for Node.js Backends

Kafka is not just a message queue — it is a distributed event log. Every event is persisted and replayable for days or weeks, enabling multiple consumers to independently process the same stream. It handles millions of messages per second while maintaining ordering within partitions.

Use Kafka when you need high-throughput event streaming, event sourcing, audit trails, or real-time data pipelines across microservices.

Installation

npm install kafkajs

Connecting to Kafka

import { Kafka, logLevel } from 'kafkajs';
 
const kafka = new Kafka({
  clientId: 'my-service',
  brokers: (process.env.KAFKA_BROKERS ?? 'localhost:9092').split(','),
  ssl: process.env.NODE_ENV === 'production',
  sasl: process.env.KAFKA_USERNAME
    ? {
        mechanism: 'plain',
        username: process.env.KAFKA_USERNAME,
        password: process.env.KAFKA_PASSWORD ?? '',
      }
    : undefined,
  logLevel: logLevel.WARN,
  retry: { retries: 8 },
});

Producer — Sending Messages

const producer = kafka.producer({
  allowAutoTopicCreation: false,
  transactionTimeout: 30_000,
});
 
await producer.connect();
 
// Send a single message
await producer.send({
  topic: 'user-events',
  messages: [
    {
      key: 'user-42',            // key determines partition
      value: JSON.stringify({
        type: 'USER_CREATED',
        userId: 42,
        email: 'alice@example.com',
        timestamp: Date.now(),
      }),
      headers: { 'content-type': 'application/json' },
    },
  ],
});
 
// Batch send for throughput
await producer.sendBatch({
  topicMessages: [
    {
      topic: 'user-events',
      messages: users.map(u => ({
        key: `user-${u.id}`,
        value: JSON.stringify({ type: 'USER_IMPORTED', ...u }),
      })),
    },
  ],
});
 
// Graceful shutdown
process.on('SIGTERM', async () => {
  await producer.disconnect();
});

Consumer — Processing Messages

const consumer = kafka.consumer({
  groupId: 'email-service',
  sessionTimeout: 30_000,
  heartbeatInterval: 3_000,
});
 
await consumer.connect();
await consumer.subscribe({ topic: 'user-events', fromBeginning: false });
 
await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    const event = JSON.parse(message.value?.toString() ?? '{}');
    console.log(`[${topic}:${partition}] offset=${message.offset}`, event);
 
    switch (event.type) {
      case 'USER_CREATED':
        await sendWelcomeEmail(event.email);
        break;
      case 'USER_DELETED':
        await cleanupUserData(event.userId);
        break;
      default:
        console.warn('Unknown event type:', event.type);
    }
  },
});

Consumer Groups and Partitions

// Each consumer in a group gets exclusive partitions
// Scale consumers up to the number of partitions
 
// Multiple groups can read same topic independently
const analyticsConsumer = kafka.consumer({ groupId: 'analytics-service' });
const notifConsumer     = kafka.consumer({ groupId: 'notification-service' });
 
// Both consume 'user-events' from the beginning independently
await analyticsConsumer.subscribe({ topic: 'user-events', fromBeginning: true });
await notifConsumer.subscribe({ topic: 'user-events', fromBeginning: true });

Error Handling and Dead Letter Topics

await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    try {
      const event = JSON.parse(message.value?.toString() ?? '{}');
      await processEvent(event);
    } catch (err) {
      if (err instanceof SyntaxError) {
        // Malformed JSON — send to DLQ, do not retry
        await producer.send({
          topic: `${topic}.dlq`,
          messages: [{ key: message.key, value: message.value }],
        });
        return;
      }
      throw err; // Retriable error — rethrow for KafkaJS retry
    }
  },
});

Transactional Producer (Exactly-Once)

const txnProducer = kafka.producer({
  transactionalId: 'order-processor',
  maxInFlightRequests: 1,
  idempotent: true,
});
 
await txnProducer.connect();
const transaction = await txnProducer.transaction();
 
try {
  await transaction.send({
    topic: 'order-confirmed',
    messages: [{ key: 'order-1', value: JSON.stringify({ orderId: 1 }) }],
  });
  await transaction.sendOffsets({
    consumerGroupId: 'order-service',
    topics: [{ topic: 'order-requests', partitions: [{ partition: 0, offset: '10' }] }],
  });
  await transaction.commit();
} catch (err) {
  await transaction.abort();
  throw err;
}

Admin Client — Topic Management

const admin = kafka.admin();
await admin.connect();
 
await admin.createTopics({
  topics: [
    { topic: 'user-events',     numPartitions: 6, replicationFactor: 3 },
    { topic: 'user-events.dlq', numPartitions: 1, replicationFactor: 3 },
  ],
});
 
// Check consumer group lag
const offsets = await admin.fetchOffsets({
  groupId: 'email-service',
  topics: ['user-events'],
});
console.log('Consumer lag:', offsets);
 
await admin.disconnect();

Common Mistakes

Using fromBeginning: true in production — new consumer groups replay all history, causing duplicate processing
Not setting allowAutoTopicCreation: false — auto-created topics get 1 partition by default, limiting throughput
Sharing a consumer group ID between different services — they compete for partitions instead of each getting the full stream
Catching errors silently in eachMessage — this advances the offset and permanently skips the message
Treating Kafka like a task queue — messages are not removed on consumption; use retention.ms to control storage

Best Practices

Use string or UUID keys to route related events to the same partition for ordering guarantees
Set partition counts at topic creation time — repartitioning requires full consumer rebalancing
Monitor consumer group lag with Kafka UI, Burrow, or Prometheus — lag signals slow consumers
Use Avro or Protobuf with a schema registry for type-safe serialization across polyglot services
Always call producer.disconnect() and consumer.disconnect() in SIGTERM/SIGINT handlers
Test locally with bitnami/kafka in Docker; use Confluent Cloud for managed production clusters

Key Takeaways

Kafka persists events as an immutable log — multiple consumer groups can independently replay any window
Partitions enable horizontal scaling — one consumer per partition, with strict ordering within a partition
Consumer groups allow multiple services to consume the same topic without interfering with each other
Message keys colocate related events on one partition, guaranteeing order for that key
Transactional producers provide exactly-once semantics across produce and offset commits
Dead letter topics capture unprocessable messages without stalling the main consumer partition
KafkaJS is the standard TypeScript client with full producer, consumer, and admin APIs
Plan partition counts upfront — adding partitions later causes rebalancing and breaks key-based ordering

Introduction#

Why Kafka for Node.js Backends#

Installation#

Connecting to Kafka#

Producer — Sending Messages#

Consumer — Processing Messages#

Consumer Groups and Partitions#

Error Handling and Dead Letter Topics#

Transactional Producer (Exactly-Once)#

Admin Client — Topic Management#

Common Mistakes#

Best Practices#

Key Takeaways#

Related reading