The Bulkhead Pattern — Isolating Failures So One Bad Dependency Doesn't Sink Everything

Introduction

A slow payment service consuming all available threads should not prevent the user profile service from responding. The bulkhead pattern solves this by giving each downstream dependency its own concurrency budget — when one fills up, the others continue operating normally. This post builds semaphore-based bulkheads in Node.js, combines them with circuit breakers, and covers tenant isolation and proper sizing formulas.

Why Bulkheads Matter

Without isolation, a single slow dependency cascades to a full outage:

Payment service starts responding in 10 seconds instead of 200ms
All 50 available thread/connection slots fill with in-flight payment requests
User profile requests, search requests, and health checks all queue behind them
Everything times out — the system looks down even though only one service is slow

Bulkheads prevent this by capping how many requests can be in-flight to any single dependency simultaneously.

Concurrency Limiter with p-limit

The simplest bulkhead uses p-limit to cap concurrent calls per service:

import pLimit from 'p-limit';
 
class BulkheadExecutor {
  constructor() {
    this.limits = {
      'payment-service': pLimit(10),
      'user-service': pLimit(50),
      'product-service': pLimit(100),
    };
  }
 
  async execute(serviceName, fn) {
    const limit = this.limits[serviceName] ?? pLimit(20);
    return limit(fn);
  }
}
 
const bulkhead = new BulkheadExecutor();
 
// Payment service: only 10 concurrent requests
app.post('/checkout', async (req, res) => {
  try {
    const result = await bulkhead.execute(
      'payment-service',
      () => paymentService.charge(req.body)
    );
    res.json(result);
  } catch (err) {
    res.status(503).json({ error: 'Service temporarily unavailable' });
  }
});
 
// User service: 50 concurrent requests — unaffected by payment slowness
app.get('/user/:id', async (req, res) => {
  const user = await bulkhead.execute(
    'user-service',
    () => userService.getUser(req.params.id)
  );
  res.json(user);
});

Semaphore-Based Bulkhead

A semaphore gives you finer control and visibility into queue depth:

class Semaphore {
  constructor(permits) {
    this.permits = permits;
    this.waiters = [];
  }
 
  acquire() {
    if (this.permits > 0) {
      this.permits--;
      return Promise.resolve();
    }
    return new Promise((resolve) => {
      this.waiters.push(() => { this.permits--; resolve(); });
    });
  }
 
  release() {
    if (this.waiters.length > 0) {
      const waiter = this.waiters.shift();
      waiter();
    } else {
      this.permits++;
    }
  }
 
  async withPermit(fn) {
    await this.acquire();
    try {
      return await fn();
    } finally {
      this.release();
    }
  }
 
  get available() { return this.permits; }
  get queued() { return this.waiters.length; }
}
 
class SemaphoreBulkhead {
  constructor(limits) {
    this.semaphores = new Map(
      Object.entries(limits).map(([name, n]) => [name, new Semaphore(n)])
    );
  }
 
  execute(service, fn) {
    const sem = this.semaphores.get(service);
    if (!sem) throw new Error(`No bulkhead for ${service}`);
    return sem.withPermit(fn);
  }
 
  metrics(service) {
    const sem = this.semaphores.get(service);
    return sem ? { available: sem.available, queued: sem.queued } : null;
  }
}
 
// Health endpoint exposes bulkhead utilization
app.get('/metrics/bulkhead', (req, res) => {
  const b = new SemaphoreBulkhead({ 'payment-service': 10, 'user-service': 50 });
  res.json({
    payment: b.metrics('payment-service'),
    user:    b.metrics('user-service'),
  });
});

Bulkhead + Circuit Breaker Combination

Bulkheads limit concurrency; circuit breakers stop calling a failing service altogether. Together they provide defense in depth:

class ResilientClient {
  constructor(serviceLimits) {
    this.bulkhead = new SemaphoreBulkhead(serviceLimits);
    this.failures = {};
    this.states = {};
  }
 
  async call(service, fn) {
    const state = this.states[service] ?? 'CLOSED';
 
    if (state === 'OPEN') {
      throw new Error(`Circuit open for ${service}`);
    }
 
    try {
      const result = await this.bulkhead.execute(service, fn);
      this.onSuccess(service);
      return result;
    } catch (err) {
      this.onFailure(service);
      throw err;
    }
  }
 
  onSuccess(service) {
    this.failures[service] = 0;
    if (this.states[service] === 'HALF_OPEN') {
      this.states[service] = 'CLOSED';
      console.log(`Circuit closed for ${service}`);
    }
  }
 
  onFailure(service) {
    this.failures[service] = (this.failures[service] ?? 0) + 1;
    if (this.failures[service] >= 5) {
      this.states[service] = 'OPEN';
      console.error(`Circuit opened for ${service}`);
      setTimeout(() => {
        this.states[service] = 'HALF_OPEN';
      }, 30000);
    }
  }
}

Tenant Isolation: Preventing Noisy Neighbors

In multi-tenant systems, one customer's burst should not degrade others:

class TenantBulkhead {
  constructor() {
    this.semaphores = new Map();
    this.planLimits = {
      enterprise: 100,
      pro: 50,
      free: 10,
    };
  }
 
  getSemaphore(tenantId, plan) {
    if (!this.semaphores.has(tenantId)) {
      const limit = this.planLimits[plan] ?? this.planLimits.free;
      this.semaphores.set(tenantId, new Semaphore(limit));
    }
    return this.semaphores.get(tenantId);
  }
 
  execute(tenantId, plan, fn) {
    return this.getSemaphore(tenantId, plan).withPermit(fn);
  }
}
 
const tenantBulkhead = new TenantBulkhead();
 
app.use(async (req, res, next) => {
  const tenantId = req.headers['x-tenant-id'];
  const plan = req.tenantPlan ?? 'free';
 
  try {
    await tenantBulkhead.execute(tenantId, plan, async () => {
      await next();
    });
  } catch (err) {
    res.status(503).json({ error: 'Tenant quota exceeded' });
  }
});

Enterprise tenants get 100 concurrent slots, free-tier users get 10. If a free tenant spams the API, their requests queue in their own semaphore — enterprise users are unaffected.

Sizing Bulkheads Correctly

Too small and you artificially limit throughput; too large and you don't provide isolation. Use Little's Law:

bulkhead_size = (target_throughput_rps * p99_latency_seconds) * safety_buffer

Worked examples:

function sizeBulkhead(throughputRps, p99LatencyMs, bufferFactor = 1.3) {
  const latencySec = p99LatencyMs / 1000;
  return Math.ceil(throughputRps * latencySec * bufferFactor);
}
 
// Payment: 10 RPS, 2s P99 -> ceil(10 * 2 * 1.3) = 26
console.log(sizeBulkhead(10, 2000));
 
// Search: 100 RPS, 500ms P99 -> ceil(100 * 0.5 * 1.3) = 65
console.log(sizeBulkhead(100, 500));
 
// Cache: 1000 RPS, 10ms P99 -> ceil(1000 * 0.01 * 1.3) = 13
console.log(sizeBulkhead(1000, 10));

The formula prevents both under-sizing (which rejects legitimate requests) and over-sizing (which defeats the isolation purpose).

Testing Bulkhead Isolation

Verify that a slow service does not degrade a fast one:

async function testBulkheadIsolation() {
  const b = new SemaphoreBulkhead({
    'slow-service': 10,
    'fast-service': 50,
  });
 
  let fastCompleted = 0;
  const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
 
  // Saturate the slow service
  for (let i = 0; i < 15; i++) {
    b.execute('slow-service', () => sleep(5000)).catch(() => {});
  }
 
  // Give saturation a moment to take hold
  await sleep(100);
 
  // Fast service should still work despite slow service being at capacity
  await Promise.all(
    Array.from({ length: 20 }, () =>
      b.execute('fast-service', async () => {
        await sleep(10);
        fastCompleted++;
      })
    )
  );
 
  console.log(`Fast service completed: ${fastCompleted}/20`);
  // Should be 20 — bulkhead isolation worked
}

Run this test in your chaos/integration suite before every release.

Key Takeaways

The bulkhead pattern gives each downstream service its own concurrency budget so a slow dependency cannot consume all available capacity
Use Little's Law (throughput x latency x safety buffer) to size bulkheads rather than picking numbers arbitrarily
Semaphore-based bulkheads provide queue depth visibility that p-limit does not expose out of the box
Combining bulkheads with circuit breakers provides defense in depth: bulkheads limit concurrency, circuit breakers stop calling a failing service entirely
Tenant isolation bulkheads prevent noisy-neighbor problems: enterprise customers get dedicated concurrency slots, free tier users cannot impact them
Monitor bulkhead exhaustion rate (requests queued or rejected) as a leading indicator that a dependency is degraded
Test isolation under synthetic load: saturate the slow service, then verify the fast service completes requests normally
Set queue depth limits on bulkheads — when a queue grows beyond a threshold, reject new requests immediately (fail fast) rather than letting latency spike for waiting callers

Introduction#

Why Bulkheads Matter#

Concurrency Limiter with p-limit#

Semaphore-Based Bulkhead#

Bulkhead + Circuit Breaker Combination#

Tenant Isolation: Preventing Noisy Neighbors#

Sizing Bulkheads Correctly#

Testing Bulkhead Isolation#

Key Takeaways#