Distributed Locking Patterns — Redis, Redlock, and Fencing Tokens

Introduction

Why This Matters

Distributed locks solve a specific problem: preventing two processes from executing the same critical section simultaneously across different servers. They are seductive — they feel safe — but they are a common source of subtle production bugs. A network partition can leave a lock held forever. A GC pause can cause a lock to expire while the holder is still executing. This post covers the tools, the failure modes, and when you actually need a lock vs. when a database transaction or idempotency key does the job better.

Redis SET NX PX: Simple Distributed Lock

The simplest distributed lock uses Redis SET with NX (only set if not exists) and PX (expiration in milliseconds).

// src/locks/redis-lock.ts
import { Redis } from 'ioredis';
import { randomUUID } from 'crypto';
 
const redis = new Redis(process.env.REDIS_URL!);
 
interface Lock {
  key: string;
  token: string;
  release: () => Promise<void>;
}
 
// Atomic release using Lua script — critical to avoid releasing another holder's lock
const RELEASE_SCRIPT = `
  if redis.call("get", KEYS[1]) == ARGV[1] then
    return redis.call("del", KEYS[1])
  else
    return 0
  end
`;
 
export async function acquireLock(
  resource: string,
  ttlMs: number = 5000
): Promise<Lock | null> {
  const key = `lock:${resource}`;
  const token = randomUUID(); // unique per acquisition attempt
 
  const result = await redis.set(key, token, 'NX', 'PX', ttlMs);
 
  if (result !== 'OK') {
    return null; // lock held by someone else
  }
 
  return {
    key,
    token,
    release: async () => {
      // Lua script ensures we only delete our own lock
      await redis.eval(RELEASE_SCRIPT, 1, key, token);
    },
  };
}
 
// Usage with automatic release
export async function withLock<T>(
  resource: string,
  ttlMs: number,
  fn: () => Promise<T>
): Promise<T | null> {
  const lock = await acquireLock(resource, ttlMs);
  if (!lock) return null;
 
  try {
    return await fn();
  } finally {
    await lock.release();
  }
}
 
// Example: prevent double-processing a payment
async function processPayment(paymentId: string) {
  const result = await withLock(`payment:${paymentId}`, 10_000, async () => {
    const payment = await db.findPayment(paymentId);
    if (payment.status !== 'pending') return { skipped: true };
 
    await chargeCard(payment);
    await db.updatePayment(paymentId, { status: 'completed' });
    return { processed: true };
  });
 
  if (result === null) {
    console.log(`Payment ${paymentId} is being processed by another instance`);
  }
}

Redlock: Multi-Node Redis Locking

Single-node Redis locks fail if the Redis primary goes down between lock acquisition and replication. Redlock acquires locks on N independent Redis nodes (typically 5) and requires a majority quorum.

// src/locks/redlock.ts
import { Redis } from 'ioredis';
import { randomUUID } from 'crypto';
 
// Redlock algorithm — acquire lock on majority of N independent Redis nodes
class Redlock {
  private nodes: Redis[];
  private quorum: number;
  private clockDriftFactor: number = 0.01;
 
  constructor(nodes: Redis[]) {
    this.nodes = nodes;
    this.quorum = Math.floor(nodes.length / 2) + 1; // majority
  }
 
  async acquire(resource: string, ttlMs: number): Promise<RedlockLock | null> {
    const token = randomUUID();
    const start = Date.now();
 
    let acquired = 0;
 
    await Promise.all(
      this.nodes.map(async (node) => {
        try {
          const result = await node.set(
            `lock:${resource}`,
            token,
            'NX',
            'PX',
            ttlMs
          );
          if (result === 'OK') acquired++;
        } catch {
          // Node unavailable — skip, count as not acquired
        }
      })
    );
 
    const elapsed = Date.now() - start;
    const drift = Math.floor(this.clockDriftFactor * ttlMs) + 2;
    const validityTime = ttlMs - elapsed - drift;
 
    if (acquired >= this.quorum && validityTime > 0) {
      return { token, resource, validityTime, nodes: this.nodes };
    }
 
    // Failed to acquire quorum — release any partial locks
    await this.releaseOnNodes(this.nodes, resource, token);
    return null;
  }
 
  private async releaseOnNodes(nodes: Redis[], resource: string, token: string) {
    const RELEASE_SCRIPT = `
      if redis.call("get", KEYS[1]) == ARGV[1] then
        return redis.call("del", KEYS[1])
      else
        return 0
      end
    `;
    await Promise.allSettled(
      nodes.map(node => node.eval(RELEASE_SCRIPT, 1, `lock:${resource}`, token))
    );
  }
}
 
interface RedlockLock {
  token: string;
  resource: string;
  validityTime: number;
  nodes: Redis[];
}
 
// Initialize with 5 independent Redis instances
const redlock = new Redlock([
  new Redis({ host: 'redis-1', port: 6379 }),
  new Redis({ host: 'redis-2', port: 6379 }),
  new Redis({ host: 'redis-3', port: 6379 }),
  new Redis({ host: 'redis-4', port: 6379 }),
  new Redis({ host: 'redis-5', port: 6379 }),
]);

PostgreSQL Advisory Locks

For workloads already using PostgreSQL, advisory locks are simpler and safer than Redis — they participate in database transactions and automatically release on disconnect.

// src/locks/pg-advisory.ts
import { Pool } from 'pg';
 
const pool = new Pool({ connectionString: process.env.DATABASE_URL });
 
// pg_advisory_lock uses a 64-bit integer key
// Use a consistent hash to convert string resource names to int
function resourceToKey(resource: string): bigint {
  let hash = BigInt(5381);
  for (const char of resource) {
    hash = ((hash << BigInt(5)) + hash) ^ BigInt(char.charCodeAt(0));
  }
  return hash & BigInt('0x7FFFFFFFFFFFFFFF'); // keep positive
}
 
// Session-level advisory lock (auto-released on disconnect)
export async function withAdvisoryLock<T>(
  resource: string,
  fn: () => Promise<T>
): Promise<T> {
  const client = await pool.connect();
  const key = resourceToKey(resource);
 
  try {
    // pg_try_advisory_lock returns false if lock is held
    const { rows } = await client.query(
      'SELECT pg_try_advisory_lock($1) AS acquired',
      [key]
    );
 
    if (!rows[0].acquired) {
      throw new Error(`Could not acquire lock for resource: ${resource}`);
    }
 
    try {
      return await fn();
    } finally {
      await client.query('SELECT pg_advisory_unlock($1)', [key]);
    }
  } finally {
    client.release();
  }
}
 
// Transaction-level advisory lock (auto-released on commit/rollback)
export async function withTransactionalAdvisoryLock<T>(
  resource: string,
  fn: (client: PoolClient) => Promise<T>
): Promise<T> {
  const client = await pool.connect();
  const key = resourceToKey(resource);
 
  try {
    await client.query('BEGIN');
 
    const { rows } = await client.query(
      'SELECT pg_try_advisory_xact_lock($1) AS acquired',
      [key]
    );
 
    if (!rows[0].acquired) {
      await client.query('ROLLBACK');
      throw new Error(`Lock contention on: ${resource}`);
    }
 
    try {
      const result = await fn(client);
      await client.query('COMMIT');
      return result;
    } catch (err) {
      await client.query('ROLLBACK');
      throw err;
    }
  } finally {
    client.release();
  }
}

Fencing Tokens: Handling Lock Expiry Correctly

The hardest problem with distributed locks: a lock can expire while the holder is still executing (due to GC pause, network delay, etc.). Fencing tokens solve this by attaching a monotonically increasing version to each lock and rejecting writes from stale lock holders.

// src/locks/fencing.ts
// Fencing tokens prevent stale lock holders from corrupting shared state
 
// The resource server (e.g., a database or storage service)
// rejects writes with a token lower than the last seen token
 
class FencedResource {
  private lastSeenToken: number = 0;
 
  async write(data: unknown, fenceToken: number): Promise<void> {
    if (fenceToken <= this.lastSeenToken) {
      throw new Error(
        `Stale write rejected: token ${fenceToken} &lt;= last seen ${this.lastSeenToken}`
      );
    }
    this.lastSeenToken = fenceToken;
    // Perform the actual write
    await this.persistData(data);
  }
 
  private async persistData(data: unknown): Promise<void> {
    // actual write logic
  }
}
 
// Lock server that issues monotonically increasing tokens
class FencedLockServer {
  private tokenCounter: number = 0;
 
  async acquireLock(resource: string): Promise<FencedLock> {
    const token = ++this.tokenCounter;
    return {
      resource,
      token,
      expiresAt: Date.now() + 5000,
    };
  }
}
 
interface FencedLock {
  resource: string;
  token: number;
  expiresAt: number;
}
 
// Practical implementation using Redis INCR for monotonic tokens
async function acquireFencedLock(
  redis: Redis,
  resource: string,
  ttlMs: number
): Promise<{ token: number; release: () => Promise<void> } | null> {
  const lockKey = `lock:${resource}`;
  const tokenKey = `lock:token:${resource}`;
 
  // Atomically: set lock + increment token
  const script = `
    local set = redis.call('SET', KEYS[1], '1', 'NX', 'PX', ARGV[1])
    if set then
      local token = redis.call('INCR', KEYS[2])
      return token
    else
      return false
    end
  `;
 
  const result = await redis.eval(script, 2, lockKey, tokenKey, ttlMs);
  if (!result) return null;
 
  const token = Number(result);
  return {
    token,
    release: async () => {
      await redis.del(lockKey);
    },
  };
}

When to Avoid Distributed Locks

Distributed locks are often the wrong tool. These alternatives are simpler and more reliable.

// Alternative 1: Database transactions with SELECT FOR UPDATE
// For in-database resources, this is almost always better than a Redis lock
async function processOrderWithLock(orderId: string) {
  const client = await pool.connect();
  try {
    await client.query('BEGIN');
 
    // SELECT FOR UPDATE acquires a row-level lock within the transaction
    const { rows } = await client.query(
      'SELECT * FROM orders WHERE id = $1 FOR UPDATE SKIP LOCKED',
      [orderId]
    );
 
    if (rows.length === 0) {
      await client.query('ROLLBACK');
      return; // row locked by another process, skip
    }
 
    await processOrder(client, rows[0]);
    await client.query('COMMIT');
  } catch (err) {
    await client.query('ROLLBACK');
    throw err;
  } finally {
    client.release();
  }
}
 
// Alternative 2: Idempotency keys — often eliminates the need for locks
async function processPaymentIdempotent(
  paymentId: string,
  idempotencyKey: string
) {
  // Try to insert idempotency record atomically
  const result = await db.query(`
    INSERT INTO idempotency_records (key, payment_id, status)
    VALUES ($1, $2, 'processing')
    ON CONFLICT (key) DO NOTHING
    RETURNING *
  `, [idempotencyKey, paymentId]);
 
  if (result.rowCount === 0) {
    // Another request already claimed this idempotency key
    const existing = await db.query(
      'SELECT * FROM idempotency_records WHERE key = $1',
      [idempotencyKey]
    );
    return existing.rows[0].response;
  }
 
  // We own this request — process it
  const response = await chargeCard(paymentId);
  await db.query(
    'UPDATE idempotency_records SET status=$1, response=$2 WHERE key=$3',
    ['completed', JSON.stringify(response), idempotencyKey]
  );
  return response;
}

Common Mistakes

Not using a unique token per lock acquisition — without this, you cannot safely release only your own lock
Not using Lua scripts for atomic get-check-delete — non-atomic release can delete another holder's lock
Setting TTL too short relative to expected operation duration — lock expires while holder is still working
Using Redlock for resources that require strict mutual exclusion — Redlock provides probabilistic, not absolute, guarantees
Not planning for lock acquisition failure — callers must handle null return gracefully
Replacing database transactions with Redis locks for database resources — transactions are simpler and safer

Best Practices

Use SELECT FOR UPDATE within a transaction for database resources — no Redis needed
Use idempotency keys to prevent duplicate processing without requiring locks
Always use unique per-acquisition tokens and atomic Lua release scripts
Set TTL based on the 99th percentile operation time, not the average
Implement retry with exponential backoff for lock acquisition
Monitor lock acquisition failure rates as a leading indicator of contention
Prefer PostgreSQL advisory locks when already on Postgres — they are connection-scoped and auto-release

Key Takeaways

Distributed locks require unique per-acquisition tokens and atomic Lua scripts to release safely — bare DEL is always wrong
Redis single-node locks are sufficient for most use cases; Redlock is needed only when a single Redis node cannot tolerate failover
PostgreSQL advisory locks are simpler and safer than Redis for workloads already on Postgres
Fencing tokens protect against stale lock holders that execute after their lock expires due to GC pauses or network delays
SELECT FOR UPDATE SKIP LOCKED in a transaction is usually better than a Redis lock for database row contention
Idempotency keys eliminate the need for distributed locks in payment and message-processing scenarios
Lock TTL should be based on the 99th percentile operation duration plus network overhead, not the average
Always handle lock acquisition failure gracefully — callers must expect and retry on contention

Introduction#

Why This Matters#

Redis SET NX PX: Simple Distributed Lock#

Redlock: Multi-Node Redis Locking#

PostgreSQL Advisory Locks#

Fencing Tokens: Handling Lock Expiry Correctly#

When to Avoid Distributed Locks#

Common Mistakes#

Best Practices#

Key Takeaways#

Related reading