Health Check Patterns — Liveness, Readiness, and Deep Dependency Checks

Introduction

Health checks drive container restarts, traffic routing, and on-call alerts. Shallow checks that only ask "is the process running?" miss real problems. Deep checks that query every dependency at probe time can cause cascading failures during a partial outage. The production-ready pattern sits in between: test what matters locally, fail gracefully when dependencies degrade, and cache results to prevent the health endpoint from becoming its own failure mode.

Liveness vs Readiness vs Startup Probes

Kubernetes uses three distinct probe types, each with different consequences when they fail.

Liveness answers: is this process still alive? A failing liveness probe triggers a container restart. Use it for detecting deadlocks, memory exhaustion, or stuck goroutines — situations where the process is running but cannot recover without a restart.

Readiness answers: can this instance handle traffic right now? A failing readiness probe removes the pod from the load balancer without restarting it. Use it for detecting dependency outages, warm-up periods, and temporary overload.

Startup answers: has the application finished initializing? Startup probes block liveness and readiness checks from firing during long initialization sequences. Without them, Kubernetes may restart a slow-starting pod before it is ready.

const express = require('express')
const app = express()
 
let isReady = false
let startupComplete = false
 
// Startup probe: blocks until initialization finishes
app.get('/startup', (req, res) => {
  if (!startupComplete) {
    return res.status(503).json({ status: 'initializing' })
  }
  res.status(200).json({ status: 'started' })
})
 
// Liveness probe: only fails for unrecoverable conditions
app.get('/healthz', (req, res) => {
  const mem = process.memoryUsage()
  const heapPercent = (mem.heapUsed / mem.heapTotal) * 100
 
  // Only fail liveness for situations that require a restart
  if (heapPercent > 95) {
    return res.status(500).json({ status: 'out-of-memory', heapPercent })
  }
  res.status(200).json({ status: 'alive' })
})
 
// Readiness probe: fails when the service cannot serve traffic
app.get('/ready', async (req, res) => {
  if (!isReady) {
    return res.status(503).json({ status: 'not-ready' })
  }
 
  const dbOk = await checkDatabase()
  if (!dbOk) {
    return res.status(503).json({ status: 'database-unavailable' })
  }
 
  res.status(200).json({ status: 'ready' })
})
 
async function initialize() {
  await connectDatabase()
  await warmCache()
  startupComplete = true
  isReady = true
}

The key distinction: liveness failures cause restarts (expensive), readiness failures cause traffic routing changes (cheap and reversible). Do not fail liveness for recoverable conditions like a slow database query.

Shallow vs Deep Health Checks

A common mistake is querying every dependency in the readiness probe. If three services each check Redis and Redis has a slow response, all three services simultaneously fail readiness — a cascading failure triggered by the health check itself.

// Dangerous: deep health check queries everything on every probe call
async function deepHealthCheck() {
  const results = await Promise.all([
    checkDatabase(),
    checkRedis(),
    checkKafka(),
    checkExternalPaymentAPI(),
    checkS3(),
  ])
  return results.every(r => r.ok)
}
 
// Better: critical-only readiness check with cached results
const cache = {
  result: null,
  timestamp: 0,
  ttl: 5000  // 5 second cache
}
 
async function cachedReadinessCheck() {
  const now = Date.now()
 
  if (cache.result && now - cache.timestamp < cache.ttl) {
    return cache.result
  }
 
  // Only check what is truly critical for serving traffic
  const [db, redis] = await Promise.all([
    checkDatabase(),
    checkRedis(),
  ])
 
  cache.result = { ok: db.ok && redis.ok, db, redis }
  cache.timestamp = now
  return cache.result
}

The cached approach reduces dependency query frequency from hundreds of probe calls per minute down to one check every five seconds. External APIs and object storage belong in a separate diagnostic endpoint, not the readiness probe.

Dependency Health Aggregation

A /health/detailed endpoint can report the status of all dependencies without affecting Kubernetes probe behavior:

async function getDependencyHealth() {
  const timeout = 2000  // 2 second max per dependency
 
  const checkWithTimeout = (name, checker) => {
    const start = Date.now()
    return Promise.race([
      checker().then(ok => ({ name, ok, latency: Date.now() - start })),
      new Promise((_, reject) =>
        setTimeout(() => reject(new Error('timeout')), timeout)
      ),
    ]).catch(err => ({
      name,
      ok: false,
      latency: Date.now() - start,
      error: err.message,
    }))
  }
 
  const results = await Promise.all([
    checkWithTimeout('database', checkDatabase),
    checkWithTimeout('redis', checkRedis),
    checkWithTimeout('kafka', checkKafka),
    checkWithTimeout('payment-api', checkPaymentAPI),
    checkWithTimeout('s3', checkS3),
  ])
 
  const criticalDeps = ['database', 'redis']
  const criticalFailed = results.filter(
    r => criticalDeps.includes(r.name) && !r.ok
  )
 
  return {
    ok: criticalFailed.length === 0,
    timestamp: new Date().toISOString(),
    dependencies: results,
    criticalFailed: criticalFailed.map(r => r.name),
  }
}
 
app.get('/health/detailed', async (req, res) => {
  const health = await getDependencyHealth()
  res.status(health.ok ? 200 : 503).json(health)
})

This endpoint is for humans and monitoring dashboards, not for Kubernetes probes. Alert on individual dependency failures here without affecting pod traffic routing.

Graceful Degradation Under Dependency Failure

When a non-critical dependency degrades, the service should continue serving requests with reduced functionality rather than failing completely:

async function getProductDetails(productId) {
  // Always fetch core product data from database
  const product = await db.products.findById(productId)
  if (!product) {
    throw new Error('Product not found')
  }
 
  // Attempt to enrich with recommendations — degrade gracefully if unavailable
  let recommendations = []
  try {
    recommendations = await Promise.race([
      recommendationService.getRelated(productId),
      new Promise((_, reject) =>
        setTimeout(() => reject(new Error('timeout')), 1000)
      ),
    ])
  } catch {
    // Recommendation service is down or slow — return product without them
  }
 
  // Attempt to fetch live reviews — degrade to cached if unavailable
  let reviews = product.cachedReviews || []
  try {
    reviews = await reviewService.getReviews(productId)
  } catch {
    // Use cached reviews baked into the product record
  }
 
  return { ...product, recommendations, reviews }
}

The core product data (critical) always loads. Recommendations and live reviews (non-critical) degrade to empty or cached values. The user sees a product page; they do not see an error.

Operational Signals in Health Responses

Health responses should communicate operational state beyond a binary up/down:

app.get('/health/operational', async (req, res) => {
  const mem = process.memoryUsage()
  const heapPercent = (mem.heapUsed / mem.heapTotal) * 100
  const avgLatency = await getAverageRequestLatency()
 
  let state = 'HEALTHY'
  const issues = []
 
  if (heapPercent > 80) {
    issues.push('High heap usage: ' + heapPercent.toFixed(1) + '%')
    state = 'DEGRADED'
  }
 
  if (avgLatency > 500) {
    issues.push('High latency: ' + avgLatency + 'ms average')
    state = 'DEGRADED'
  }
 
  if (heapPercent > 95) {
    state = 'FAILING'
  }
 
  res.status(state === 'FAILING' ? 500 : 200).json({
    state,
    issues,
    metrics: { heapPercent, avgLatency, uptime: process.uptime() },
    recommendation:
      state === 'FAILING' ? 'Restart immediately' :
      state === 'DEGRADED' ? 'Monitor closely' :
      'All systems nominal',
  })
})

Monitoring systems can alert on state: DEGRADED before the service actually fails, giving operations teams time to act proactively.

Key Takeaways

Liveness probes should only fail for unrecoverable conditions that require a restart — failing liveness for a slow database causes unnecessary restarts
Readiness probes control traffic routing and should fail when the service genuinely cannot serve requests, not for optional features
Startup probes prevent Kubernetes from prematurely restarting slow-initializing services — set failureThreshold * periodSeconds to exceed your worst-case startup time
Cache health check results for 5 seconds to prevent thundering herd on shared dependencies during a probe storm
Only include critical dependencies (database, cache) in readiness checks — external APIs and storage belong in diagnostic-only endpoints
Graceful degradation means returning partial data when non-critical dependencies fail, not returning errors
Set 2-second timeouts on all dependency checks within health probes — a slow dependency check blocks the probe and can trigger false failures
Track health check response times as a metric; a health endpoint with p99 latency above 100ms is a problem waiting to happen

Introduction#

Liveness vs Readiness vs Startup Probes#

Shallow vs Deep Health Checks#

Dependency Health Aggregation#

Graceful Degradation Under Dependency Failure#

Operational Signals in Health Responses#

Key Takeaways#

Related reading