Config Drift Across Environments — When Prod Behaves Differently Than Staging
Sanjeev SharmaAdvertisement
Introduction
Config drift is a slow-motion disaster. An engineer SSH-es into production to fix an incident and tweaks a timeout value — never documented, never replicated to staging. A database pool size gets bumped during a traffic spike and stays bumped. Six months later, staging and production are effectively different systems, and "it works on staging" means nothing. This guide covers how to detect drift, prevent it, and make configuration a first-class engineering concern.
Common Sources of Config Drift
Drift rarely happens all at once. It accumulates from small, individually reasonable decisions:
Incident-driven tweaks: Someone increases the database pool size from 10 to 50 during a traffic spike. The incident is over. The change is never replicated to staging. Three months later, a memory leak that would have been caught in staging is hidden by the larger pool headroom in production.
Expired secrets: A third-party API key is rotated in production but nobody updates staging. Staging is now broken, but since nobody monitors staging health proactively, nobody notices for weeks.
Feature flag divergence: A new feature is enabled in staging for testing, then forgotten. The flag is never turned on in production. The feature never ships.
Infrastructure asymmetry: Production runs on 16GB instances; staging was set up years ago on 4GB. Memory-related bugs only surface in staging, making them appear more serious than they are.
Undocumented version differences: A third-party service config (Stripe webhook timeout, Redis connection settings) differs between environments with no documented reason.
Fix 1: Config as Code with Documented Differences
Every configuration value must live in version control. Differences between environments must be explicit and justified:
// config/base.js — shared defaults
const baseConfig = {
database: {
poolMin: 5,
poolMax: 50,
connectionTimeout: 30000,
idleTimeout: 600000,
},
redis: {
connectTimeout: 5000,
maxRetriesPerRequest: 3,
},
api: {
timeout: 30000,
retries: 3,
},
cache: {
ttl: 300,
},
};
// config/staging.js — intentional, documented differences only
const stagingConfig = {
...baseConfig,
database: {
...baseConfig.database,
poolMax: 20, // staging has smaller capacity — intentional, documented
},
};
// config/production.js
const productionConfig = { ...baseConfig };
module.exports = { baseConfig, stagingConfig, productionConfig };If staging differs from production in any undocumented way, that is a bug. The PR diff is the audit trail.
Fix 2: Parity Check in CI
Automated tooling should catch undocumented config divergence before it reaches production:
// scripts/check-config-parity.js
const { stagingConfig, productionConfig } = require('../config');
// These are the only allowed differences — explicitly documented
const ALLOWED_DIFFERENCES = [
'database.poolMax', // staging has smaller instances
'database.poolMin', // staging keeps fewer idle connections
];
function deepDiff(obj1, obj2, path = '') {
const diffs = [];
const allKeys = new Set([...Object.keys(obj1), ...Object.keys(obj2)]);
for (const key of allKeys) {
const currentPath = path ? `${path}.${key}` : key;
const val1 = obj1[key];
const val2 = obj2[key];
if (typeof val1 === 'object' && typeof val2 === 'object') {
diffs.push(...deepDiff(val1, val2, currentPath));
} else if (val1 !== val2) {
diffs.push({ path: currentPath, staging: val2, production: val1 });
}
}
return diffs;
}
const diffs = deepDiff(productionConfig, stagingConfig);
const unexpected = diffs.filter(d => !ALLOWED_DIFFERENCES.includes(d.path));
if (unexpected.length > 0) {
console.error('Config parity check FAILED. Undocumented differences:');
unexpected.forEach(d => {
console.error(` ${d.path}: prod=${d.production}, staging=${d.staging}`);
});
process.exit(1);
}
console.log('Config parity check passed.');Run this check in CI on every PR that touches configuration files. A failing parity check blocks the merge.
Fix 3: Infrastructure as Code for All Environments
Every infrastructure difference between environments must be declared in code, not applied through the cloud console:
# terraform/modules/app/variables.tf
variable "environment" {}
variable "db_pool_max" {}
variable "api_timeout_ms" {}
variable "cache_ttl_sec" {}
# terraform/environments/staging/main.tf
module "app" {
source = "../../modules/app"
environment = "staging"
instance_type = "t3.large" # intentionally smaller than prod
db_pool_max = 20
api_timeout_ms = 30000
cache_ttl_sec = 300
}
# terraform/environments/production/main.tf
module "app" {
source = "../../modules/app"
environment = "production"
instance_type = "t3.xlarge"
db_pool_max = 50
api_timeout_ms = 30000
cache_ttl_sec = 300
}Any manual change to a cloud resource that is not reflected in Terraform is drift by definition. Use terraform plan in CI to surface unapproved changes.
Fix 4: Secret Health Checks
Validate that secrets are valid in all environments on every deployment and on a daily schedule:
// scripts/check-secret-health.js
async function checkSecretHealth() {
const checks = [
{
name: 'DATABASE_URL',
test: async () => {
const { Pool } = require('pg');
const pool = new Pool({ connectionString: process.env.DATABASE_URL });
await pool.query('SELECT 1');
await pool.end();
},
},
{
name: 'STRIPE_SECRET_KEY',
test: async () => {
const Stripe = require('stripe');
const stripe = Stripe(process.env.STRIPE_SECRET_KEY);
await stripe.balance.retrieve();
},
},
{
name: 'REDIS_URL',
test: async () => {
const { createClient } = require('redis');
const client = createClient({ url: process.env.REDIS_URL });
await client.connect();
await client.ping();
await client.disconnect();
},
},
];
let allPassed = true;
for (const check of checks) {
try {
await check.test();
console.log(`[OK] ${check.name}`);
} catch (err) {
console.error(`[FAIL] ${check.name}: ${err.message}`);
allPassed = false;
}
}
if (!allPassed) process.exit(1);
}
checkSecretHealth();A broken secret in staging caught on deploy day is a 10-minute fix. The same broken secret discovered during an on-call incident is a 2AM outage.
Fix 5: Production Config Change Log
Every production configuration change must be documented with reason, owner, and a follow-up ticket:
# Production Config Change Log
## 2026-05-01
- Changed DB_POOL_MAX: 10 -> 50 (reason: traffic spike during marketing launch)
- TODO: Replicate to staging (ticket: ENG-2341)
- Rotated STRIPE_SECRET_KEY (new key expires 2027-05-01)
## 2026-04-12
- Enabled FEATURE_NEW_CHECKOUT: false -> true (100% rollout)
- TODO: Remove flag from codebase (ticket: ENG-2290)Make this change log a required artifact for production config changes. It becomes the audit trail that explains why staging and production differ and who is responsible for closing the gap.
Key Takeaways
- Config drift accumulates through incident-driven tweaks, undocumented manual changes, and forgotten temporary fixes — each harmless alone, dangerous in aggregate.
- Every configuration value should live in version control; differences between environments must be explicit, documented, and the minimal set necessary.
- An automated CI parity check that compares staging and production configs prevents undocumented drift from sneaking through code review.
- Infrastructure as Code (Terraform, Pulumi) must cover all environments; any manual cloud console change that is not reflected in IaC is drift.
- Secret health checks run on every deployment catch expired or invalid credentials in staging before they become production incidents.
- A production config change log with required reason and follow-up ticket fields creates accountability and an audit trail.
- The minimum bar: staging and production must share the same config schema; differences in values must be justified and documented in the codebase.
- Treat configuration changes with the same review rigor as code changes — they cause the same class of production incidents.
Advertisement
