Knowing When Architecture Is Overkill — The Senior Engineer's Restraint Problem

How over-engineering happens:
 
1. Conference-driven development
   - Senior engineer attended Kafka meetup
   - "We should be using Kafka for this"
   - Problem: you send 100 emails/day, not 100M events/day
 
2. Resume-driven development
   - "This will look good on my portfolio"
   - Pattern chosen for sophistication, not fit
   - Team learns complex system for a simple problem
 
3. Scale paranoia
   - "What if we need to handle 100x traffic?"
   - "We should build for that now"
   - Cost: 6x complexity for a problem that may never arrive
 
4. Pattern cargo-culting
   - Netflix uses microservices, we should too
   - Netflix has 200 engineers per service
   - You have 5 engineers and 8 services
 
5. Architect ego
   - Simple solution seems below senior engineers
   - Complex architecture demonstrates expertise
   - Result: system the junior engineers cannot maintain

// Scenario: build a notification system for 500 users
 
// Overkill architecture:
// - Kafka for event streaming
// - Separate notification microservice
// - CQRS + event sourcing for notification history
// - GraphQL subscriptions for real-time
// Complexity: 6 weeks to build, hard to debug, 3 engineers to maintain
 
// Right for the stage:
async function sendNotification(userId, notification) {
  // Store in database
  await db.query(`
    INSERT INTO notifications (user_id, type, content, read, created_at)
    VALUES ($1, $2, $3, false, NOW())
  `, [userId, notification.type, JSON.stringify(notification.content)])
 
  // Send real-time via WebSocket if user is online
  const socket = connectedUsers.get(userId)
  if (socket) {
    socket.emit('notification', notification)
  }
 
  // Send email if not online and important
  if (notification.priority === 'high' && !socket) {
    await emailQueue.add({ userId, notification })
  }
}
 
// Complexity: 30 lines, works for 100k users, easy to debug
// When you need to scale this: you will know exactly what to change
// The right time to add Kafka: when this is a measured bottleneck
// AND the team is ready to operate Kafka

Pattern: Microservices
Complexity: 8/10
Benefit: Independent deployment, team autonomy, targeted scaling
Problem it solves: Large team coordination, disparate scaling requirements
Have this problem at 5 engineers? No.
When does it make sense? 3+ teams on independent domains who cannot deploy independently.
 
Pattern: Kafka for events
Complexity: 7/10
Benefit: High-throughput event streaming, replay, fan-out
Problem it solves: Event processing at millions of events/second
Have this problem at 1,000 events/day? No.
When does it make sense? When Redis or SQS are measured bottlenecks.
 
Pattern: Event Sourcing
Complexity: 9/10
Benefit: Complete audit trail, time travel, rebuild projections
Problem it solves: Regulatory audit requirements, complex state reconstruction
Have this problem on a CRUD app? No.
When does it make sense? Financial systems with regulatory requirements.

1. "What problem does this solve that we have RIGHT NOW?"
   - If the answer is "at scale" — how far are we from that scale?
   - Is the migration path from simple to complex well-understood?
 
2. "What is the simplest thing that could work?"
   - Have we eliminated every unnecessary component?
   - Would a junior engineer understand this in 30 minutes?
 
3. "Who will be on-call for this at 3 AM?"
   - Does the team have operational expertise?
   - What is the failure mode and how do you debug it?
 
4. "How do we test this?"
   - Complex architectures are often harder to test
   - If testing requires a full distributed system locally: too complex
 
5. "What happens when we get this wrong?"
   - What is the rollback strategy?
   - Can we migrate from this to something simpler if needed?
 
Red flags in proposals:
- Multiple new technologies introduced simultaneously
- "This is how Netflix/Google/Airbnb does it"
  (they have 100x your scale and 20x your headcount)
- No clear articulation of the current problem it solves
- Estimated implementation time greater than 3 sprints for a non-critical path
- Requires hiring a specialist to operate

// The interface enables future migration without requiring Kafka today
 
class NotificationSender {
  async send(userId, notification) {
    throw new Error('Not implemented')
  }
}
 
// Start with the simplest implementation
class InProcessNotificationSender extends NotificationSender {
  async send(userId, notification) {
    await sendNotificationDirectly(userId, notification)
  }
}
 
// When you need to scale: swap the implementation, keep the interface
class QueuedNotificationSender extends NotificationSender {
  async send(userId, notification) {
    await queue.add('notifications', { userId, notification })
  }
}
 
// The interface boundary means: you can migrate when you need to
// Migration cost is low because the boundary was designed first
// You do not need to build queue-based notifications today
// You need to make it easy to switch when you need it

Boring technology: PostgreSQL, Redis, SQS, nginx, Node.js
Exciting technology: CockroachDB, Kafka, GraphQL Federation, Service Mesh
 
Rules for introducing exciting technology:
1. You have exhausted what boring technology can do for this problem
2. You have clear data that boring technology will not work
3. At least one engineer deeply understands the exciting technology
4. You have 20% overhead budget for the learning curve
5. You have a fallback if the exciting technology does not work out
 
New technology budget for a team of 5:
- Introduce maximum 1 new major technology per quarter
- Team can deeply learn 1 new thing at a time
- More than that: surface knowledge, no operational mastery