Knowing When Architecture Is Overkill — The Senior Engineer's Restraint Problem

How over-engineering happens:

1. Conference-driven development
   → Senior engineer attended Kafka meetup
   → "We should be using Kafka for this"
   → Problem: you send 100 emails/day, not 100M events/day

2. Resume-driven development
   → "This will look good on my portfolio"
   → Pattern chosen for its sophistication, not its fit
   → Team learns complex system for a simple problem

3. Scale paranoia
   → "What if we need to handle 100x traffic?"
   → "We should build for that now"
   → Cost: 6x complexity for a problem that may never arrive

4. Pattern cargo-culting
   → Netflix uses microservices → we should use microservices
   → Netflix has 200 engineers per service
   → You have 5 engineers and 8 services

5. Architect's ego
   → Simple solution seems "below" senior engineers
   → Complex architecture demonstrates expertise
   → Result: system the junior engineers can't maintain

// YAGNI: You Ain't Gonna Need It
// Applied to architecture: don't add complexity for problems you don't have

// Scenario: build a notification system

// ❌ Over-engineered for a startup:
// - Kafka for event streaming
// - Separate notification service
// - CQRS + event sourcing for notification history
// - GraphQL subscriptions for real-time
// - Kubernetes for deployment
// Complexity: 6 weeks to build, hard to debug, 3 engineers to maintain

// ✅ Right for the stage:
async function sendNotification(userId: string, notification: Notification): Promise<void> {
  // Store in database
  await db.query(`
    INSERT INTO notifications (user_id, type, content, read, created_at)
    VALUES ($1, $2, $3, false, NOW())
  `, [userId, notification.type, JSON.stringify(notification.content)])

  // Send real-time via WebSocket if user is online
  const socket = connectedUsers.get(userId)
  if (socket) {
    socket.emit('notification', notification)
  }

  // Send email if not online and important
  if (notification.priority === 'high' && !socket) {
    await emailQueue.add({ userId, notification })
  }
}

// Complexity: 30 lines, works for 100k users, easy to debug
// When you need to scale this: you'll know exactly what to change

// The right time to add Kafka: when this function is a bottleneck
// (measurable in production) AND the team is ready to operate Kafka

// Every architectural decision adds complexity
// Complexity has a cost: harder to debug, harder to hire for, more operational burden
// The question is: does the benefit justify the cost?

interface ArchitectureDecision {
  pattern: string
  complexity: number  // 1-10 scale
  benefit: string
  alternativeComplexity: number  // Simpler alternative
  problemItSolves: string
  currentlyHaveThisProblem: boolean
  whenWouldMakeSense: string
}

const architectureComparisons: ArchitectureDecision[] = [
  {
    pattern: 'Microservices',
    complexity: 8,
    benefit: 'Independent deployment, team autonomy, targeted scaling',
    alternativeComplexity: 3,
    problemItSolves: 'Large team coordination, disparate scaling requirements',
    currentlyHaveThisProblem: false,  // 5 engineers, uniform load
    whenWouldMakeSense: 'When you have 3+ teams working on independent domains AND can\'t deploy independently',
  },
  {
    pattern: 'Kafka for events',
    complexity: 7,
    benefit: 'High-throughput event streaming, replay, fan-out',
    alternativeComplexity: 2,
    problemItSolves: 'Event processing at millions of events/second, multiple consumers',
    currentlyHaveThisProblem: false,  // 1000 events/day
    whenWouldMakeSense: 'When you\'ve hit the limits of Redis or SQS for your event volume',
  },
  {
    pattern: 'Event Sourcing',
    complexity: 9,
    benefit: 'Complete audit trail, time travel, rebuild projections',
    alternativeComplexity: 3,
    problemItSolves: 'Regulatory audit requirements, complex state reconstruction',
    currentlyHaveThisProblem: false,  // CRUD app
    whenWouldMakeSense: 'Financial systems with regulatory audit requirements, or when replay is a core feature',
  },
]

// Rule: if currentlyHaveThisProblem is false, don't add the pattern
// Exception: if whenWouldMakeSense is 6 months away and the migration is painful later

Questions to ask when reviewing an architecture proposal:

1. "What problem does this solve that we have RIGHT NOW?"
   → If the answer is "at scale" — how far are we from that scale?
   → Is the migration path from simple to complex well-understood?

2. "What's the simplest thing that could work?"
   → Have we eliminated every unnecessary component?
   → Would a junior engineer understand this in 30 minutes?

3. "Who will be on-call for this at 3 AM?"
   → Does the team have the operational expertise?
   → What's the failure mode and how do you debug it?

4. "How do we test this?"
   → Complex architectures are often harder to test
   → If testing requires a distributed system locally: probably too complex

5. "What happens when we get this wrong?"
   → What's the rollback strategy?
   → Can we migrate from this to something simpler if needed?

Red flags in proposals:
→ Multiple new technologies introduced simultaneously
→ "This is how Netflix/Google/Airbnb does it"
   (they have 100x your scale and 20x your engineering headcount)
→ No clear articulation of the current problem it solves
→ Estimated implementation time > 3 sprints for a non-critical path
→ Requires hiring a specialist to operate

// The best architecture for an uncertain future: the one that's easiest to change
// Not the one that handles every possible future requirement

// Design principle: don't lock yourself in to one pattern early
// Use simple, well-understood patterns that you can replace

// ✅ Abstraction that makes change easy:
interface NotificationSender {
  send(userId: string, notification: Notification): Promise<void>
}

// Start with the simplest implementation:
class InProcessNotificationSender implements NotificationSender {
  async send(userId: string, notification: Notification): Promise<void> {
    await sendNotificationDirectly(userId, notification)
  }
}

// When you need to scale: swap the implementation, keep the interface
class KafkaNotificationSender implements NotificationSender {
  async send(userId: string, notification: Notification): Promise<void> {
    await kafka.produce('notifications', { userId, notification })
  }
}

// The interface boundary means: you can migrate when you need to
// The migration cost is low because the interface was designed first

// This is the key insight:
// You don't need to build Kafka-based notifications today
// You need to make it easy to switch to Kafka when you need it
// Good abstractions do this without requiring Kafka today

"Boring technology" in the Dan McKinley sense:
→ Well-understood, battle-tested, widely deployed
→ Your team knows how to debug it at 3 AM
→ Good documentation, mature tooling, easy to hire for
→ Failure modes are well-known

Boring vs Exciting technology:
Boring: PostgreSQL, Redis, SQS, nginx, Node.js
Exciting: CockroachDB, Kafka, GraphQL Federation, Service Mesh, WebAssembly

Rules for introducing exciting technology:
1. You've exhausted what boring technology can do for this problem
2. You have clear evidence (data) that boring technology won't work
3. You have at least one engineer who deeply understands the exciting technology
4. You have 20% overhead budget for the learning curve and operational complexity
5. You have a fallback if the exciting technology doesn't work out

"New technology" budget for a team of 5:
→ Introduce maximum 1 new major technology per quarter
→ Team can deeply learn 1 new thing at a time
→ More than that: surface knowledge, no operational mastery