Why Spaghetti Code Beat Clean Architecture

Eric Roby

23,106 views • 3 days ago

Video Summary

A senior engineer's focus on clean code and elegant patterns led to a production failure when replacing a legacy system. Despite a mid-level engineer's repeated concerns about scalability, caching, and observability, these were dismissed in favor of architectural purity. The new system crashed during beta with minimal load, forcing a rollback to the older, messier but robust application, highlighting that code quality is only one dimension of a successful production system. The legacy system, though architecturally unappealing, had critical infrastructure, database design, and observability elements that allowed it to handle significant load, a lesson starkly learned when the new system failed to perform.

An interesting fact is that the legacy system, despite its messy code, achieved a 91% cache hit rate and handled 85,000 concurrent users with 99.4% uptime during Black Friday, numbers the new, "cleaner" system couldn't even approach during beta testing.

Short Highlights

A senior engineer's clean code failed in production, performing worse than a legacy system at one-tenth the scale.
The legacy system handled 85,000 concurrent users and 77,000 orders on Black Friday with 99.4% uptime and a 91% cache hit rate.
The new system, with only basic logging and 23% cache hit rate, crashed during beta with approximately 1,000 users due to connection pool exhaustion and slow queries.
Four pillars of a production system were identified: code quality, infrastructure and scaling strategy, observability, and operational culture.
The importance of deep infrastructure, robust observability (beyond basic logs), and a culture that values resilience over mere elegance was underscored.

Key Details

The Cost of Beautiful Code vs. Production Reality [00:00]

Beautiful code, even with 8 years of experience and solid test coverage, proved worthless when it crashed in production at a fraction of the scale of the legacy system it was meant to replace.
A mid-level engineer's concerns about scalability, raised three times, were ignored by management and the senior engineer.
The new system was built with a focus on architectural patterns like repository patterns and event sourcing, adhering to "clean code" principles.
The senior engineer dismissed the legacy system as messy, emphasizing that "Clean code is what matters."

"This is how I learned that beautiful code is completely worthless."

Legacy System's Hidden Strength [01:54]

The legacy PHP system, despite a file with 4,847 lines and no classes, SQL concatenation, and "spaghetti code," was currently running properly.
Digging deeper revealed a hidden infrastructure: a three-tier caching system with a 91% cache rate, 47 database indexes on the orders table, partitioning by month, and three read replicas.
Custom client metrics (240) tracked every checkout step, along with error rates by payment method, latency percentiles, and real-time dashboards.
During Black Friday 2017, the legacy system handled 85,000 concurrent users, 77,000 orders, with an average response time of 246 milliseconds and 99.4% uptime.
The legacy team lead explained that "Code quality is just one dimension," and that "Infrastructure, database design, observability, those matter, too."

"Code's just one dimension."

Ignoring Red Flags: The New System's Approach [04:10]

Over two months, the new system was built with very clean code, but infrastructure was delegated to a DevOps engineer new to scaling e-commerce.
Caching in staging for the new system hit only 23%, a stark contrast to the legacy app's 91%.
Observability was limited to basic logs, lacking custom metrics and query visibility.
Production readiness concerns, including low cache hit rates and lack of query monitoring or sustained load testing, were documented and sent to the senior engineer.
The response was to avoid "overengineering the first MVP" and address issues if they appeared in beta, despite beta being only a week before Black Friday.

"We don't want to overengineer the first MVP. We'll address issues if they show up in beta."

The Beta Launch Failure [06:16]

On beta launch day, with the new checkout enabled for 5% of users, basic metrics (CPU, memory, error rate) looked good.
However, critical metrics like connection pool utilization, cache hit rate, and query performance were not being monitored.
Within 30 minutes, customer support reported slow checkouts, and eventually, a "database connection error: connection pool exhausted" occurred.
The rollback to the legacy application took 47 minutes, a painful experience as it wasn't even the high-traffic Black Friday.
The new system, tested with only 1,000 users, was already on the verge of crashing, far from the 85,000 concurrent users handled by the legacy system.

"And here's what we couldn't see. All the issues because we weren't adding in the metrics behind the dashboard showing the CPU and the memory."

The Four Pillars of Production Systems [07:55]

Pillar 1: Code Quality: While clean code is maintainable and easier to onboard, it doesn't guarantee scalability or building the right thing. The new system's repository pattern, following clean code principles, led to the N+1 query trap, where simple operations triggered numerous database calls. The legacy system, though ugly, used batched queries to achieve efficiency.
Pillar 2: Infrastructure and Scaling Strategy: The legacy system had robust infrastructure (multi-tier caching, database partitioning, read replicas, connection pooling). The new system lacked this, with minimal caching and a single database instance. The assumption that "clean architecture won't need all that infrastructure" proved dangerously false, as infrastructure is the foundation for handling load.
Pillar 3: Observability: The legacy system had 247 metrics, providing deep insights. The new system had only three basic metrics, leaving the team blind to the root cause of failures. Essential metrics for query performance, resource utilization, and business logic were missing, preventing proactive issue resolution.
Pillar 4: Operational Culture: Concerns were raised three times and dismissed with phrases like "don't overengineer" and "trust me, bro." Confidence, especially from experienced individuals, overshadowed data and logical concerns. The team's culture rewarded elegance over resilience, treating production readiness as optional rather than a requirement.