AWS re:Invent 2025 - Architecture lessons: Three failures and how to prevent them (DEV341)

Architect Perspective on Failure [00:17]

• The core idea is that perfect architectures often falter when confronted with real-world scenarios and implementation challenges.
• Many clients fail because they focus on prevention without adequate testing, leading to unexpected issues when reality strikes.
• The "perfect" architecture can be flawed when it's implemented but the problems it might encounter are not tested.

The problem the real problem of perfect architectors when we draw implement but we don't actually the problem.

Resilience Failure in E-commerce [02:12]

• A Brazilian e-commerce customer, hosting multiple major mobile companies, experienced significant issues due to inadequate resilience in their infrastructure.
• Their architecture, though seemingly standard with load balancers, web tiers, and databases, was not robust enough for peak seasonal demands like Black Friday.
• Instances were unevenly distributed across availability zones, with a primary database in a different zone than most instances, increasing latency and risking downtime.
• The lack of caching for high-traffic assets like mobile phone images and a manual scaling process that required support intervention for new instances exacerbated performance issues and prolonged recovery times.
• During peak events, like the launch of the iPhone 14, their websites went down, leading to lost revenue for the companies.

So we know AWS has a region component and in the region we have availability zones AWS say ok region availabilisecs of time in latency when go instance RS database add communication the web application database in this scenario.

Security & Operational Excellence Breakdown [14:45]

• A critical enterprise bank customer with mission-critical workloads had security hub enabled, but foundational security best practices, including timely updates for Kubernetes, were neglected.
• Security alerts for necessary version updates were consistently ignored or suppressed, leading to a vulnerable state.
• The situation escalated to a point where an emergency rollback was necessary, requiring the involvement of AWS support to downgrade the cluster, bring the application back online, and mitigate the financial impact.
• The failure stemmed from a lack of a proper DevOps cycle, broken silos, and insufficient communication, leading to the suppression of alerts and a failure to plan and execute updates.

Security alert was alerting version update support soon right so the whole CSPM was running very good.

Cost Inefficiency and Multi-Account Strategies [21:10]

• A significant failure point identified is in multi-account and general cost management, particularly with unoptimized instance sizing and a lack of operational policies like shutdown schedules for non-production environments.
• This oversight resulted in substantial financial waste, with one example showing $60,000 in monthly expenditure that could be saved annually.
• The problem highlights a lack of clear policies for shutting down unused resources, leading to continuous operational costs.

The point here is numbers right for 60000 per months and the annual saving wasy.

Final Thoughts and Lessons Learned [23:08]

• Architectural decisions about availability are critical and involve trade-offs, not just ticking boxes.
• Addressing issues like manual scaling, inadequate caching, distributed instance deployment, and proactive security updates are paramount.
• Consolidating resources, such as reducing over 3,000 load balancers to a more manageable number, can lead to significant cost savings and improved efficiency.
• Breaking down silos and fostering better communication between development and operations teams is essential for a smooth DevOps cycle.

Yeah, pretty good. Bring the lessons for this architectury.