The incident

On 19 October 2025, DynamoDB in AWS us-east-1 began returning API errors. What appeared routine quickly evolved into something far more serious. A race condition in DynamoDB’s DNS automation produced an empty DNS record for the regional endpoint. In that moment, DynamoDB effectively disappeared from the network.

AWS restored the DNS records, but the damage had already cascaded. Essential AWS services depend on DynamoDB, and when it became unreachable, the platform’s internal control systems began to fail. EC2 workflow managers lost state. New instances launched without network configuration, or failed to launch at all. Load balancers removed healthy capacity. Lambda throttled invocations. ECS and EKS stalled.

This was not a zone issue, hardware failure, or capacity exhaustion. It was a control-plane outage, the part of AWS that customers do not see and cannot design around.

The resilience assumption

For more than a decade, hyperscalers have instructed enterprises and trained cloud architects to build for failure: distribute workloads across Availability Zones, use retries and queues, adopt managed services, automate scaling, and avoid single points of failure. Many of the affected workloads followed those patterns. They were redundant, fault tolerant, and aligned with best practice.

None of it mattered.

The architectural weakness

The Shared Responsibility Model implies a clean division: AWS owns the infrastructure, and customers own what they build on top of it. But there is a third layer the model never addresses: the control plane. DNS, identity, network propagation, scaling workflows, and orchestration pipelines are the mechanisms that make cloud behave like cloud. They are centralized, invisible, and completely outside customer control.

The cloud solved distributed compute, storage, and networking. It did not solve distributed control. The control plane remains a single logical dependency inside an otherwise decentralized system. When it fails, customers cannot recover, even if every server and region underneath is healthy.

The multi-cloud misconception

Multi-cloud is often proposed as the answer, but very few enterprises operate true active-active systems across providers. Most simply distribute workloads, which increases dependency, not redundancy. A system that uses AWS for transactions and Azure for analytics is not resilient to an AWS control-plane outage. It is just more complex.

Even organizations attempting real cross-cloud failover face the same structural limitation. Each provider still relies on its own centralized control systems. If those systems fail, the platform cannot orchestrate recovery, regardless of available capacity elsewhere.

The path forward

Real resilience will not come from more zones, more replicas, or more automated retries. It requires decentralizing the control plane itself. Identity, routing, service resolution, workload scheduling, and state propagation cannot remain locked inside a single provider’s orchestration pipeline. Workloads should be able to survive when the provider’s automation behaves incorrectly, not only when hardware disappears.

Enterprises did not fail to design for resilience. The outage proved that the part of the cloud designed to keep everything reliable can itself become the single point of failure, and when it does, customers have no control, visibility, or mitigation path.

Resilience was never truly shared. It was assumed.
Written by Oded NahumMore writing →