How to Survive an AWS Outage: A Playbook for Provider-Level Failure

Surviving an AWS outage depends on which parts of your system still work when an AWS control plane does not. Workloads spread across availability zones and regions still share IAM, DNS, and the AWS APIs that launch, scale, and reconfigure them, and those shared services are where a provider-level failure spreads. Teams that come through with little impact have three things in place: a map of every AWS dependency their production and recovery paths share, a failover that uses only data-plane operations, and a rehearsed plan for the first hour.

The October 2025 US-EAST-1 outage is the clearest recent example of the mechanics. A fault in one DNS record for DynamoDB spread to EC2, load balancers, Lambda, and IAM sign-in, and some services took more than a day to recover fully. The sections below trace that cascade, then turn it into a dependency map, a design standard, a first-hour runbook, and a test program.

What happened in the October 2025 AWS US-EAST-1 outage?

AWS's post-event summary traces the outage to "a latent race condition in the DynamoDB DNS management system that resulted in an incorrect empty DNS record." Two components that apply DNS updates for DynamoDB collided. One was working through delayed updates while the other applied a newer plan and cleaned up old ones, and the cleanup deleted every IP address for the regional endpoint, dynamodb.us-east-1.amazonaws.com. From 11:48 PM PDT on October 19, nothing that resolved that name could reach DynamoDB in the region, including AWS's own internal services. Engineers identified DNS as the source at 12:38 AM, restored the records by 2:25 AM, and customer connections recovered at 2:40 AM as cached DNS entries expired.

The DynamoDB outage lasted under three hours. The rest of the event lasted until the afternoon, because EC2 depended on DynamoDB in a way customers could not see. The EC2 subsystem that manages the physical servers behind instances, which AWS calls DropletWorkflow Manager, holds a lease on each server and checks its state in DynamoDB. While DynamoDB was unreachable, those leases slowly timed out. When DynamoDB came back, the subsystem tried to re-establish thousands of leases at once and fell into what AWS calls "congestive collapse": the backlog of work grew faster than it could be completed. Engineers throttled incoming work and restarted hosts at 4:14 AM, and new instance launches began succeeding at 5:28 AM.

The new instances then exposed a third failure. They entered service before their network configuration had finished propagating, so Network Load Balancer health checks failed on them. The failing checks triggered automatic DNS failover between availability zones, which removed healthy capacity. Engineers disabled automatic health-check failover at 9:36 AM, EC2 APIs returned to normal at 1:50 PM, and automatic failover was re-enabled at 2:09 PM.

Each of those three failures reached other services:

Service	Impact window (PDT)	What failed
IAM console sign-in	Oct 19, 11:51 PM to Oct 20, 1:25 AM	IAM user sign-in failed, and root and federated sign-in were also affected, across all regions
AWS STS	Oct 19, 11:51 PM to Oct 20, 9:59 AM	Recovered at 1:19 AM, then errors returned from 8:31 AM when load balancer health checks failed
Lambda	Oct 19, 11:51 PM to Oct 20, 2:15 PM	Function create and update failures, delayed SQS and Kinesis processing, then throttling after instance terminations
Amazon Connect	Oct 19, 11:56 PM to Oct 20, 1:20 PM	Recovered at 1:20 AM, then failed again at 7:04 AM: inbound calls got busy tones and agents could not sign in
ECS, EKS, Fargate	Oct 19, 11:45 PM to Oct 20, 2:20 PM	Container launch failures and delayed cluster scaling
Redshift	Oct 19, 11:47 PM to Oct 21, 4:05 AM	Cluster replacement blocked by EC2 launch failures; customers in every region could not use IAM user credentials

Two lessons carry through the rest of this playbook. A regional failure reached every region through IAM sign-in and Redshift's use of IAM in US-EAST-1. Recovery also came in waves: Amazon Connect and STS recovered once DynamoDB returned, then failed again hours later when the load balancer problem began. The first sign of recovery marked the end of one failure, and the event continued for hours after it.

Why multi-AZ and multi-region designs still fail during an AWS outage

Availability zones and regions isolate the data plane, the part of AWS that serves running traffic: instances, load balancer routing, DNS answers, reads and writes to a table. The control plane, the part that creates, changes, and deletes those resources, is not always isolated the same way. AWS's fault isolation whitepaper lists the global services whose control plane runs in a single region while their data plane runs everywhere. The IAM, Organizations, Route 53 public DNS, CloudFront, and ACM-for-CloudFront control planes all run in US-EAST-1.

This split decides whether a recovery plan works. During an event in US-EAST-1, a workload in another region keeps serving traffic, but any recovery step that edits a Route 53 record, creates an IAM role, provisions a load balancer, or changes a CloudFront origin depends on the region that is failing. AWS's whitepaper names these as common anti-patterns: failing over by changing record weights, creating IAM resources during failover, and provisioning disaster recovery resources such as load balancers or databases after the event has started.

The customer's side of the arrangement follows from this. NIST's guidance on cloud ecosystem risk divides the work: the provider commits to the availability of its services, and the consumer is responsible for judging how much residual risk its own design carries. AWS publishes where its control planes live. Whether a team's recovery path runs through them is a design choice the team makes, often without noticing.

The same pattern appears at other providers, which is why it belongs in any provider-failure plan. In February 2025, DNS records for a Microsoft Entra ID endpoint were removed, and a five-minute DNS TTL made the authentication failure spread gradually as caches expired. In November 2025, an oversized configuration file spread across Cloudflare's network and failed products customers thought of as separate, including CDN traffic and Cloudflare Access sign-in. In each case, a component every copy of a workload depended on was the one that failed.

How to map the AWS dependencies your production and recovery paths share

Because shared control planes carry the risk, the map has to show them. The Financial Stability Board's third-party risk toolkit sets a useful standard: map every dependency needed to deliver each critical service through the whole supply chain, including SaaS providers and the dependencies of the recovery process itself. In practice this means one graph per critical customer journey, such as checkout or sign-in, showing which dependency sits under how many journeys.

For every dependency on the graph, record how it fails and how you would notice:

• The AWS service, region, account, and the team that owns the dependency.
• Whether it serves runtime traffic, authentication, deployment, or recovery.
• The signal that shows it failing, and who receives that signal.
• The fallback, and whether that fallback relies on the same region, IAM, DNS, or AWS API.

Then mark each dependency that appears on both a production path and a recovery path. Those are the entries that decide whether a region failure becomes a multi-hour outage. Replicated data in a second region helps only if the team can sign in, reach the failover control, and route traffic without touching the failed region.

The output for each shared dependency is a failure-path row a team can defend in review:

Trigger	First symptoms	Cascade	Fallback	Owner	Recovery time target	Evidence
The AWS event that starts the failure	What synthetic checks, dashboards, and users see first	Which services, sign-in paths, and recovery steps fail next	The alternative path and how it is switched on	Named team and individual	Target, and the last measured time	Test date, ticket, or postmortem link

The recovery time target is usually called RTO (recovery time objective): how long the service can be down. Its companion, RPO (recovery point objective), is how much recent data the business can afford to lose. Both come from the business, and the map shows whether the design can meet them.

Designing AWS failover that uses only the data plane

The map shows where recovery depends on a control plane. The design goal is static stability: running workloads keep serving, and failover completes, without any call to an API that creates or changes resources. AWS's disaster recovery whitepaper states the rule directly: "For maximum resiliency, you should use only data plane operations as part of your failover operation."

The whitepaper describes four recovery strategies, and the choice among them determines how much of the recovery path depends on a control plane:

Strategy	What runs in the recovery region	Control-plane work at failover
Backup and restore	Data backups only	Redeploy infrastructure and restore data, all control-plane operations
Pilot light	Replicated data and switched-off core infrastructure	Switch on and scale out application servers
Warm standby	A scaled-down, fully working copy	Scale up, usually through Auto Scaling
Multi-site active/active, or hot standby	Full production capacity	None for capacity; traffic routing only

Auto Scaling is itself a control-plane activity, so a warm standby that must scale during the event carries the dependency the design is meant to remove. AWS describes the trade-off: provision enough capacity to absorb initial traffic at failover, and treat anything that must be created during the event as a risk.

Four practices remove most control-plane dependencies from an AWS recovery path:

• Route traffic with data-plane controls. Route 53 health checks run on the data plane. Amazon Application Recovery Controller provides routing controls, switches built on those health checks, that move traffic without editing DNS records. Changing record weights or Global Accelerator traffic dials during an event is a control-plane operation.
• Pre-provision everything recovery needs. Load balancers, S3 buckets and their configuration, API Gateway endpoints, and IAM roles should exist in the recovery region before the event, because creating them depends on control planes in US-EAST-1.
• Make sign-in regional. Configure SDKs and the CLI to use regional STS endpoints; they default to US-EAST-1. Let IAM role trust policies accept SAML sign-in from more than one region, and keep break-glass users for the case where your identity provider is also affected.
• Store recovery data where the data plane can read it. If a failover script needs a value, such as the endpoints of your routing-control cluster, keep it in a parameter store, table, or bucket in more than one region instead of querying a control plane for it during the event.

Retry behavior belongs in the same design review. The EC2 lease subsystem collapsed when thousands of requests arrived at once after DynamoDB returned, and client applications do the same thing to their own dependencies when every instance retries at the same moment. Exponential backoff with jitter, which spreads retries out over time with some randomness, plus circuit breakers that stop calling a failing dependency, keep a recovering service from being pushed back down by the traffic waiting for it.

The first hour of an AWS outage

A tested design still needs a runbook that tells the on-call team what to do and in what order. The first hour decides whether the team fails over cleanly or spends the morning trying to work out which system is at fault.

Minutes 0 to 10: confirm whether the failure is AWS or you. The AWS Health Dashboard is an official source, and it can lag. During the February 2017 S3 outage, AWS reported that it "was unable to update the individual services' status on the AWS Service Health Dashboard" because of that dashboard's own dependency on S3; AWS has since run it across multiple regions. Detection that works during a provider failure comes from signals outside the failing region: synthetic checks run from another region or provider, error rates on calls to AWS APIs, DNS resolution failures, and sign-in errors. A burst of those across unrelated services at the same moment points to a shared dependency and away from your last deploy. Correlating signals across the stack is how a team tells the two apart in minutes.

Minutes 10 to 30: decide against pre-agreed criteria. Failing over costs something even when it works, because some data and availability are lost in the switch. AWS recommends caution with fully automatic regional failover for that reason, and suggests a manually started failover whose steps are automated. Write the decision criteria in advance: the error rate, the duration, and the named person who can approve the switch.

Minutes 30 to 60: fail over and communicate. Use the data-plane controls from the design section. Post updates on a status page hosted outside the failing provider, since a status page on the same infrastructure fails with it.

After the first hour: recover in stages. The October 2025 event showed that recovery comes in waves. Bring traffic back gradually, keep throttles in place until backlogs drain, and watch for a second failure in services that looked healthy an hour earlier.

Rehearsing provider-level failure with game days

Design without rehearsal produces fallbacks nobody has switched on. Rank dependencies by the number and business importance of the services that share them, and start testing at the top of the list. IAM, DNS, and the region that hosts global control planes usually rank first.

The scenarios that match recent incidents are a regional DynamoDB or DNS failure, loss of IAM sign-in, a failed configuration push to a CDN, and loss of the AWS APIs in one region. Each exercise should record:

• Time to detect and time to isolate the failing dependency.
• Time to switch on the fallback.
• The share of traffic that stayed serviceable.
• The number of manual steps.

The manual step count predicts how a real incident will go, because each manual step is a point where an operator may need access to a system that is down. Test the highest-ranked dependencies more than once a year, and after every architecture change that touches them.

An executive checklist for provider-level failure

Provider concentration belongs on the executive risk register alongside financial and security exposure. Keeping a high-concentration dependency is a decision about tolerable downtime, and a named executive should accept that risk explicitly.

• Name the most critical services and how long each can be down.
• Build the dependency map, covering runtime, identity, DNS, CDN, deployment, observability, and recovery.
• List every dependency shared by production and recovery, with an owner for each.
• Confirm failover uses only data-plane operations, and that recovery resources are pre-provisioned.
• Create and test break-glass access that works when the identity provider is down.
• Run observability and status communication from outside the region they report on.
• Rehearse the top scenarios at least annually, and the highest-ranked ones more often.
• Record what each incident taught: the causal chain, the decisions, and the fix that worked.

How NeuBird helps teams detect, route, and resolve a provider-level failure

The playbook above is the engineering team's work: the architecture, the pre-provisioned recovery path, the game days. NeuBird is the Agentic Reliability Center, one governed platform that unifies access to your telemetry and LLMs, records institutional operations memory, and audits every agentic action in production to build resilient systems. Its Production Ops Agent runs on that platform and helps at each stage of a provider-level failure.

Before the failure: find the gaps in the map. A dependency map is out of date the week after it is drawn. The Production Ops Agent watches for degradation before alert thresholds trip, detecting it 30 to 60 minutes early, and it surfaces blind spots such as unmonitored services, missing SLOs, and services with no clear owner. On AWS it reads CloudWatch, EKS, EC2, Lambda, and RDS signals where they live, alongside the rest of the 50+ tools a team already runs. See how it connects to AWS.

During the failure: answer "is it us or AWS?" and reach the right people. The first 10 minutes of the runbook depend on correlating errors across services that have no dashboard in common. The Production Ops Agent analyzes 15+ sources in parallel and finds root cause in under 5 minutes at 94% accuracy, with the causal chain shown, the same approach described in the guide to automated root cause analysis. It delivers that investigation in the foreground, in the Slack channel, Jira issue, or ServiceNow ticket where the owning team is already working, so the people named in the failure-path table get a completed investigation instead of another alert.

During the failure: stage the failover for a human to approve. Actions run under a Suggest, Recommend, Act policy. For production failover the agent works at Recommend: it stages the runbook step, such as the pre-agreed routing change, and an engineer approves it. Every action carries a named approver and sits in one audit trail.

After the failure: keep what the incident taught. Institutional operations memory records each investigation's conclusions, causal chain, and approved fixes, with zero telemetry storage: raw logs, metrics, and traces stay in the tools that own them. The next provider-level incident starts from the last dependency map and the fix that worked. Between incidents, the agent keeps tuning noisy alerts and identifying recurring patterns.

The platform itself has to survive the failure it is watching. NeuBird runs in your cloud, in your VPC, on-premises, or air-gapped, so it can be deployed outside the region or provider its workloads depend on and treated like any other entry in the dependency map.

Stage	What the team needs	What NeuBird does
Before	A current dependency map and early warning	Detects degradation 30 to 60 minutes early and surfaces unmonitored or unowned services
During, first 10 minutes	To know whether the fault is AWS or their own change	Correlates 15+ sources in parallel; root cause in under 5 minutes at 94% accuracy
During, failover	A fast, approved switch	Stages the runbook step at Recommend; an engineer approves; one audit trail
After	Lessons that survive staff changes	Records conclusions and approved fixes as memory, with zero telemetry storage