At 11:48 p.m. PDT on October 19, 2025, customers began reporting increased DynamoDB API error rates in AWS’s US-East-1 region. A latent race condition in DynamoDB’s automated DNS management system produced an empty DNS record for the regional endpoint. Two automated components, a DNS Planner and a DNS Enactor, fell out of sync: a delayed Enactor completed a stale update just as a second Enactor triggered cleanup, wiping all IP addresses from the Route 53 record. Within minutes, any service in US-East-1 that depended on DynamoDB lost its ability to resolve the endpoint.
The cascading effects hit unevenly. DynamoDB API errors were the first symptom. EC2 instance launches began failing because the internal workflow manager depended on DynamoDB-backed leases, though existing EC2 instances continued running. Lambda invocations, Network Load Balancers, ECS, EKS, and Fargate experienced distinct impact periods as each service encountered its own DynamoDB dependency. The incident spanned approximately 15 hours and affected more than 100 AWS services before full recovery.
Snapchat experienced extended login failures. Signal went dark. Disney+, Fortnite, Coinbase, Duolingo, Pinterest, and Alexa followed. For the companies behind those names, the outage quickly moved beyond a technology problem into revenue loss, customer trust damage, and executive attention.
Services whose architectures did not depend on a single provider’s availability in a single region were in a different position. Organizations running workloads across independent failure domains, whether multiple regions or multiple providers, continued serving traffic because no single regional endpoint failure could take down their entire stack.
A race condition cascading through tightly coupled dependencies is a known architectural risk. What made this one severe was the depth of those dependencies within a single region. High Availability, Disaster Recovery, and Business Continuity are often treated as checkboxes. In practice, they are the decisions that determine whether your business absorbs a moment like October 20th or becomes part of the headline.
What resilience actually means in cloud architecture
High Availability, Disaster Recovery, and Business Continuity address different failure modes and operate on different timescales. Conflating them leads to systems that are expensive to run but brittle under real pressure. Each layer covers a distinct class of failure, and your architecture needs all three working independently before any of them can work together.
High availability absorbs failure in real time
High Availability keeps systems running while something is actively going wrong. A node dies, a zone degrades, a service becomes unresponsive. HA architecture absorbs those events without users noticing. It treats failure as a normal operating condition rather than an emergency.
A common HA mistake is confusing redundancy with resilience. Adding a standby node in the same availability zone gives you two points of failure in the same blast radius. HA requires distributing workloads across independent failure domains and selecting a failover strategy that matches your availability requirements and operational capacity.
Failover strategies exist on a spectrum. A pilot light environment keeps minimal infrastructure running in a secondary region, enough to bootstrap a full deployment when needed. A warm standby maintains a scaled-down but functional copy of the production environment. A hot standby runs a near-identical environment ready for immediate traffic. Active-active deployment takes this further: every location carries live traffic simultaneously, so failure is absorbed as a load distribution event rather than a failover event.
Each approach involves trade-offs. Active-active minimizes failover time because all environments are already serving traffic, but it introduces operational complexity in areas like data consistency, deployment coordination, and cost. A properly designed and regularly tested warm or hot standby can still provide strong availability for many workloads. The choice depends on your availability targets, operational maturity, and the nature of the workloads involved.
During the October 20th AWS outage, services distributed across independent regions or providers continued operating because they were not wholly dependent on US-East-1. Organizations that had designed and tested their failover strategies, whether active-active or a well-maintained standby, were better positioned to absorb the disruption.
Disaster recovery: when HA is not enough
Some failures go beyond what HA can mask. Silent data corruption propagates across replicas before anyone notices. Ransomware encrypts your data and your most recent backups. Disaster Recovery exists for these scenarios, and its quality is measured by two numbers: how much data you can afford to lose (RPO, or Recovery Point Objective) and how long you can afford to be down (RTO, or Recovery Time Objective).
The architecture decisions that determine those numbers are made long before any incident. Continuous replication keeps RPO tight by mirroring writes across locations in near real time, rather than relying on periodic snapshots alone. Immutable, versioned snapshots make ransomware recovery viable because without them, you risk restoring a clean backup of already-compromised data. Recovery paths need to be tested under realistic conditions, not just documented.
The threat environment has also shifted. Ransomware campaigns have grown in scale and sophistication, and ransomware-aware recovery has moved from specialized preparation to a baseline requirement for any organization running production workloads in the cloud.
Most organizations discover their real RPO and RTO during an incident. The delta between the numbers in the DR plan and the numbers on the incident timeline is where businesses get hurt.
Business continuity keeps the organization running
Business Continuity is broader than infrastructure uptime. It asks whether the organization can still function financially, operationally, and legally during a prolonged disruption. A company can have excellent HA and DR and still fail at BC if its payment systems, customer communications, or compliance obligations depend on infrastructure that the same resilience model does not cover.
BC also extends beyond architecture. It includes organizational coordination, manual operating procedures, supplier dependencies, customer communication plans, and regulatory processes. A multi-cloud or provider-independent architecture is an infrastructure strategy that supports BC, but it does not constitute a complete BC plan on its own.
The architecture principle that supports BC is eliminating provider-level single points of failure. Distributing workloads across independent providers, combined with global load balancing that routes users to the nearest operational environment, means a systemic failure at one provider is less likely to become a systemic failure for your business.
The compliance environment is also changing. The EU Cyber Resilience Act (CRA), which entered into force on December 10, 2024, establishes cybersecurity requirements for products with digital elements placed on the EU market. Manufacturers must begin reporting actively exploited vulnerabilities and severe security incidents from September 11, 2026, with full product cybersecurity requirements applying from December 11, 2027. The CRA requires technical documentation, conformity assessment, CE marking, and lifecycle vulnerability management. The CRA is centered on products with digital elements rather than cloud services as a category. However, where remote data processing constitutes an integral function of the product, it may fall within scope, and applicability depends on the specific product and service model.
In December 2021, the municipality of Kalix, Sweden, was hit by a large-scale ransomware attack that severely disrupted its operations. Most servers were encrypted, and the full restoration of business systems took approximately three weeks. What kept essential public services running was not rapid technical recovery but organizational coordination: the municipality activated a central crisis management team that met twice daily, elderly care reverted to paper-based processes, salary payments were manually processed with external consultants, and experienced staff maintained critical services through analog workarounds. The case illustrates that Business Continuity depends as much on people, processes, and organizational coordination as it does on infrastructure resilience.
That principle scales. The organizations that fared best on October 20th were the ones whose architectures and operational procedures had already assumed the failure would happen.
The resilience architecture scorecard
Use this as a self-assessment. For each capability, ask honestly whether your stack has it today, not in the roadmap, not partially, but actually in production and tested.
Cloud disaster recovery solutions range from managed snapshot-and-replication services to multi-region failover platforms. The meaningful difference between basic backup software and production-grade DR is whether your recovery path is tested, repeatable, aligned to defined RPO and RTO targets, appropriately automated, and operationally executable under real incident conditions. A mature recovery process may still include controlled human approval steps.
High availability
(the failure you absorb before anyone notices)
| Capability | What it means |
|---|---|
| Active-active or tested standby deployment | All locations carry live traffic simultaneously (active-active), or a well-tested warm/hot standby can absorb failover within your SLO targets. The strategy matches your availability requirements and operational capacity. |
| No single point of failure in the critical path | Every component (compute, storage, DNS, and load balancers) has been audited and has a redundant counterpart in a different failure domain. |
| Automated self-healing with no human dependency | Node failure triggers automatic replacement and traffic rerouting. Recovery does not wait for an engineer. |
| Multi-region traffic distribution | A full regional failure results in automatic rerouting to another region. No manual intervention required. |
| Defined and tested availability SLOs | Uptime targets are architecturally enforced and tested, not aspirational numbers on a slide. |
Disaster recovery
(the failure HA could not absorb)
| Capability | What it means |
|---|---|
| Continuous replication with a defined RPO | Data is mirrored in near real time. Your RPO is measured under load, not assumed from vendor documentation. |
| Immutable, versioned snapshots | Immutable, versioned snapshots provide recovery points that reduce the risk of backup tampering. Successful recovery depends on retention policies, integrity validation, and tested restore procedures. |
| Tested RTO, not estimated | Recovery time has been validated through an actual DR exercise in the last 12 months, not calculated theoretically. |
| Ransomware-aware recovery path | Before restoring, integrity is verified against a known-clean baseline. The restore process assumes recent backups may be compromised. |
Business continuity
(keeping the organization operational)
| Capability | What it means |
|---|---|
| Multi-cloud or provider-independent architecture | Critical workloads are distributed across multiple providers so a systemic failure at a single provider is less likely to take the business offline. This is the infrastructure component of BC, not a complete BC plan. |
| Organizational continuity planning | Manual operating procedures, customer communication plans, supplier dependency mapping, and staff coordination are documented and rehearsed. The Kalix case showed that organizational readiness kept services running when infrastructure was unavailable. |
| Rehearsed continuity procedures | Roles, escalation paths, manual communication plans, and approval steps are documented and regularly exercised. The organization can sustain critical operations even when normal infrastructure and automated workflows are unavailable. |
| Documented resilience evidence | Architecture diagrams, DR exercise logs, and recovery outcomes exist and are current. Auditors and insurers can verify resilience on request, and cyber insurers are increasingly requiring it as a condition of coverage. The EU Cyber Resilience Act establishes related obligations for manufacturers of products with digital elements from September 2026 (reporting) and December 2027 (full requirements). |
If you can check every item in the HA column, your architecture was likely in a better position to absorb a failure like October 20th. Checking DR and BC as well means your organization treats outages as a normal operating condition rather than an emergency.
How Control Plane addresses these capabilities
Every capability in the scorecard above represents an architectural decision that takes time, expertise, and ongoing maintenance to get right. Active-active deployments need to be designed from the ground up, because retrofitting them onto an existing architecture is expensive and error-prone. Immutable snapshots need to be isolated from the credentials that ransomware would target first. Multi-cloud failover needs to be tested under realistic conditions, not just documented in a runbook.
Most engineering teams build these capabilities incrementally, under pressure, after something has already gone wrong. Control Plane, founded by Doron Grinstein, the former Chief Software Architect of VMware’s Cloud Services Platform, is a cloud virtualization platform that was built with several of these capabilities as defaults.
During the October 20th AWS outage, which spanned approximately 15 hours and affected more than 100 AWS services, not a single Control Plane customer experienced downtime. The architecture handled the incident automatically, without emergency scaling or manual runbook execution.
Twenty percent of Control Plane customers had AWS US-East-1 locations within their Global Virtual Cloud (GVC), the declared set of cloud provider locations where workloads can be served. Teams define location references in the GVC’s placement spec (spec.staticPlacement.locationLinks), listing the provider regions where workloads in that GVC are eligible to run. Once the platform detected the outage, failover was automatic. Customer-facing services remained available as workloads were routed to healthy locations and were active and running again within 10 seconds.
What Control Plane provides:
HA is the baseline. Control Plane supports both latency-based traffic distribution and priority-based failover across GVC locations. Locations on the lowest routing tier are eligible for latency-based traffic routing, while higher-tier locations remain failover targets that receive traffic only when lower-tier locations become unavailable. This means not every configured location is necessarily carrying live traffic at all times; some locations serve as failover capacity until needed. The October 20th scenario, a cascading failure through tightly coupled dependencies in a single region, is what this architecture is designed to handle.
Control Plane provides DR building blocks including multi-location workload placement and scheduled snapshots for supported volume types. Snapshots can be scheduled and retained for a configurable duration. Snapshot support applies to traditional volumes, not shared filesystems, and snapshots are not necessarily immutable or isolated from compromised credentials by default. Traditional volumes are not replicated across locations as workload replicas are; cross-location replication for stateful workloads must be implemented at the application level.
Multi-cloud deployment is an architectural property of the platform. Control Plane runs natively across AWS, Azure, and GCP. Traffic routing follows the GVC’s routing-tier and health-check settings. For GVCs that configure a workload-level alias via spec.aliasWorkloadLink, location failover is based on workload readiness rather than relying only on the default ingress TCP health check. Universal Cloud Identity lets workloads assume cloud-native identities (AWS IAM roles, GCP service accounts, and Azure app registrations or managed identities, depending on the integration) at runtime, using temporary credentials without storing long-lived provider credentials in application code or deployment pipelines. Because workloads do not store or distribute long-lived provider credentials, this simplifies identity handling across multi-cloud deployments. When workloads are deployed across providers and healthy routing targets are configured, traffic can shift to locations on another provider during a regional disruption, reducing the potential blast radius.
What remains the customer’s responsibility:
Control Plane provides the infrastructure primitives for multi-location deployment, health-aware routing, workload identity, and scheduled snapshots. However, several critical resilience capabilities remain the customer’s responsibility: defining and meeting application-level RPO and RTO targets, replicating state across locations for stateful workloads, maintaining and verifying known-clean restore points, building ransomware-isolated backup strategies, testing DR exercises under realistic conditions, maintaining business continuity procedures (including the organizational and process-level elements described in the BC section above), and producing documentation for audit or insurance purposes.
Frequently Asked Questions
High Availability keeps systems running during active, ongoing failures by distributing workloads across independent failure domains so individual node or zone failures are absorbed without user impact. Disaster Recovery addresses failures that HA cannot mask (such as data corruption or ransomware) by defining how much data loss is acceptable (RPO) and how quickly systems must be restored (RTO), then building tested recovery paths to meet those targets.
RPO stands for Recovery Point Objective. It defines the maximum amount of data loss your organization can tolerate, measured in time. An RPO of one hour means you can afford to lose up to one hour of data. Continuous replication keeps RPO tight by mirroring writes across locations in near real time, rather than relying on periodic snapshots that may leave hours of data unprotected.
RTO stands for Recovery Time Objective. It defines how long your systems can be offline before the business impact becomes unacceptable. Most organizations discover their real RTO during an incident, when the delta between the number in the DR plan and the actual recovery timeline becomes visible. RTO should be validated through tested DR exercises, not calculated theoretically from vendor documentation.
Services whose architectures distributed workloads across independent failure domains, whether multiple regions or multiple providers, kept serving traffic because they were not wholly dependent on a single region’s availability. When US-East-1 became impaired, traffic routed to healthy regions or providers where those services had active deployments. The outcome depended on the specific architecture, failover strategy, and how well it had been tested.
Active-active deployment means every location in your infrastructure carries live traffic simultaneously across multiple zones or regions. When a location fails, the remaining locations absorb the load without a switchover event. Active-passive strategies (including pilot light, warm standby, and hot standby) keep a secondary environment at varying levels of readiness. Active-active minimizes failover time but introduces operational complexity around data consistency and deployment coordination. A properly designed and regularly tested warm or hot standby can still meet demanding availability requirements for many workloads.
Ransomware-aware disaster recovery means your recovery process verifies data integrity against a known-clean baseline before restoring, because recent backups may themselves be compromised. It also requires immutable, versioned snapshots that ransomware cannot encrypt or delete, giving you multiple restore points that exist outside the blast radius of a production compromise.
The EU Cyber Resilience Act (CRA), which entered into force on December 10, 2024, applies to manufacturers and distributors of products with digital elements placed on the EU market. It requires reporting of actively exploited vulnerabilities and severe security incidents from September 11, 2026, with full product cybersecurity requirements applying from December 11, 2027. The CRA mandates technical documentation, conformity assessment, CE marking, and lifecycle vulnerability management. The CRA is centered on products with digital elements rather than cloud services as a category. However, where remote data processing constitutes an integral function of the product, it may fall within scope, and applicability depends on the specific product and service model.
Multi-cloud architecture distributes workloads across independent providers (AWS, Azure, and GCP) so a systemic failure at a single provider is less likely to propagate to your business. Combined with global load balancing that routes users to the nearest operational environment automatically, multi-cloud addresses the infrastructure component of business continuity. A complete BC plan also requires organizational readiness: manual operating procedures, customer communication plans, supplier dependency management, staff coordination, and rehearsed recovery processes.
Designing for what has already happened
Outages are a normal operating condition of modern, distributed systems. The question is not whether one will happen, but whether your architecture and your organization are designed to absorb it when it does.
Resilience used to be framed as insurance: money spent to protect against something unlikely. That framing made it easy to defer and easy to underinvest. What October 20th demonstrated, again, is that the failure is not hypothetical. It has already happened, and the organizations that stayed online had made their architectural decisions before the incident, not during it.
If your architecture depends on a single provider’s availability in a single region, the scorecard above is a reasonable place to start evaluating what would need to change.
Want to run the scorecard against your own architecture with a Control Plane engineer? We can walk through where the gaps are and what it would take to close them.