A lost zone is the failure banks plan for. Google’s London outage tested five layers of that plan

The week

Early on Tuesday 20 October 2026, one of the three zones in Google Cloud’s London region lost power. A zone is a block of data center capacity with its own power, cooling and networking, built to fail independently of its neighbors. The London region is called europe-west2, and the zone that failed was europe-west2-b.

Google’s later incident reports set out the physical sequence. At 05:53 UTC the facility hosting the zone lost its primary utility feed, and the emergency generators did not start. Battery-backed uninterruptible power supplies carried the load for about seven minutes. When they ran out, at 06:00 UTC, the servers, network switches and local storage in the zone went dark. Automated monitoring alerted Google’s engineers four minutes later. A Google spokesperson told Reuters the disruption had occurred at a third-party facility.

The public picture took longer to form. Google’s first status notice appeared at about 08:35 UTC, some two and a half hours after the impact began. It reported elevated errors and intermittent network failures for multiple services in europe-west2, with no workaround. Within the hour Google suggested that customers move affected workloads to another zone or region if they could. The update that named a power event in europe-west2-b came at about 11:12 UTC. Google reported recovery for most workloads shortly after noon UTC. At about 14:03 UTC it said the problem had been mitigated for most customers, with a very small number still affected.

The Register, which covered the outage as it unfolded, listed affected Google services including Kubernetes Engine, Cloud Run, App Engine, Filestore, NetApp Volumes and the managed Kafka service. Google’s later reports listed dozens of products, from Compute Engine and Persistent Disk to Cloud SQL, AlloyDB, Memorystore, Pub/Sub, BigQuery and the Apigee API platform.

Financial services felt it most visibly. Revolut, the app-based financial firm, told customers it was experiencing a disruption affecting multiple services and that their money and data were safe. Reuters reported that logging in, card payments and cash withdrawals were disrupted. Euronews reported problems among users in the UK, Ireland, France, Spain and Japan. In the evening, Revolut attributed the disruption to the Google Cloud outage and said services had been restored. Reuters also named Lloyds and HSBC among the banks affected; neither responded to its requests for comment. City AM reported problems at Lloyds, HSBC and Barclays, and spikes in user complaints about Royal Mail, Rightmove and NHS England.

Regulators responded within a day. The Financial Conduct Authority said it was aware of an incident affecting services provided by Google and was engaging with Google and affected firms to understand the impact. According to City AM, it added that it expected firms to fix problems quickly and keep customers informed. The Bank of England declined to comment to Reuters. The European Banking Authority said it had been in contact with Google Cloud to understand the incident and its potential impact on EU financial entities. On the day of the outage, Dame Meg Hillier, chair of the House of Commons Treasury Committee, wrote to the Governor of the Bank of England and the chief executive of the FCA. She asked how they would use their powers in response, with an initial reply due by 4 November and a full one within three months.

Those powers were new. On 10 July 2026, HM Treasury designated four cloud providers as critical third parties to the UK financial sector, with effect from 13 July: Microsoft Ireland Operations Limited, Google Cloud EMEA Limited, Amazon Web Services EMEA SARL and Oracle Corporation UK Limited. The Bank of England, the Prudential Regulation Authority and the FCA oversee the services they supply to the sector. The Treasury’s announcement said financial firms remain responsible for managing the risks that arise from their third-party suppliers. In the EU, Reuters noted, Google Cloud was among 19 technology firms designated as critical providers to the financial sector the previous year. The outage fell a year to the day after a failure in Amazon Web Services’ Northern Virginia region disrupted financial apps on 20 October 2025.

The cloud outages that have hurt banks most in recent years came in shapes a bank’s own design could do little about. In June 2025 a faulty configuration change at Google reached every region at once, which pointed bank due diligence toward how providers release changes. In October 2025 a failure in Amazon Web Services’ Northern Virginia region stalled services that ran across several zones there, because the whole region was what broke. The useful responses then were dependency mapping, rehearsed degraded service and, for a few functions, a second region or provider.

The London outage of 20 October 2026 was a different kind of event. One zone lost power. The other two kept it, and Google found no impact outside the London region. Losing a zone is the failure cloud architecture is built to absorb. Google’s documentation tells customers to treat each zone as a single failure domain and to deploy across several zones for high availability. A banking service built to that standard should have slowed briefly and carried on. Revolut’s customers instead lost logins, card payments and cash withdrawals for hours, and customers of other UK banks reported problems too.

Our argument is that zone tolerance belongs to each layer of each service, and a service survives a lost zone only when every layer holds at the same moment. Google’s reports on the incident, published on 27 October and updated on 4 November, are unusually detailed about its own managed services. They show five layers, and they show each one failing somewhere on the day:

1. Placement: whether the service’s compute runs in more than one zone.
2. State: whether its data, and the replicas that vote on that data, survive the loss of any one zone.
3. Capacity: whether the surviving zones have room for the load when every other tenant is moving too.
4. Control: whether the tools used to move, scale and repair the service still work while a zone is down.
5. Decision: who declares the zone lost, on what evidence, and how quickly.

The layers are separate because they fail separately. A service can run in three zones and keep its database primary in one. It can keep replicas everywhere and have no room to run them. It can have room and no working tools to use it. It can have all four and still wait hours for someone to decide. We do not know which layer failed inside any particular bank on 20 October, and we have not seen any bank publish that. Google’s reports describe its own services, and those are what many banks build on.

We take the layers in that order, after a short account of how the zone itself lost power. We then look at how they differ by type of firm, from UK banks to US credit unions and broker-dealers. After that we set out the strongest case against spending on all of this, and describe what a credible zone-loss test should prove.

Three layers of power, reduced to one

A large data center is designed so that no single fault cuts its power. This one had two utility feeds, a plant of generators and banks of batteries to bridge the gap between a utility failure and generator start. Google’s 27 October report explains how that design came down to a single layer.

On 14 October, a quarterly generator load test turned up problems. To make corrective repairs, the facility opened the breaker on utility feed B and locked it out. That left the site in what Google called a temporary single-feed, or 1N, maintenance posture. The generators and batteries were still there to cover a failure of the remaining feed. No software release was involved, yet a change still set the conditions: a planned maintenance step that reduced redundancy, of a kind that software-focused change reviews do not usually reach.

On the night of 19 October, Pacific time, a cable fault outside the facility cut feed A. The generators should then have started on their own. They are started by programmable logic controllers, the industrial computers that watch the incoming supply and switch to backup when it fails. The controllers’ undervoltage sensing was wired on the line side of the open feed B breaker, upstream of the isolation. They kept seeing live voltage on a feed that was disconnected from the building, so they never issued a start command. The batteries carried the load as designed until they were exhausted.

Three features of this chain matter for a bank’s own architecture. The redundancy was real, and planned maintenance had reduced it, which is when most systems are least protected. The failure was in sensing: the system that decides whether to fail over read a signal that no longer meant what it was designed to mean. And the gap between the utility failure and the outage was seven minutes of battery, too short for any person to intervene.

Each has a counterpart inside a bank’s cloud estate. A database cluster rebuilding a replica runs, for a while, without the redundancy its diagram shows. So does a Kubernetes node pool upgraded zone by zone, or a firewall pair with one member out for patching. Health checks that test whether a process is running, but not whether it can reach its data, can stay green while customers fail. Automatic failover that has never been exercised in production is, like the generator controllers, a design that has been inspected and not proven.

Power came back to the data halls about 80 minutes after the zone went dark. Google began re-admitting traffic to the zone about three and a half hours after that. Restoring a zone safely takes time, because systems have to be brought back and checked before load returns. A bank service that depends on its zone coming back is exposed for the whole recovery, and the power cut is the shortest part of it. A service that survives the zone’s absence needs all five layers to hold.

Placement: where the compute runs

The first layer is the simplest to state. Some cloud resources are zonal: a virtual machine, a zonal disk, a zonal Kubernetes cluster. They live and die with their zone. Google’s 4 November report is blunt about what happened to them. Zonal Kubernetes clusters in europe-west2-b were completely unavailable. Single-cluster Bigtable instances in the zone were unavailable, which Google described as expected. Its reports reminded customers that zonal resources require them to provision their own cross-zone failover.

Regional resources fared better, though not cleanly. Regional Kubernetes clusters, whose nodes are spread across zones, lost their worker nodes in zone b, and Google reported a spike in unavailable pods across the region. Spreading a cluster across zones protects the cluster. Whether it protects a given application depends on where that application’s own replicas landed.

How single-zone placement gets into a bank

Few banks decide to run a critical service in one zone. Single-zone placement arrives by accumulation, in ways an architecture review rarely sees:

• Migrated servers. Virtual machines lifted from an on-premises data center often arrive one per function, because the old site had one of everything and relied on its own power and cooling for resilience.
• Appliance pairs. Firewalls, load balancers and other network appliances often run as active and passive pairs. If the active member is in the failed zone, survival depends on the appliance’s own failover, which few teams test.
• Supporting services. Bastion hosts, build runners, license servers, batch schedulers and file-transfer gateways for payment files tend to be built quickly and placed wherever was convenient.
• Cost decisions that outlived their context. A single-zone database tier chosen for a pilot carries into production when the pilot succeeds.
• Vendor components. Software that a vendor deploys into the bank’s own cloud project follows the vendor’s defaults, which the bank may never have checked.
• Scheduler drift. Kubernetes places pods where there is room. Without explicit spread rules, three replicas of a service can end up in one zone after a busy week of redeployments.

Finding it and governing it

The architecture diagram is the wrong place to look. Every major cloud keeps an inventory of resources and their locations. The useful exercise joins that inventory to the bank’s list of important business services and the components each one calls. It then counts the components on each critical path that would stop if one named zone lost power. Done for card authorization, login, payment submission and balance display, the result is usually a short list. Each item on it is a decision to make, and some components can reasonably stay zonal if losing them for a day costs little.

Ownership splits three ways. The platform team owns guardrails, such as policies that reject single-zone deployments for anything tagged as part of an important business service, and Kubernetes spread rules that place replicas across zones. Service owners own the exceptions and the reasons for them. The risk function records accepted exceptions with an expiry date, so they come back for review. The metric worth reporting is simple: the number of single-zone components on each critical path, and whether it is falling.

Spreading has costs. More instances run, cross-zone network traffic often carries a charge, and chatty components pay a latency price for every call that crosses a zone boundary. Those costs are why the exceptions exist. They are also known in advance, which cannot be said for the cost of the outage they protect against.

State: where the data lives and how replicas agree

Stateless tiers are easy to spread, because any copy can serve any request. Data is harder. A database has to decide which copy is current, keep the others in step and promote a new primary when the old one disappears. The second layer asks whether that machinery survives the loss of any one zone.

Three designs are common. A zonal database with backups can be restored elsewhere, usually in hours. A primary and standby in different zones can fail over in minutes. Their replication is either synchronous, where every write waits for the second zone, or asynchronous, where the standby may lag behind. Quorum systems keep three or five replicas and require a majority to agree before a write counts. Each design fails differently when a zone is lost, and Google’s report shows several of those failures.

Quorum is a placement question

A three-replica quorum survives the loss of one zone only if the replicas are in three different zones. In Google’s Apigee X API platform, a scheduling misconfiguration had placed two of the three replicas of a datastore in the zone that lost power. The datastore lost quorum. Apigee’s runtime in the region returned elevated server errors for 7 hours and 19 minutes, longer than the zone itself was without power.

API gateways like this are where open banking traffic, partner integrations and mobile app calls enter many institutions. The weakness lies in how replica placement is verified. A design document can specify three zones, and a scheduler can later move a replica for good local reasons. Placement has to be checked continuously against live configuration, with an alert when two replicas share a zone.

Failover that starts late or stalls

Managed databases with high-availability options mostly did what they promised, with a tail. Cloud SQL detected the failures and started automated failovers immediately for its regional high-availability instances. Those completed for instances without Regional Persistent Disk failures. AlloyDB regional primaries saw delays of up to 95 minutes before automated failovers began, and about 4 percent of those failovers stalled on Persistent Disk problems and needed manual recovery. Some Memorystore for Redis instances remained stuck in a repair state until Google fixed them by hand, within 24 hours. Bigtable instances configured with multiple clusters and high-availability routing failed over automatically.

Two conclusions follow. The configuration a customer chose mattered: single-cluster Bigtable in the failed zone went down, and multi-cluster Bigtable kept serving. And even a correct choice carried a tail of delay and manual work that a bank’s recovery plan has to cover. A runbook that assumes failover is automatic needs a branch for the failover that never completes. That branch should set out the steps and permissions for a manual promotion, and the team should have practiced it.

Where state hides in a banking service

The obvious data stores are ledgers and customer records. The ones that cause trouble in a zone loss are often smaller:

• Session stores and caches. If a cache holding login sessions is lost, every customer may be signed out at once. The rush to sign back in then lands on the identity service at the worst moment.
• Message queues. Payment instructions in flight may be held in a queue partition in the failed zone. The design has to say whether they are replayed, held or lost.
• Idempotency records. A payment service that retries after failover must recognize a request it already processed. If the records that make that possible are zonal, duplicates follow.
• Card authorization. Card networks and processors can approve transactions on an issuer’s behalf when the issuer is unreachable, within limits the issuer sets. Those limits are a zone-loss decision, made months before the zone fails.

For each data store on a critical path, the evidence worth holding is short. It covers the replica zones taken from live configuration, the high-availability mode, and the date and measured result of the last forced failover. If replication is asynchronous, it also covers the data lost in that test. Database administrators can produce most of it. Service owners need to know what it means for their recovery point.

Capacity: room in the surviving zones

Arithmetic sets the third layer. A service spread evenly across three zones that loses one must carry its whole load on the remaining two. Each surviving zone then takes half the load instead of a third, an increase of 50 percent. Either every zone runs at no more than about two-thirds of its capacity at peak, or the service adds capacity during the outage.

Many cloud designs choose the second option, because idle headroom is paid for every day. Autoscaling adds instances when load rises. The weakness is timing. On 20 October, every affected tenant in the region tried to do the same thing at once.

Google’s 4 November report described instantaneous demand surges on the neighboring zones, a and c, which contributed to impact for some customers running workloads in those unaffected zones. Its 27 October report said some failovers were unsuccessful because of capacity constraints and difficulty replicating the underlying storage. During recovery Google prioritized existing workloads and warned that customers might not yet be able to create new ones. Regional Kubernetes clusters saw scaling delays. A critical BigQuery job that needed to scale up during the failover could not, and Google identified that as the cause of about five hours of elevated latency and errors.

For a bank, capacity that has to be created during the incident is capacity it cannot count on. Three options exist, and most firms will mix them:

• Pre-provisioned headroom. Run critical services with enough spare capacity in each zone to absorb a lost neighbor. It costs money every day and works on the day it is needed.
• Reservations. Clouds let customers reserve capacity in a named zone. Reserved capacity is generally billed whether it is used or not, but it means the instances can be started. Starting them still needs a working control plane.
• Planned load shedding. Decide in advance which functions get the reduced capacity. Card authorization, balances and payments come first. Analytics, marketing content, statement generation and non-urgent batch jobs wait.

Second-order effects

Load does not arrive politely. Mobile apps whose requests fail tend to retry, and many thousands of devices retrying at once can multiply load on the surviving zones. Client code needs exponential backoff with random jitter, so retries spread out instead of arriving together. Shared vendors face the same squeeze: a fraud-screening or identity service hosted in the same region competes for the same capacity. Scheduled work makes it worse when the outage overlaps a batch window or a busy payroll morning.

Retry storms also distort the evidence. A surviving zone that looks overloaded may be absorbing retries more than new demand, and adding capacity to it treats the symptom. Rate limits at the edge, applied per customer and per device, keep a recovering service from being flattened by its own clients.

The metric to track is the share of peak load each critical service can carry with any one zone removed and no new instances created. Measure it with a planned zone removal at a busy hour that is short of the peak. The answer turns headroom from an engineering preference into a cost the service owner can weigh against an impact tolerance.

Control: the tools that move a workload

Cloud systems have two planes. The data plane serves traffic: it runs the instances, moves the packets and answers the queries. The control plane changes things: it creates instances, attaches disks, updates load balancers and deploys code. Most failover designs assume the control plane will be there to carry them out.

On 20 October it was impaired where it was needed. Google reported control-plane dependencies across zonal boundaries: services in zones a and c made calls to components in zone b, those calls timed out, and some regional APIs returned server errors. Google did not finish returning traffic to its regional services and compute management systems until about five and a half hours after the zone went dark. Persistent Disk problems lay behind the stalled AlloyDB failovers and the delayed Cloud SQL ones.

A pattern Google has seen before

In October 2024, an electrical arc flash in a power distribution unit took down part of zone europe-west3-c in Frankfurt and affected its cooling. Customers in all Frankfurt zones then saw increased latency and errors when creating instances and attaching Persistent Disk volumes. Google traced that to problems in access-control services for multiple zones, caused by a misconfiguration. Its report committed to further hardening Persistent Disk to prevent regional impact during single-zone issues.

Two years later, Persistent Disk problems again delayed failovers into the zones that had kept their power. Google’s reports do not say whether the causes were related, and we do not assume they were. The pattern is enough for a bank’s planning: a provider’s shared control layers are where a zone failure leaks into the rest of a region.

The bank’s own control plane

A bank has a control plane of its own, and it is often less resilient than its production services. Its recovery tools include deployment pipelines and build runners, artifact registries, infrastructure-as-code state files and secrets managers. They also include privileged access tools, the identity provider engineers use to sign in, the VPN or bastion that reaches the environment, and the monitoring and logging stack. If one of them is zonal, or depends on a regional API that is returning errors, responders lose the ability to act when they need it.

The design principle that answers this is called static stability. A statically stable service keeps serving from the surviving zones without making any control-plane change at all. Capacity is already running. Load balancers remove failed backends on their own through health checks. Database failover is automatic and tested. Nothing in the first hour requires creating a resource, changing DNS through an API or running a pipeline.

A practical test follows from it. Run a zone-loss exercise in which the control plane is treated as read-only for the first hour: no creates, no scaling, no deployments. Whatever the service cannot do under that rule is a dependency on the layer most likely to be impaired. Break-glass access for engineers, with credentials and network paths that do not depend on the affected region, belongs in the same exercise.

Static stability has a price. Pre-running capacity is the headroom cost from the previous layer, and automatic failover that never asks for permission can fire on a false signal. The trade is deliberate: a bank accepts the cost of idle capacity and the risk of an occasional unnecessary failover in exchange for not needing working management APIs during the worst hour of an outage.

Decision: who calls a zone lost

The fifth layer is human. Someone has to decide that a zone is lost and act on it: drain traffic, promote a replica, switch a channel to a reduced mode. Automatic failover covers the clean cases. Real zone failures are often gray. Some requests fail and others succeed, errors come and go, and health checks that test whether a process is alive keep passing.

The provider’s own account arrived slowly. Google’s first public notice came about two and a half hours after impact began and described elevated errors and intermittent network failures across the region. It named the power event in zone b about five hours in. Its 27 October report committed to an automated alert-to-communication mechanism for incidents that affect a location, which acknowledges the lag. A bank that waited for the status page before acting would have spent much of the outage waiting.

The decision therefore has to rest on the bank’s own evidence. Three things make that possible:

• Per-zone telemetry. Synthetic probes for each critical journey, run from each zone, and dashboards that split error rates and latency by zone. A gray failure that disappears in a regional average is obvious when one zone’s line diverges.
• Thresholds set in advance. A written rule that a zone is drained when its error rate or latency stays beyond a set level for a set time. The exact numbers matter less than agreeing them before the incident.
• Delegated authority. The on-call engineer or incident commander may drain a zone without a change board, informs the service owner afterward, and is backed for a reasonable call made inside the rule.

The decision layer depends on the capacity layer. Draining a healthy zone by mistake removes a third of capacity. In a design without headroom, that mistake could cause the outage it was meant to prevent, so people hesitate. In a design with headroom, a false alarm is cheap, and fast decisions follow. Firms that skimp on capacity often pay for it in slower decisions.

The decision also starts other clocks. Under the UK operational resilience rules, what counts against an impact tolerance is how long the service is disrupted, whatever the cause. The FCA said on the day that it expected firms to fix problems quickly and keep customers informed. A firm’s own status page, in-app messages and contact-center scripts should reflect what customers are experiencing, before the provider has confirmed anything.

Escalation to the provider through its support channels still matters, and the new critical third parties regime gives regulators their own line to Google. Neither speeds up a firm’s own call. A bank that has rehearsed draining a zone, with the people who would do it on the night, will act in minutes. A bank that has only written the procedure will spend the first hour finding out who is allowed to use it.

How the layers differ by type of firm

The five layers apply to any firm running in the cloud. Who controls each layer, and which rules frame the decision, vary.

UK banks and building societies

The PRA and FCA operational resilience rules require firms to identify important business services, set impact tolerances, map the resources behind each service and test against severe but plausible scenarios. Since 31 March 2025, firms have been expected to be able to remain within those tolerances. Losing one zone of a cloud region is plausible by any reading. At many large banks, digital channels run in the public cloud while core ledgers remain in the bank’s own data centers. For them, the five layers apply mainly to login, mobile and online banking, notifications and the integration services that connect channels to the core. Google’s designation as a critical third party brings it under direct oversight, and the Treasury was explicit that firms remain responsible for their own third-party risks.

App-based banks, e-money and payment firms

Firms built in the cloud tend to run everything there, often with one provider and one home region for the core. Their exposure covers all five layers, including the ledger. They also control more of the answer, because their engineering teams own the architecture end to end. FCA-regulated payment and e-money firms fall under the FCA’s operational resilience requirements as well. Firms with EU-licensed entities are subject to the EU’s Digital Operational Resilience Act, which has applied since January 2025 and sets rules for classifying and reporting ICT incidents and for testing resilience. For these firms, moving the core to another region is an expensive project with data-location consequences. Making the core survive a lost zone is the cheaper and more urgent job.

US banks and credit unions

The outage was in London, but the physics are the same in every US region. For US banks, the 2021 computer-security incident notification rule comes from the Federal Reserve, the FDIC and the OCC. It requires a banking organization to notify its primary federal regulator within 36 hours of determining that a qualifying incident has occurred. Bank service providers must notify affected bank customers as soon as possible when an incident has materially affected, or is reasonably likely to materially affect, their services for four or more hours.

Whether a cloud provider or software vendor counts as a bank service provider for a given service, and whether a given outage qualifies, are questions for counsel. Federally insured credit unions report under the NCUA’s separate rule, which has a 72-hour clock and is framed around cyber incidents. Many community banks and credit unions reach the cloud through core processors and digital banking vendors. For them, the five layers are questions to put to those vendors, with the answers requested in writing.

Broker-dealers and wealth managers

FINRA Rule 4370 requires member firms to maintain business continuity plans. SEC Regulation SCI applies to SCI entities such as exchanges, clearing agencies and certain alternative trading systems, and not to most broker-dealers. For a firm handling orders, tolerance for a lost order path during market hours is measured in minutes, so the decision layer and static stability matter most. A wealth manager’s client portal can tolerate a slower recovery. Trade instructions and money movements near a cut-off cannot. UK investment firms within scope of the FCA’s operational resilience rules face the same impact-tolerance discipline as banks.

Fintechs and processors serving banks

A processor or banking-as-a-service platform that loses a zone takes its bank clients with it. Its zone tolerance becomes their third-party risk, and the five layers become their due diligence questions. Processors that can show the test results described below will find those reviews shorter, and those that cannot should expect them to get longer.

The case for accepting zonal risk

The strongest objection deserves a fair hearing. Zone failures are rare. Most workloads in this one recovered within about six hours, physical power returned in under an hour and a half, and Google found no impact beyond the region. Google’s own high-availability services mostly failed over as designed. Full zone tolerance, by contrast, is expensive. It means headroom of up to 50 percent on critical services, synchronous replication that slows every write, engineering time, and production tests that carry their own risk. For many services, such as internal analytics, document generation and marketing content, accepting zonal risk with a tested recovery plan is the rational choice.

A second objection is sharper. When the provider’s own regional services degrade, as they did here, a customer’s careful design may not save it. If control-plane dependencies and storage problems cross zone boundaries, why pay for zone tolerance the provider cannot guarantee?

Both objections hold for most services and fail for a few. The few are the services inside an impact tolerance: card authorization, payments, login and the channels that tell customers what is happening. For them, the costs of zone tolerance are bounded and knowable in advance. The cost of failing is open-ended: customers unable to pay at the till, with a regulator and a parliamentary committee both engaged on the day itself.

The second objection has a specific answer. The layers a bank controls most completely, static stability and decision speed, are the ones that do not depend on the provider’s control plane. A service that keeps running without control-plane calls, and a team that drains a zone on its own evidence, are protected against much of what the provider’s coupling can do. The residue, such as a managed database whose failover stalls, is what the manual runbook and the vendor evidence are for.

A related response concerns multi-cloud. A second provider is a weak answer to a zone loss. A lost zone is the case where staying inside one provider and one region should work, and the effort of making it work is smaller than the effort of running a second provider well. Spending on a second cloud while a critical datastore keeps two of three replicas in one zone buys the wrong protection.

The argument is weakest for software bought as a service. A bank that buys a service cannot set replica placement or headroom. Its levers are contract terms, evidence and exit plans. Those levers are slower and blunter, which is why the evidence requested from vendors, described next, has to be specific.

What a zone-loss test should prove

An architecture diagram records intent. A zone-loss test shows behavior. Google’s own reports on 20 October show the gap between the two inside a large, mature provider, and banks have no reason to assume their estates are free of it.

A credible test removes a zone from a production-like environment under realistic load, for long enough that slow failures show. Fault-injection tools from cloud providers and open-source projects can simulate zone failures. Simpler methods also work: shutting down instances, blocking network paths to a zone, or draining it at the load balancer. The test should treat the control plane as read-only for the first hour. Responders should not be told in advance which zone will fail or when, so the decision to drain comes from their own telemetry.

It should produce five results, one for each layer:

1. Placement: the components on each critical path that stopped, compared with the inventory’s prediction.
2. State: for each data store, time to a writable primary, any data lost, and whether every quorum held.
3. Capacity: the share of peak load carried with the zone removed and no new instances.
4. Control: every recovery step that needed a control-plane call, and whether it succeeded.
5. Decision: minutes from the first anomaly to the decision to drain, and from that decision to stable service.

The same five results are the right request to critical vendors. A core processor, card processor, digital banking platform or API gateway provider should be able to say how its service behaved in its last zone-loss test. Google’s product-by-product account of 20 October shows that such answers can be given in useful detail. A vendor that cannot give them has told its clients something as well.

Two habits round out the program. First, record when redundancy is reduced for maintenance, in the bank’s own estate and, where providers will share it, in theirs. The London facility failed while in a maintenance posture that followed a generator test six days earlier. Second, report the results to the board as one page per important business service, showing the five results, the trend and the accepted exceptions with their owners. A board that sees those pages knows which services will survive a lost zone. A board that sees only a provider’s uptime figure does not.

With hindsight

Seventeen days have passed, and most of what is known comes from Google itself. Its account changed as the investigation went on. That is to its credit, and a caution for anyone drafting their own incident report from a provider’s first version.

On 21 October, Reuters reported that the European Banking Authority was in contact with Google Cloud and that the FCA was engaging with it, while the Bank of England declined to comment. A Google spokesperson said the disruption had been resolved and that the company would update its status dashboard with an analysis.

Google’s preliminary report, posted on 22 October, said the generators failed to take the load because both the primary and backup logic controllers failed to start the transfer. The 27 October report explained why. The controllers’ undervoltage sensing was wired upstream of the open feed B breaker, so they saw voltage that was not reaching the building and never called for the generators. The facility had been on a single feed while repairs followed problems found in a generator load test on 14 October. A bank that copied the preliminary explanation into its own report would have had to revise it within days. Provider root causes are provisional until the full report, and a firm’s internal and regulatory reports should say so.

The updated report of 4 November added the product-by-product detail used above, and a remediation list. Several items were done by 26 October: the cable fault located, the controllers reconfigured and validated, and emergency operating procedures updated after being practiced on 24 October. A freeze on planned maintenance at sites with similar topologies, with tighter change approvals, was in place from 30 October.

Other dates moved. The 27 October report expected the damaged feed to be repaired by 30 October and the facility back on both utility feeds by 31 October. The 4 November report put those at 13 and 17 November. Validation of controller configurations at similar sites was due by 15 November. A new item, reviews during commissioning and a second-team check to prevent similar misconfigurations, was due by 18 December.

Those slipping dates matter to banks. On Google’s current estimate, from the outage until mid-November the zone runs on a facility supplied from a standby utility source instead of its two normal feeds, with generator controls corrected and tested. Customers know that only because Google published it. Few providers disclose reduced redundancy as a matter of routine, and it is a reasonable thing for a critical customer to ask about.

On the regulatory side, the Treasury Committee asked for an initial reply by 4 November and a full one within three months. The substance of the regulators’ answers, and any use of their new powers over Google, will be clearer in the new year. The FCA and the EBA described their contacts as efforts to understand the impact. We have seen no regulatory findings announced by early November, and no detailed technical accounts from the affected banks.

Some things have held up so far. The failure stayed inside the London region, and services configured for high availability mostly failed over. Others did not. A scheduler had undermined replica placement, and some failovers started late or stalled. Capacity was not there when everyone needed it, the provider’s control layers had cross-zone dependencies, and public communication lagged the event by hours. Google’s remediation list addresses the last of those on its side. The rest are shared work between customer and provider.

The horizon is too short for firmer judgments. Nobody outside the affected firms yet knows whether any breached its impact tolerances, or what they reported to supervisors. Nor is it known whether the first real test of the critical third parties regime will lead to changes at Google beyond those in its own report. Those answers will take months. The architectural work described here does not need to wait for them.