{"id":136061,"date":"2026-07-21T07:04:23","date_gmt":"2026-07-21T07:04:23","guid":{"rendered":"https:\/\/mailinvest.blog\/index.php\/2026\/07\/21\/google-cloud-outage-shows-its-still-hard-to-understand-hyperscalers-real-resilience-regimes\/"},"modified":"2026-07-21T07:05:25","modified_gmt":"2026-07-21T07:05:25","slug":"google-cloud-outage-shows-its-still-hard-to-understand-hyperscalers-real-resilience-regimes","status":"publish","type":"post","link":"https:\/\/mailinvest.blog\/index.php\/2026\/07\/21\/google-cloud-outage-shows-its-still-hard-to-understand-hyperscalers-real-resilience-regimes\/","title":{"rendered":"Google Cloud outage shows it\u2019s still hard to understand hyperscalers\u2019 real resilience regimes"},"content":{"rendered":"

<\/a>\n
<\/p>\n

\n

off prem<\/p>\n

Single datacenter and simply three providers taken down by \u2018upstream\u2019 energy drawback, whereas the remainder of a zone and area stored buzzing<\/p>\n<\/div>\n

\n

Google Cloud final week skilled an outage that analysts say demonstrates that not all guarantees of cloudy resilience are created equal.<\/p>\n

Google\u2019s incident report<\/a> defined that three providers \u2013 the VMware Engine (GCVE), NetApp Volumes, and Naked Steel Options (BMS) \u2013 skilled a 15-hour outage because of a cooling failure in its europe-west4-a zone.<\/p>\n

The report contains the next element: \u201cThe datacenter serving europe-west4-a for GCVE, BMS, and NetApp has skilled an influence failure, which subsequently triggered a cooling failure.\u201d<\/p>\n

The essential element there’s that Google makes use of a discrete datacenter for these three providers.<\/p>\n

One other notable aspect of the incident report is the admission that \u201c{An electrical} fault occurred on the utility grid upstream of the datacenter, disrupting {the electrical} distribution gear and cooling gear.\u201d<\/p>\n

Google hasn\u2019t defined how an upstream failure triggered that disruption however did say it \u201cproactively turned down workloads with a view to shield buyer information from any dangers posed by operating infrastructure in a excessive temperature surroundings.\u201d<\/p>\n

Every time your correspondent talks to hyperscalers or datacenter operators about how they guarantee resilience, they inform me about their use of a number of redundant items of power infrastructure, plus on-site technology capabilities that may hold a datacenter powered for days if mandatory.<\/p>\n

We\u2019ve requested Google if it had mills or different power sources at this web site, and if that’s the case, why it nonetheless needed to flip down workloads. We\u2019ve not obtained a response on the time of writing. Google instructed us its incident evaluation \u201cis at the moment ongoing\u201d and promised to observe up as soon as it’s out there.<\/p>\n

Hidden dependencies<\/h3>\n

We additionally requested Google if it advertises the truth that a few of its providers are tied to a single datacenter, a matter of curiosity as a result of like different hyperscalers it divides its cloud into \u201careas\u201d that sometimes comprise a number of \u201czones\u201d unfold throughout a metropolis or different locale. Like its hyperscale friends, Google recommends inserting workloads throughout completely different zones and areas to make sure resilience. But this incident reveals some providers could be tied to a single datacenter in a zone \u2013 and that these single datacenters can expertise issues whereas the remainder of the zone retains working.<\/p>\n

Analysts instructed The Register<\/em> the outage reveals organizations have to dig into clouds\u2019 guarantees of resilience.<\/p>\n

\u201cThe true difficulty is transparency: prospects are usually instructed to make use of a number of zones and areas for resilience however are not often given visibility into whether or not a selected managed service has a single-datacenter dependency inside a zone,\u201d mentioned Biswajeet Mahapatra, principal analyst at Forrester. \u201cBecause of this, many organizations assume the cloud abstraction gives extra facility-level redundancy than may very well exist for specialised providers.”<\/p>\n

\u201cThe underlying structure will not be essentially uncommon,\u201d he added. \u201cAWS, Azure, and Google all function providers that depend on devoted {hardware}, storage platforms, or tightly coupled infrastructure that might not be distributed throughout a number of services in the identical method as core compute and storage providers.\u201d<\/p>\n

Gartner Director Analyst Adrian Wong reminded The Register<\/em> of the 2023 outage at Google Cloud\u2019s europe-west9-a region<\/a>, the reason for which was a water leak that Google said<\/a> \u201coriginated in a non-Google portion of the power.\u201d<\/p>\n

Google makes use of a instrument referred to as \u201cSpanner\u201d to copy information throughout zones, however within the flooded zone Google\u2019s Spanner configuration didn\u2019t work as soon as one constructing grew to become unavailable.<\/p>\n

\u201cIt is extremely arduous to determine how a person area is architected,\u201d Wong mentioned. \u201cOur prospects are sometimes stunned by that,\u201d he added.<\/p>\n

The incident report for final week\u2019s outage contains an apology.<\/p>\n

\u201cWe all know how a lot you depend on Google Cloud, and we remorse the impression in your productiveness,\u201d the doc states, earlier than promising a last incident report will element \u201cpreventative actions.\u201d<\/p>\n

However as this incident reveals, realizing how Google plans to keep away from future incidents of this kind gained\u2019t arm prospects with the data to know if these mitigations will handle hidden design points that may cut back resilience. \u00ae<\/p>\n<\/div>\n