Microsoft blames router IP address change for global outage • The Register

The worldwide outage of Microsoft 365 providers that final week prevented some customers from accessing sources for greater than half a working day was right down to a packet bottleneck brought on by a router IP deal with change.

Microsoft’s extensive space community toppled a bunch of services from 07:05 UTC on January 25 and though some areas and providers had come again on-line by 09:00, intermittent packet loss woes weren’t absolutely mitigated till 12:42. The wobble additionally affected Azure Authorities cloud providers.

In a postmortem, Microsoft stated that modifications made to its WAN had hit connectivity between purchasers and Azure, throughout areas and cross-premises through ExpressRoute.

“As a part of a deliberate change to replace the IP deal with on a WAN router, a command given to the router prompted it to ship messages to all different routers within the WAN, which resulted in all of them recomputing their adjacency and forwarding tables. Throughout this re-computation course of, the routers had been unable to appropriately ahead packets traversing them.

“The command that prompted the problem has completely different behaviors on completely different community gadgets, and the command had not been vetted utilizing our full qualification course of on the router on which it was executed.”

This meant customers had been unable to entry sources hosted in Azure or different Microsoft 365 and Energy Platform providers.

Microsoft stated monitoring methods detected DNS and WAN-related troubles at 07:12, some seven minutes after they started.

By 08:20, resident techies at Microsoft had noticed the “problematic command that triggered the problems” and a few 40 minutes later networking telemetry indicated most of the providers had been working once more.

Nevertheless, Microsoft stated the preliminary drawback with the WAN meant automated methods for sustaining its well being had been paused. This included methods for figuring out and expelling unhealthy gadgets, in addition to the visitors engineering system for optimizing the stream of information throughout the community.

“As a result of pause in these methods, some paths within the community skilled elevated packet loss from 09:35 UTC till these methods had been manually restarted, restoring the WAN to optimum working situations. This restoration was accomplished at 12:43 UTC,” the postmortem added.

Efforts Microsoft is taking to make comparable incidents much less possible or extreme embrace blocking “extremely impactful command from getting executed on the gadgets” and requiring all command execution on gadgets to comply with secure tips.

The ultimate post-incident report is scheduled to be printed a fortnight after the outage. ®