{"id":20842,"date":"2022-12-04T13:20:36","date_gmt":"2022-12-04T13:20:36","guid":{"rendered":"https:\/\/mailinvest.blog\/index.php\/2022\/12\/04\/rackspace-hosted-exchange-outage-due-to-security-incident\/"},"modified":"2022-12-04T13:20:36","modified_gmt":"2022-12-04T13:20:36","slug":"rackspace-hosted-exchange-outage-due-to-security-incident","status":"publish","type":"post","link":"https:\/\/mailinvest.blog\/index.php\/2022\/12\/04\/rackspace-hosted-exchange-outage-due-to-security-incident\/","title":{"rendered":"Rackspace Hosted Exchange Outage Due to Security Incident"},"content":{"rendered":"

<\/a>\n
<\/p>\n

\n

Rackspace hosted Exchange suffered a catastrophic outage beginning December 2, 2022 and is still ongoing as of 12:37 AM December 4th. Initially described as connectivity and login issues, the guidance was eventually updated to announce that they were dealing with a security incident.<\/p>\n

Rackspace Hosted Exchange Issues<\/h2>\n

The Rackspace system went down in the early morning hours of December 2, 2022. Initially there was no word from Rackspace about what the problem was, much less an ETA of when it would be resolved.<\/p>\n

Customers on Twitter reported that Rackspace was not responding to support emails.<\/p>\n

\n

This has been quite the day with #Rackspace<\/a>. Every hosted exchange client has been down for 14 hours or so. Support isn’t reading\/responding to tickets. Updates are unhelpful. <\/p>\n

I am concerned now that they fell victim to something bad like the ProxyNotShell PoC hack. https:\/\/t.co\/jchKsAO3Z7<\/a><\/p>\n

\u2014 Joe Sinkwitz (@CygnusSEO) December 2, 2022<\/a><\/p>\n<\/blockquote>\n

A Rackspace customer privately messaged me over social media on Friday to relate their experience:<\/p>\n

\n

\u201cAll hosted Exchange clients down over the past 16 hours.<\/p>\n

Not sure how many companies that is, but it\u2019s significant.<\/p>\n

They\u2019re serving a 554 long delay bounce so people emailing in aren\u2019t aware of the bounce for several hours.\u201d<\/p>\n<\/blockquote>\n

The official Rackspace status page offered a running update of the outage but the initial posts had no information other than there was an outage and it was being investigated.<\/p>\n

The first official update<\/a> was on December 2nd at 2:49 AM:<\/p>\n

\n

\u201cWe are investigating an issue that is affecting our Hosted Exchange environments. More details will be posted as they become available.\u201d<\/p>\n<\/blockquote>\n

Thirteen minutes later Rackspace began calling it a \u201cconnectivity issue.\u201d<\/p>\n

\n

\u201cWe are investigating reports of connectivity issues to our Exchange environments.<\/p>\n

Users may experience an error upon accessing the Outlook Web App (Webmail) and syncing their email client(s).\u201d<\/p>\n<\/blockquote>\n

By 6:36 AM the Rackspace updates described the ongoing problem as \u201cconnectivity and login issues\u201d then later that afternoon at 1:54 PM Rackspace announced they were still in the \u201cinvestigation phase\u201d of the outage, still trying to figure out what went wrong.<\/p>\n

And they were still calling it<\/a> \u201cconnectivity and login issues\u201d in their Cloud Office environments at 4:51 PM that afternoon.<\/p>\n

Rackspace Recommends Migrating to Microsoft 365<\/h2>\n

Four hours later Rackspace referred to the situation as a \u201csignificant failure\u201dand began offering their customers free Microsoft Exchange Plan 1 licenses on Microsoft 365 as a workaround until they understood the problem and could bring the system back online.<\/p>\n

The official guidance stated:<\/p>\n

\n

\u201cWe experienced a significant failure in our Hosted Exchange environment. We proactively shut down the environment to avoid any further issues while we continue work to restore service. As we continue to work through the root cause of the issue, we have an alternate solution that will re-activate your ability to send and receive emails.<\/p>\n

At no cost to you, we will be providing you access to Microsoft Exchange Plan 1 licenses on Microsoft 365 until further notice.\u201d<\/p>\n<\/blockquote>\n

Rackspace Hosted Exchange Security Incident<\/h2>\n

It was not until nearly 24 hours later at 1:57 AM on December 3rd that Rackspace officially announced that their hosted Exchange service was suffering from a security incident.<\/p>\n

The announcement further revealed that the Rackspace technicians had powered down and disconnected the Exchange environment.<\/p>\n

Rackspace posted:<\/p>\n

\n

\u201cAfter further analysis, we have determined that this is a security incident.<\/p>\n

The known impact is isolated to a portion of our Hosted Exchange platform. We are taking necessary actions to evaluate and protect our environments.\u201d<\/p>\n<\/blockquote>\n

Twelve hours later that afternoon they updated the status page with more information that their security team and outside experts were still working on solving the outage.<\/p>\n

Was Rackspace Service Affected by a Vulnerability?<\/h2>\n

Rackspace has not released details of the security event.<\/p>\n

A security event generally involves a vulnerability and there are two severe vulnerabilities currently in the wile that were patched in November 2022.<\/p>\n

These are the two most current vulnerabilities:<\/strong><\/p>\n