{"id":26437,"date":"2022-12-29T11:43:32","date_gmt":"2022-12-29T11:43:32","guid":{"rendered":"https:\/\/mailinvest.blog\/index.php\/2022\/12\/29\/it-took-just-four-days-from-elon-gleefully-admitting-hed-unplugged-a-server-rack-for-twitter-to-have-a-major-outage\/"},"modified":"2022-12-29T11:43:32","modified_gmt":"2022-12-29T11:43:32","slug":"it-took-just-four-days-from-elon-gleefully-admitting-hed-unplugged-a-server-rack-for-twitter-to-have-a-major-outage","status":"publish","type":"post","link":"https:\/\/mailinvest.blog\/index.php\/2022\/12\/29\/it-took-just-four-days-from-elon-gleefully-admitting-hed-unplugged-a-server-rack-for-twitter-to-have-a-major-outage\/","title":{"rendered":"It Took Just Four Days From Elon Gleefully Admitting He\u2019d Unplugged A Server Rack For Twitter To Have A Major Outage"},"content":{"rendered":"

<\/a>\n<\/p>\n

\n

from the whoops<\/i> dept<\/h3>\n

I know, I know. Some of the more angry commenters around here keep insisting that I should stop talking about Elon Musk and Twitter, and I want to do exactly that. I planned<\/em> to do exactly that and not write another post about it all until next week. And then\u2026 Twitter crashed hard last night<\/a>. Downdetector has the receipts:<\/p>\n

\"DownDetector<\/figure>\n

Here\u2019s what happened when I went to visit Twitter:<\/p>\n

\n
\"Twitter<\/figure>\n<\/div>\n

I especially like that \u201cit\u2019s not your fault\u201d bit, because, well, yeah. It\u2019s not.<\/p>\n

As I write this, there hasn\u2019t been anything official about what happened, but I\u2019m assuming that Elon will show up at some point to blame the \u201cwoke mind virus\u201d or the federal reserve or SBF or Anthony Fauci.<\/p>\n

And, it may be a total coincidence, but it was just four days ago that he bragged about pulling the plug on an \u201cimportant server rack.\u201d<\/p>\n

\n
\"Elon<\/figure>\n<\/div>\n

Separately, there have been reports that Musk decided (with little to no notice, and almost no planning) to shut down its Sacramento data center<\/a> and massively downsize their Atlanta data center. Twitter only has one other data center in the US, in Portland, Oregon. Twitter\u2019s use of data centers rather than the cloud is something that\u2019s been discussed over the years, and two years ago the company did sign a deal<\/a> to start using Amazon Web Services, though I don\u2019t think the company relies to heavily on it yet, and the first link in this paragraph notes that Elon has been trying to renegotiate the AWS contract as well (which might mean he\u2019s also stopped paying the bills<\/a> as he seems to have done that with many vendors as part of his \u201crenegotiation\u201d efforts).<\/p>\n

Separately, I\u2019ve heard from three separate people that Elon more or less ordered the shutdown of the an entire data center (presumably the Sacramento one) with basically one day\u2019s notice and no planning.<\/p>\n

And, with that in mind, I\u2019ll remind people that one part of former Twitter security chief Peiter \u201cMudge\u201d Zatko\u2019s whistleblower report<\/a> noted that the company had a deep need for more redundancy, not less:<\/p>\n

\n


\nInsufficient data center redundancy<\/b>, without a plan to cold-boot or
\nrecover from even minor overlapping data center failure, raising the risk of
\na brief outage to that of a catastrophic and existential risk for Twitter\u2019s
\nsurvival<\/em><\/p>\n<\/blockquote>\n

That report also presented a redacted version of the \u201cthreat matrix\u201d Mudge claims he wanted to show the Board, though was urged only to give a high level overview, orally, rather than present a more complete written report. It again notes that a data center failure could be catastrophic.<\/p>\n

\n
\"\"<\/figure>\n<\/div>\n

Later in the report, Mudge notes that this almost happened in the past:<\/p>\n

\n


\nCascading data center problems:<\/b> In or around the spring of 2021, Twitter\u2019s
\nprimary data center began to experience problems from a runaway engineering
\nprocess, requiring the company to move operations to other systems outside of this
\ndatacenter. But, the other systems could not handle these rapid changes and also
\nbegan experiencing problems. Engineers flagged the catastrophic danger that all
\nthe data centers might go offline simultaneously. A couple months earlier in
\nFebruary, Mudge had flagged this precise risk to the Board because Twitter data
\ncenters were fragile, and Twitter lacked plans and processes to \u201ccold boot.\u201d That
\nmeant that if all the centers went offline simultaneously, even briefly, Twitter was
\nunsure if they could bring the service back up. Downtime estimates ranged from
\nweeks of round-the-clock work, to permanent irreparable failure.<\/em><\/p>\n

\u201cBlack Swan\u201d existential threat:<\/b> In fact, in or about Spring of 2021, just such an
\nevent was underway, and shutdown looked imminent. Hundreds of engineers
\nnervously watched the data centers struggle to stay running. The senior executive
\nwho supervised the Head of Engineering, aware that the incident was on the verge of taking Titer offine for weeks, months or permanently, insisted the Board of Directors be informed of an impending catastrophic \u201cBlack Swan\u201d event. Board
\nMember [REDACTED] responded with words to the effect of \u201cIsn\u2019t this exactly
\nwhat Mudge warned us about?\u201d Mudge told [REDACTED] that he was correct. In the end, Twitter engineers working around the clock were narrowly able to stabilize the problem before the whole platform shut down.\n<\/p>\n<\/blockquote>\n

That\u2019s not to say that this has anything to do with the outages last night, but at the very least there are strong arguments that Twitter\u2019s infrastructure is inherently fragile, and shutting down \u201csensitive\u201d server racks or closing down entire data centers without careful planning seems like the sort of thing that could, well, backfire pretty badly.<\/p>\n

Meanwhile, the only comment so far from Musk appears<\/em> (it\u2019s tough to know because Twitter only loads intermittently) is him responding to someone saying \u201cworks for me\u201d when they asked about site problems. Also, in context, Musk is replying to a joke<\/em> about the site being down, rather than a legitimate concern (someone asks if anyone can see or respond to their tweet, and one of Musk\u2019s biggest fans tweeted \u201cI can\u2019t see or respond to it\u201d (obviously making light of the whole thing) and then Musk responds with \u201cworks for me.\u201d<\/p>\n

\n
\"Musk<\/figure>\n<\/div>\n

So it\u2019s not entirely fair to say this is a comment directly about the widespread outages. Assuming Musk realizes Billy is joking, then\u2026 it could just be a weak attempt at playing along? But here\u2019s the actual funny part. The Guardian has an article<\/a> about Musk\u2019s tweet saying stuff \u201cworks for me\u201d except that stuff isn\u2019t working, because the Twitter embed is not showing properly, but instead is showing in failover mode, where if the embed won\u2019t load it just shows the alt-text in as \u201ctweet-like\u201d a form as possible. This screenshot is just pure irony.<\/p>\n

\n
\"image<\/figure>\n<\/div>\n

I eagerly await the comments from folks who were insisting to me just yesterday that Twitter under Musk was functioning much better<\/em> than before, and that this all proved he was right to get rid of approximately 75% of the workforce who obviously did nothing\u2026<\/p>\n

Oh and just as this post was being completed, Elon has a new story, claiming that Twitter was just rolling out \u201csignificant backend server architecture changes\u201d and that \u201cTwitter should feel much faster\u201d (it doesn\u2019t, unless you\u2019re talking about the difference from not working at all\u2026 to kinda working some of the time?).<\/p>\n

\n
\"\"<\/figure>\n<\/div>\n

Even if that was the cause of the outage (and\u2026 I\u2019m doubtful), that still raises all sorts of questions about how the company prepared for the switchover, if it caused such a massive disruption in the process. That\u2019s\u2026 not how any of this should work.<\/p>\n

\nFiled Under: datacenters<\/a>, downtime<\/a>, elon musk<\/a>, fail whale<\/a>, mudge<\/a>
\n
Companies: twitter<\/a>\n<\/p>\n<\/div>\n