CrowdStruk

My experience with the Internets Extinction Level Event

Posted by msatterfield on August 10, 2024

Woke up to a phone call from my overseas boss at roughly 12:30am. She was concerned that our local site had lost power.

Mere seconds later she was concerned we were being hacked.

About two minutes into me being awake, I didn’t have a clue what was going on. Thankfully I was able to still VPN in, and get into vcenter, where I was assaulted by a sea of blue screens.

Honestly, the whole thing is absurd, I spent the entire night getting systems back on track, and still went in at six am. Worked to around 3:00pm, and by that time we were 99% back.

We had around one hundred and thirty-some-odd servers affected, and dozens of workstations, not to mention user laptops.  I believe that we addressed the situation beautifully and

Two things saved us from complete destruction.

  1. I, and my European counterparts were vehemently against CrowdStrike from the beginning.  As a rule no admin who’s worth his salt, wants to hand over kernel level access to a third-party MSP on their production environment. So, we cut a deal and were able to keep Sophos for our Factory Talk, and Chromatography machines, keeping our production up in spite of everything else.
  2. Communication is key, thankfully we were able to get the fix communicated to the masses prior to most of them having to deal with it. We had two techs walking the plant, helping, and a station where users could come get their PC repaired.

So what did we learn ?, well, we learned that even though we were told we were on a seven day wait period for CrowdStrike updates (which I believe happened to be the case in a lot of companies) that, that was all bullshit. We also proved the benefits of not having all your eggs in one basket.

Personally, I am amazed they are still in business after nuking 8.5 million computers, and grounding thousands of flights. I think these bloggers who are defending them are crazy, and I would never willingly use them again.

In this new world of segregated IT, it’s much to easy to outsource right’s to MSPs. I get the appeal from a management perspective, but one you start talking that level of access without internal staff being able to manipulate, the argument for it fails.