CrowdStrike Strikes First
Let's not give them another chance...
July 19, 2024 will go down as the biggest “blunder” of the modern cloud-based Internet dependent world. ClowdStrike's Falcon Sensor “endpoint detection and response” (EDR) software was responsible for the “largest ever” IT outage. I had to make the Beaky Buzzard logo for CrowdStrike, because that was the first thing that came to mind.
Here’s my experience with CrowdStrike as of today – starting at a little before 2am.
I work from home as a consultant for my current employer. I have a corporate (client provided) laptop that I use for email, Microsoft Teams meetings, and general interactions with everyone I work with. Of course, I still call people on the phone and send text messages, but the majority of my work is done on my laptop. The client I support is more than 800 miles away.
At just before 2am Friday, one of my computer monitors lit up. I was just about to fall asleep. I peeked over and saw the normal corporate login screen, so I assumed that one of our cats was on my desk and moved a mouse, so I started to drift back to sleep again. Then it lit up again. This time there was a BSoD (Blue Screen of Death).
I thought that was rather odd, because I wasn’t on the computer and it had been idle for a long time. Updates usually aren’t pushed, and there’s ample warning before an update is going to happen.
In my personal experience, Blue Screens usually happen because of driver issues, or bad RAM. The corporate machines are seriously locked down, so there’s not much I can do. But I was able to get into the UEFI hardware test options. I ran the memory test overnight, it was similar to Memtest86. It came back in the morning with zero errors.
I exited the test, and rebooted the machine, then it Blue Screened again, this time I was ready, and I snapped a photo of it.
I noticed that the Stop Code was “PAGE_FAULT_IN_NONPAGED_AREA. Mostly useless, but it also listed “What Failed,” which was “csagent.sys.” CS for CrowdStrike. I went to one of my computers and looked to see what might be going on, and immediately saw the news about how CrowdStrike was causing errors on machines all over the world.
The problem first arose overnight for the East Coast of the U.S.
"CrowdStrike is aware of reports of crashes on Windows hosts related to the Falcon Sensor," the cybersecurity company wrote in an alert confirming the outage at 1:30 a.m. ET on Friday. CrowdStrike's Falcon Sensor is software designed to prevent computer systems from cyber attacks.
"Symptoms include experiencing a bugcheck\blue screen error related to the Falcon Sensor," CrowdStrike's alert continued. "Our Engineering teams are actively working to resolve this issue and there is no need to open a support ticket."
…
Fortunately, CrowdStrike has since announced at 2:30 a.m. ET that it has identified the update causing the issue and rolled it back. The company also offered a workaround for anyone having problems:1. "Boot Windows into Safe Mode or the Windows Recovery Environment
2. "Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
3. "Locate the file matching 'C-0000029*.sys', and delete it.
4. "Boot the host normally."
Of course, having to do this for every single computer in multiple companies across the globe is still likely to take some time.
I figured that was an easy enough fix. I could boot the machine in recovery mode get to the command prompt, path to the file and delete it, or rename it.
Nope. That was not an option, because the machine is encrypted. So the recovery utility takes me to a drive “X:\” and “C:\” is not accessible. Without the drive encryption key, there’s nothing I can do to fix the issue. And, I highly doubt that this corporate client is going to give me the drive encryption key. They’re likely going to have to mail me another laptop.
This outage caused serious damage, not just to businesses, but to critical infrastructure, air travel, and emergency services. 911 was down in some areas, because their dispatching systems are Windows based, and they were using CrowdStrike Falcon Sensor.
The problem is worse for large corporations, because now they have computers already deployed that can’t be corrected remotely. Requiring someone to go get the computer, or get in front of the computer to correct the issue.
“You have to physically walk over to every computer and power it down and then bring it up, and when the screen comes up, you have to hit F3 to go into what they call Safe Mode and then go and delete a file somewhere,” the New York law firm CISO explained. “It's just a nightmare.”
Good luck getting quickly to this one.
I have never liked the idea of cloud-based services.
“The cloud” sounds great, until something goes wrong. Then you’re stuck, and your IT people are powerless to fix it.
Once upon a time large corporations, and many local governments had a well staffed IT department. They would test systems in a development environment. Say they support (5) different hardware platforms, they could test Windows updates, or other software changes on those dev hardware platforms first. If there are no issues, they can, within reason, deploy the changes across all their systems and have a reasonable expectation that everything will keep working. Same goes for servers or VMs.
But with CrowdStrike that doesn’t happen. These corporations and government agencies outsourced all of that. The 3rd party company has free range to make a change – without anyone on the business or government side knowing that it’s happening. SaaS (Software As A Service) programs are sold this way. Everything is always “up to date” because the 3rd party manages it, and now local IT people don’t have to. But they also just signed their digital life away.
CrowdStrike is probably a bad investment – now.
Founded in 2011, CrowdStrike quickly became an industry leader in cybersecurity and has only grown in popularity in recent years as demand for such services has increased.
It has made an aggressive marketing push that included Super Bowl ads and has worked to tailor its products to the needs of large organizations with complex security. At least one 2023 analysis, from Canalys, found the company controls roughly 20 percent of the cybersecurity market.
Reviews for the company’s products were broadly positive in the past, describing them as user-friendly and accurate in finding threats. According to the Motley Fool, its [CrowdStrike's] stock surged 400 percent in the last five years.
This whole experience got me thinking, “Wow, I bet someone at CrowdStrike is getting canned over this! And I wonder what’s happening to the stock price? You can’t take a large portion of the world’s critical services down and not have a serious stock drop.”
Well, their stock dropped, but nowhere near as much as I would have figured.
Strangely, CrowdStrike stock started to drop significantly in the days leading up to the event. Did somebody know something?
But then, I thought, who are their largest investors?
Perhaps that’s how their stock isn’t tanking. After all, I know from personal experience that CrowdStrike is big into ESG and DEI. I actually applied for a position at CrowdStrike around four years ago. Their website was bragging about diversity and other such nonsense. During the online application process it asked for my preferred pronouns. I didn’t even know what that meant. I had to look it up!
CrowdStrike sent me test material with parts of virus code, I had to determine how the threat/malware was supposed to work. I was able to do that relatively easy. But I never got a call back.
I’m assuming that their stock is being propped up by the large investment firms. Because the big firms aren’t pulling money out, the stock is staying higher than it otherwise would.
But then, those “investment firms” do a lot more investing based on social issues, not ROI. They tend to invest a lot into companies that are pushing an agenda, like DEI. I’m sure that CrowdStrike got plenty of dollars for pushing woke nonsense – over the past five years. Perhaps that’s why their stock increased 400% during that time.
Can this be explained by incompetence?
Yes and no.
Yes: insofar as many IT departments have sold their soul to 3rd parties. That was never a good idea, because very little could be tested in-house and guaranteed. Single points of failure are never good.
No: when it comes to the changes made on the CrowdStrike side. From what I’ve seen the computers effected were primarily Windows 10 and Windows 11 machines – that were upgraded from Windows 10. Not so much an issue for the machines that were originally Windows 11.
This means that the issue could be Windows build version dependent. Nevertheless, CrowdStrike, fresh with a 400% stock increase, should have the resources available to test their product on a wide array of potential Windows builds. Falcon Sensor has two-way communication with the machines that run the software. CrowdStrike knows what Windows builds are out there, and what machines are running their software.
CrowdStrike didn't gain nearly 20% of the the global IT security market through sheer incompetence. Don't fall for the DEI excuse.
Was this some kind of test?
Only the people calling the shots could possibly know that.
I suspect that someone knew about this. After CrowdStrike Falcon Sensor was installed on my work laptop. I did encounter a few Blue Screens. The Blue Screens increased in frequency before the 07/19/24 event.
I was going to request a new laptop last week, because I assumed that bad RAM was the most likely culprit. Maybe CrowdStrike was testing something, and getting feedback from the machines running their Falcon Sensor software?
I can’t say for sure, but I can’t see the world’s largest IT outage being a "huge mistake," where everything just lined up perfectly.
Is there a pattern here, involving DEI and failures?
Of course there is.
That’s the entire point of DEI. To ensure that the most critical economic, and safety related aspects of our economy – are staffed by incompetent people.
As soon as some large multinational company goes all in with DEI initiatives, they begin to see problems. Boeing is the probably the first example that comes to mind. Anheuser-Busch and the whole Bud Light trans promoter is another.
In each of these examples there were no “mistakes” made.
Everyone knows that airplanes need to be built and maintained by the most skilled personnel.
Everyone knows that anybody drinking Bud Light is NOT going to be happy watching some trans chick (dude) promoting their alcoholic beverage. The official beer of the NFL.
These aren’t mistakes, anyone can see that these decisions would lead to bad outcomes. It doesn’t take polling or research to understand that doing the opposite of what works, might cause issues.
Where does this leave us?
People likely died because emergency services like 911 were down for hours. Flight cancellations and delays will likely continue for a few days at minimum. Hospital services and many surgeries were canceled. This outage will result in billions of dollars lost.
However, unless things change drastically, and IT departments start doing things in-house, this will become the norm.
Just like there are rolling blackouts in California, there will be rolling IT outages globally.
Do I think this world changing event will force change? Absolutely not. Because the people who made the decisions that got us to this point were incompetent, and they have plausible deniability. They didn’t cause the outage, CrowdStrike did. But the fact remains, if they would have managed things the way it was done in years prior, this kind of thing wouldn't have happened.
That being said, the primary stock holders in CrowdStrike are going to keep it propped up to ensure it's viability. This will cause many corporations and governments to consider this a one-off kind of event, and dismiss the entire ordeal. As they say in business, "That's what insurance is for right? I’m sure this is a rare thing."
It appears that many of these companies are being corralled into a single point of failure scenario, for whatever reason, and they are too incompetent to see what's coming next.
What do you think?
If you enjoy my writing, you might want to check out my original blog WindUpRubberFinger.com. I have decided to mirror my posts on Substack, and WindUpRubberFinger.com. I will work on moving older posts over to Substack as references for newer articles.
I've never seen an "update" like this pushed to a large group at once. Normally, updates are pushed to smaller groups and results are carefully monitored before a large push is attempted.
Enjoyed and learned a few things from your article.