Do you have your VMware ESXi hypervisor joined to Active Directory? Well, the latest news from Microsoft serves as a reminder that you might not want to do that given the recently patched vulnerability that has security experts deeply concerned.
…
Essentially, if an attacker was able to add an AD group called “ESX Admins,” any user added to it would by default be considered an admin.
Crowdstrike fix for blue screen issue Workaround Steps: 1Boot Windows into Safe Mode or the Windows Recovery Environment 2Navigate to the C:WindowsSystem32driversCrowdStrike directory 3Locate the file matching “C-00000291*.sys”, and delete it. 4Boot the host
— SANS.edu Internet Storm Center (@sans_isc) July 19, 2024
It started on a Thursday USA time
I'm in awe of the scale of the Crowdstrike / Windows BSOD issue.
Here are the most startling images I've seen morning.
Let's start with this: at 10pm PT yesterday, famous @troyhunt notices that something odd is happening to Windows systems:https://t.co/0KWDELUycT
Note "channel updates …bypassed client's staging controls and was rolled out to everyone regardless" https://t.co/UecaAmJdqc
A few IT folks who had set the CS policy to ignore latest version confirmed this was, ya, bypassed, as this was "content" update (vs. a version update)
"They pushed a new kernel driver out to every client without authorization to fix an issue with slowness and latency that was in the previos Falcon sensor product. They pissed over everyone's staging and rules and just pushed this to production"https://t.co/XVEJoLTBeMhttps://t.co/eYq3Fy0fAS
https://t.co/daXQLipeLv "This is going to turn out to be the biggest cyber incident ever in terms of impact, just a spoiler, as recovery is so difficult," says one expert
Am I reading this right? This news story came out _yesterday_ about how companies are getting sloppier with reviewing major app updates. The story was based on research done to promote a cybersecurity company called……. CrowdStrike https://t.co/hQ85SPEsmz
Crazy visual: 12-hour timelapse shows plane traffic over the US with the FAA grounding Delta, United, and American Airlines flights during this morning's outage pic.twitter.com/KRuL3HjZVf
CrowdStrike CEO is getting pummeled for his response to the global outage.
Why everyone hates it:
1) WEAPONS-GRADE CORPO SPEAK
Let’s be clear. Legalese doublespeak is designed to dodge and obfuscate rather than inform or communicate. This statement was obviously written by a… pic.twitter.com/oLua908QR2
This statement conveys that the CEO thinks you’re overreacting. Everyone calm down; it was only a global outage that took down emergency rooms and the London Stock Exchange.
The smart thing in light of the Crowdstrike global outage is to look not to Crowdstrike, but your own company:
What happens when someone (anyone!) pushes code that passes all internal tests but crashes prod for most customers? When do you discover it? Is it before customers?
Here’s the thing folks. I’ve been coding 32 years. When something like this happens it’s an organizational failure. Yes, some human wrote a bad line. Someone can “git blame” and point to a human and it’s awful. But it’s the testing, the Cl/CD, the A/B testing, the metered…
For those who don't remember, in 2010, McAfee had a colossal glitch with Windows XP that took down a good part of the internet. The man who was McAfee's CTO at that time is now the CEO of Crowdstrike. The McAfee incident cost the company so much they ended up selling to Intel. pic.twitter.com/DgWid6MSK0
Seems legit. Per LinkedIn he worked in 2010 as McAfee CTO and in April 2010 there was a faulty antivirus update that sent WinXP into a reboot loop.https://t.co/2zOyRun6P8https://t.co/hlq45zIWIA
We created a no-prompt bootable ISO with WinPE that auto-deletes the bad crowdstrike file. Then automount to VDI machines and have them boot to it. We've done hundreds this way.
Rebooting 3 and up to 15 or more times is working on a large percentage of machines. It appears that sometimes the network stack is up long enough and crowdstrike update mechanism is able to fix the broken .sys file. Try rebooting over and over and over and over. Seriously.
How we did this in the old days: When I was on Windows, this was the type of thing that greeted you every morning. Every. Single. Morning.
You see, we all had a secondary "debug" PC, and each night we'd run NTStress on all of them, and all the lab machines. NTStress would… pic.twitter.com/rZkvpujbcr
Yup. Config (and input data generally) is just another form of control flow, only it's chunks of your application code that are the control flow primitives. Data format interpreters are not fundamentally different to virtual machine interpreters.
(Others may have mentioned this?) but we find many references "channel files" in @CrowdStrike's patents that provide more insight into their purpose, format, etc.
I worked as a Linux distro dev 25 years ago. We tried migrating the whole company to our OS, but our core business functions like sales & HR could not work so we switched them all back. Even if you migrated only servers, business users would still be on Windows. Stop fantasizing.
— Katie🌻Moussouris (she/her/she-ra/she-hulk) (@k8em0) July 20, 2024
So am I right in thinking the whole thing about that .sys file not being a kernel driver is bullshit because it’s a configuration file for a kernel driver?
IMHO the root of today's kernel issues with Windows go back to Windows NT 3.51. Then MS isolated the kernel from 3rd party drivers but the result was games, printer drivers, and AV sw stopped working & would have to be rewritten. So MS relented & changed architecture with NT 4.0
— Briain Ó hEoghanáin (Brian Honan) #BLM He/Him (@BrianHonan) July 22, 2024
Complex systems fail in complex ways.
Someone asked for my take on the root cause of the @CrowdStrike debacle.
Looking at this from the lens of systems engineering, I conclude that there's not so much a root cause as there is a cascading set of causes:
Sixty years ago today, IBM announced the System/360, a line of computers that took over the computer industry. The big idea was that all the systems were compatible and supported all applications (a 360° range from business to scientific). 1/9 pic.twitter.com/rWydq4zmrX
If you have e/acc in your bio, I will not engage with you; it is a waste of my time, not unlike trying to have a rational conversation with a MAGA supporter.