Most affected organisations have found out the hard way why out of band management is important.
Job 1 in repairing CrowdStrike.. get access to computer. https://t.co/zHsl0zw2Tq pic.twitter.com/g8tNIK42s4
— techAU (@techAU) July 21, 2024
“Do I have the Crowdstrike?” as a question in the public domain.
— Maaike (on 🟦 🌫️) (@iktwiet) July 20, 2024
I think there is such a thing as bad publicity.
And this is it.
Being (unconsciously) labelled as a virus as a security company. One would think that’s not advantageous. https://t.co/F94YoKv8uR
Build secure out-of-band communication into your incident response plan.
— Whitney Merrill (@wbm312) July 19, 2024
Crowdstrike fix for blue screen issue
— SANS.edu Internet Storm Center (@sans_isc) July 19, 2024
Workaround Steps:
1Boot Windows into Safe Mode or the Windows Recovery Environment
2Navigate to the C:WindowsSystem32driversCrowdStrike directory
3Locate the file matching “C-00000291*.sys”, and delete it.
4Boot the host
It started on a Thursday USA time
I'm in awe of the scale of the Crowdstrike / Windows BSOD issue.
— Gene Kim (@RealGeneKim) July 19, 2024
Here are the most startling images I've seen morning.
Let's start with this: at 10pm PT yesterday, famous @troyhunt notices that something odd is happening to Windows systems:https://t.co/0KWDELUycT
The potentially faulty Crowdstrike CSagent.sys hit VT last night. Compiled on July 9th. https://t.co/yyfY0v5dJk pic.twitter.com/aHO1AGSXSp
— Costin Raiu (@craiu) July 19, 2024
For years to come, the IT admin team will bring this up whenever you ask them to install another agent on the endpoints#CrashStrike #CrowdStrike
— Florian Roth (@cyb3rops) July 19, 2024
Note "channel updates …bypassed client's staging controls and was rolled out to everyone regardless" https://t.co/UecaAmJdqc
— Patrick Wardle (@patrickwardle) July 19, 2024
A few IT folks who had set the CS policy to ignore latest version confirmed this was, ya, bypassed, as this was "content" update (vs. a version update)
"They pushed a new kernel driver out to every client without authorization to fix an issue with slowness and latency that was in the previos Falcon sensor product. They pissed over everyone's staging and rules and just pushed this to production"https://t.co/XVEJoLTBeM https://t.co/eYq3Fy0fAS
— 🦆 SchizoDuckie 🦆 (@SchizoDuckie) July 19, 2024
https://t.co/daXQLipeLv "This is going to turn out to be the biggest cyber incident ever in terms of impact, just a spoiler, as recovery is so difficult," says one expert
— The Register (@TheRegister) July 19, 2024
Am I reading this right? This news story came out _yesterday_ about how companies are getting sloppier with reviewing major app updates. The story was based on research done to promote a cybersecurity company called……. CrowdStrike https://t.co/hQ85SPEsmz
— Tom Rivlin (@TomRivlin) July 19, 2024
Crazy visual: 12-hour timelapse shows plane traffic over the US with the FAA grounding Delta, United, and American Airlines flights during this morning's outage pic.twitter.com/KRuL3HjZVf
— Morning Brew ☕️ (@MorningBrew) July 19, 2024
CrowdStrike CEO is getting pummeled for his response to the global outage.
— Lulu Cheng Meservey (@lulumeservey) July 19, 2024
Why everyone hates it:
1) WEAPONS-GRADE CORPO SPEAK
Let’s be clear. Legalese doublespeak is designed to dodge and obfuscate rather than inform or communicate. This statement was obviously written by a… pic.twitter.com/oLua908QR2
6) WOEFULLY INSUFFICIENT SH*TS GIVEN
— Lulu Cheng Meservey (@lulumeservey) July 19, 2024
This statement conveys that the CEO thinks you’re overreacting. Everyone calm down; it was only a global outage that took down emergency rooms and the London Stock Exchange.
In fact, not even that — it was “a defect found in a single… pic.twitter.com/5qDg8qXAJL
The fun of doing risk analyses. pic.twitter.com/TEDFAvATay
— Queen Fennec (@Queen_fennec) July 19, 2024
The smart thing in light of the Crowdstrike global outage is to look not to Crowdstrike, but your own company:
— Gergely Orosz (@GergelyOrosz) July 20, 2024
What happens when someone (anyone!) pushes code that passes all internal tests but crashes prod for most customers? When do you discover it? Is it before customers?
Here’s the thing folks. I’ve been coding 32 years. When something like this happens it’s an organizational failure. Yes, some human wrote a bad line. Someone can “git blame” and point to a human and it’s awful. But it’s the testing, the Cl/CD, the A/B testing, the metered…
— Scott Hanselman 🌮 (@shanselman) July 20, 2024
For those who don't remember, in 2010, McAfee had a colossal glitch with Windows XP that took down a good part of the internet. The man who was McAfee's CTO at that time is now the CEO of Crowdstrike. The McAfee incident cost the company so much they ended up selling to Intel. pic.twitter.com/DgWid6MSK0
— Anshel Sag (@anshelsag) July 19, 2024
Seems legit. Per LinkedIn he worked in 2010 as McAfee CTO and in April 2010 there was a faulty antivirus update that sent WinXP into a reboot loop.https://t.co/2zOyRun6P8https://t.co/hlq45zIWIA
— biased estimator (@biasedestimator) July 20, 2024
We created a no-prompt bootable ISO with WinPE that auto-deletes the bad crowdstrike file. Then automount to VDI machines and have them boot to it. We've done hundreds this way.
— Brooks Peppin (@brookspeppin) July 19, 2024
Rebooting 3 and up to 15 or more times is working on a large percentage of machines. It appears that sometimes the network stack is up long enough and crowdstrike update mechanism is able to fix the broken .sys file. Try rebooting over and over and over and over. Seriously.
— A:a.ron (@_aarony) July 19, 2024
How we did this in the old days:
— Dave W Plummer (@davepl1968) July 20, 2024
When I was on Windows, this was the type of thing that greeted you every morning. Every. Single. Morning.
You see, we all had a secondary "debug" PC, and each night we'd run NTStress on all of them, and all the lab machines. NTStress would… pic.twitter.com/rZkvpujbcr
Don’t worry. #Crowdstrike did the same thing to RedHat Linux last month https://t.co/ljDNuj3wdt caused kernel panic https://t.co/RzttnmYnLN
— Scott Hanselman 🌮 (@shanselman) July 20, 2024
— Scott Hanselman 🌮 (@shanselman) July 20, 2024
Have you ever seen a Wiki created with 165 references about a security incident, within 12 hours?https://t.co/lPV4I2eTlh pic.twitter.com/xEF9J2Rg8B
— Packymancard (@packymancard) July 19, 2024
Yup. Config (and input data generally) is just another form of control flow, only it's chunks of your application code that are the control flow primitives. Data format interpreters are not fundamentally different to virtual machine interpreters.
— Barry Kelly (@barrkel) July 21, 2024
(Others may have mentioned this?) but we find many references "channel files" in @CrowdStrike's patents that provide more insight into their purpose, format, etc.
— Patrick Wardle (@patrickwardle) July 21, 2024
Search:
"channel file" assignee:(Crowdstrike, Inc.)
For example in US11822515B2 & US11645397B2: pic.twitter.com/cGMWADBe3x
I worked as a Linux distro dev 25 years ago. We tried migrating the whole company to our OS, but our core business functions like sales & HR could not work so we switched them all back.
— Katie🌻Moussouris (she/her/she-ra/she-hulk) (@k8em0) July 20, 2024
Even if you migrated only servers, business users would still be on Windows.
Stop fantasizing.
So am I right in thinking the whole thing about that .sys file not being a kernel driver is bullshit because it’s a configuration file for a kernel driver?
— Mark Rendle 🇺🇦 (@markrendle) July 20, 2024
The potential attack surfaces for 3rd party windows kernel drivers is massive
All these third-party AV/EDR kernel drivers. Now if this isn't a nice attack surface… https://t.co/4kpdps3gXt pic.twitter.com/FbtZhBcGr3
— x0rz (@x0rz) July 21, 2024
IMHO the root of today's kernel issues with Windows go back to Windows NT 3.51. Then MS isolated the kernel from 3rd party drivers but the result was games, printer drivers, and AV sw stopped working & would have to be rewritten. So MS relented & changed architecture with NT 4.0
— Briain Ó hEoghanáin (Brian Honan) #BLM He/Him (@BrianHonan) July 22, 2024
Complex systems fail in complex ways.
— Grady Booch (@Grady_Booch) July 22, 2024
Someone asked for my take on the root cause of the @CrowdStrike debacle.
Looking at this from the lens of systems engineering, I conclude that there's not so much a root cause as there is a cascading set of causes:
There clearly a…
crowdstrike conspiracy theories pic.twitter.com/C6TiMK8C1Y
— Forrest Brazeal (@forrestbrazeal) July 25, 2024
Rate this:
Share this:
- Click to share on Mastodon (Opens in new window) Mastodon
- Click to share on Bluesky (Opens in new window) Bluesky
- Share on Tumblr
- Click to share on Reddit (Opens in new window) Reddit
- Click to share on Threads (Opens in new window) Threads
- Tweet
- Click to share on Telegram (Opens in new window) Telegram
- Click to share on Nextdoor (Opens in new window) Nextdoor
- Click to share on WhatsApp (Opens in new window) WhatsApp
- Click to print (Opens in new window) Print
- Click to email a link to a friend (Opens in new window) Email
Earlier this week there were only three sponsors for [
The Wiert Corner - irregular stream of stuff 