CrowdStrike Update Causes Widespread Windows Crashes: A Crisis Unfolds

In a significant security incident, infosec vendor CrowdStrike has rolled out an update that is causing widespread crashes and rendering Windows machines inoperable. Users around the globe have reported that their Windows 10 PCs are encountering the notorious Blue Screen of Death (BSOD) following the deployment of the Falcon Sensor update.

The Issue

Reports have flooded in about a malfunctioning update to CrowdStrike’s Falcon Sensor, which is designed to block attacks and detect threats on systems. Instead of providing protection, the sensor is now causing critical failures. The problem is specifically linked to the “csagent.sys” file, which is crashing Windows PCs and preventing them from rebooting.

A user reported, “We’re seeing BSOD Org wide that are being caused by csagent.sys, and it’s taking down critical services. I’ll open a ticket, but this is a big deal.” The issue is severe, affecting numerous organizations and critical services.

Official Acknowledgement and Response

CrowdStrike has acknowledged the problem and issued an advisory, which unfortunately is behind a registration wall accessible only to customers. The advisory confirms that the Falcon Sensor is indeed causing blue screen errors on Windows hosts, though Mac and Linux systems remain unaffected.

CrowdStrike’s chief threat hunter, Brody Nisbet, has been actively communicating about the issue on social media platform X. He clarified that the problem stems from a faulty channel file rather than a direct update. Nisbet provided a temporary workaround for affected users:

1. Boot Windows into Safe Mode or the Windows Recovery Environment (WRE).
2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory.
3. Locate and delete the file matching “C-00000291*.sys”.
4. Reboot the machine normally.

Detailed Workaround

The official Tech Alert issued by CrowdStrike offers further instructions for those who are still experiencing crashes despite the company’s efforts to revert the problematic changes:

1. Reboot the host to allow it to download the reverted channel file.
2. If the host crashes again, boot into Safe Mode or WRE.
3. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory.
4. Delete the channel file “C-00000291*.sys” with a timestamp of 0527 UTC or later, which is the good version.

Workaround Steps for Public Cloud or Similar Environments, Including Virtual Machines

Option 1:

1. Detach the operating system disk volume from the impacted virtual server.
2. Create a snapshot or backup of the disk volume before proceeding further as a precaution against unintended changes.
3. Attach/mount the volume to a new virtual server.
4. Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory.
5. Locate the file matching “C-00000291*.sys” and delete it.
6. Detach the volume from the new virtual server.
7. Reattach the fixed volume to the impacted virtual server.

Option 2:

1. Roll back to a snapshot before 0409 UTC.

Workaround Steps for Azure via Serial Console

1. Login to Azure console –> Go to Virtual Machines –> Select the VM.
2. In the upper left of the console –> Click “Connect” –> Click “More ways to Connect” –> Click “Serial Console”.
3. Once SAC has loaded, type in ‘cmd’ and press enter.
4. Type in the following commands:
   • cmd
   • ch -si 1
5. Press any key (space bar). Enter Administrator credentials.
6. Type the following commands:
   • bcdedit /set {current} safeboot minimal
   • bcdedit /set {current} safeboot network
7. Restart the VM.
8. Optional: To confirm the boot state, run the command:
   • wmic COMPUTERSYSTEM GET BootupState

For additional information please see this Microsoft article.

Current Status and Future Actions

CrowdStrike’s engineering team has identified the cause and reverted the changes responsible for the crashes. However, not all hosts are staying online long enough to receive the corrected channel file, making manual intervention necessary for some users.

This incident highlights the critical nature of updates and the cascading effects they can have across systems worldwide. CrowdStrike is actively working on a permanent fix and advises users to follow the provided workaround steps in the interim.

As this is a developing story, further updates will be provided as new information emerges.

Official incident link

Update 2: 11:05 AM GMT+1 : Add workarround Azure and Public Cloud