Blue Screen Saga: CrowdStrike and Microsoft Outage Sparks Global Tech Crisis!

Dear Readers,

We hope this newsletter finds you well. This month, the tech world was rocked by a significant outage caused by an unforeseen glitch involving CrowdStrike and Microsoft.

What Happened?

On July 19, 2024 at 04:09 UTC, as part of ongoing operations, CrowdStrike released a sensor configuration update to Windows systems. Sensor configuration updates are an ongoing part of the protection mechanisms of the Falcon platform. This configuration update triggered a logic error resulting in a system crash and blue screen (BSOD) on impacted systems.

The sensor configuration update that caused the system crash was remediated on Friday, July 19, 2024 05:27 UTC.

This issue is not the result of or related to a cyberattack.

Impact

Customers running Falcon sensor for Windows version 7.11 and above, that were online between Friday, July 19, 2024 04:09 UTC and Friday, July 19, 2024 05:27 UTC, may be impacted. 

Systems running Falcon sensors for Windows 7.11 and above that downloaded the updated configuration from 04:09 UTC to 05:27 UTC were susceptible to a system crash.

Configuration File Primer

The configuration files mentioned above are referred to as Channel Files and are part of the behavioural protection mechanisms used by the Falcon sensor. Updates to Channel Files are a normal part of the sensor’s operation and occur several times daily in response to novel tactics, techniques, and procedures discovered by CrowdStrike. This is not a new process; the architecture has existed since Falcon’s inception.

Technical Details

On Windows systems, Channel Files reside in the following directory:

C:\Windows\System32\drivers\CrowdStrike\

and have a file name that starts with “C-”. Each channel file is assigned a number as a unique identifier. The impacted Channel File in this event is 291 and will have a filename that starts with “C-00000291-” and ends with a .sys extension. Although Channel Files end with the SYS extension, they are not kernel drivers.

Channel File 291 controls how Falcon evaluates named pipe1 execution on Windows systems. Named pipes are used for normal, interprocess or intersystem communication in Windows.

The update that occurred at 04:09 UTC was designed to target newly observed, malicious-named pipes being used by common C2 frameworks in cyberattacks. The configuration update triggered a logic error that resulted in an operating system crash. 

Channel File 291

CrowdStrike has corrected the logic error by updating the content in Channel File 291. No additional changes to Channel File 291 beyond the updated logic will be deployed. Falcon is still evaluating and protecting against the abuse of named pipes. 

This is not related to null bytes contained within Channel File 291 or any other Channel File. 

Here are some (initial) details about why the CrowdStrike's CSAgent.sys crashed Faulting inst: mov r9d, [r8]

R8: unmapped address

...taken from an array of pointers (held in RAX), index RDX (0x14 * 0x8) holds the invalid memory address.

The other "drivers" (e.g. 'C-00000291-...32.sys') appear to be obfuscated data ...and are x-ref'd (perhaps ingested?) by CSAgent.sys ...so maybe invalid (config/signature) data triggered the fault in CSAgent.sys This would be easier to tell/confirm via debugging 😉 

A big outstanding questions is; what are the 'C-00000291-...xxx.sys' files? As deleting them fixes the crash, this seems imply their contents matter (as its CSAgent.sys that has references to them, that is crashing).

“The .sys files causing the issue are channel update files, they cause the top level CS driver to crash as they're invalidly formatted. It's unclear how/why Crowdstrike delivered the files and I'd pause all Crowdstrikes updates temporarily until they can explain. This is going to turn out to be the biggest 'cyber' incident ever in terms of impact, just a spoiler, as recovery is so difficult.”--Kevin Beaumont

Note "channel updates ...bypassed client's staging controls and was rolled out to everyone regardless" ….

A few IT folks who had set the CS policy to ignore latest version confirmed this was, ya, bypassed, as this was "content" update (vs. a version update)

An update from CrowdStrike confirms the analysis:

Namely: The C-...sys files aren't kernel drivers, but rather are "configuration files" dubbed "Channel Files" C-00000291- "triggered a logic error that resulted in an OS crash" (via CSAgent.sys)

we find many references "channel files" in CrowdStrike’s patents that provide more insight into their purpose, format, etc. Search: "channel file" assignee:(Crowdstrike, Inc.) For example in US11822515B2 & US11645397B2:

A Witty Response

In the world of programming, why play with C++ pointers when you can rust-proof your code? Say goodbye to those pesky null pointers and hello to a safer, more efficient future with Rust! It’s like swapping out your flimsy umbrella for a high-tech weather shield.