What caused the big CrowdStrike outage this week ?

We reported yesterday that a massive world wide IT outage caused chaos for many people. Initially Microsoft got the blame, as many PC’s and servers started to get a blue screen errors, it was estimated that 8.5 million Microsoft Windows devices were affected. The fault was traced to Falcon sensor software provided by the cyber security company CrowdStrike.

This outage was BIG it caused problems around the world, here in the UK one of the biggest news channels Sky News was unable to broadcast its breakfast show, other TV channels were also taken offline. The outage also took down the London Stock Exchange, and caused chaos across the NHS specifically at GP surgeries who used the EMIS clinical system, even now a day later many GP sites still report that the system is not fully fixed.

In the US, state troopers were reporting that many 911 call centres were down, some hospitals also had to cancel appointments. Around the world airports were badly hit with many reporting problems with their scanning technology, which couldn’t register passengers’ boarding passes.

Despite CrowdStrike resolving the initial cause of the outage, many services and businesses, including hospitals and flights, continue to be affected.

Microsoft said they are collaborating with other cloud providers and stakeholders, including Google Cloud Platform (GCP) and Amazon Web Services (AWS), to share awareness on the state of impact they are each seeing across the industry and inform ongoing conversations with CrowdStrike and customers. 

CrowdStrike engineering identified a content deployment related to this issue and reverted those changes as quick as they could, but it was to late for many people as the software had already auto updated.

The workaround is straightforward, but it is not scalable because it must be manually applied to each system individually.

In a letter posted online to customers and partners late Friday, CrowdStrike CEO George Kurtz stated that he wanted to “sincerely apologize directly to all of you for today’s outage.”

The most up-to-date remediation recommendations and information can be found on the CrowdStrike blog or in the Support Portal. 

Timeline of events

On July 19, 2024 at 04:09 UTC, as part of ongoing operations, CrowdStrike released a sensor configuration update to Windows systems. Sensor configuration updates are an ongoing part of the protection mechanisms of the Falcon platform. This configuration update triggered a logic error resulting in a system crash and blue screen (BSOD) on impacted systems. The sensor configuration update that caused the system crash was remediated on Friday, July 19, 2024 05:27 UTC.

Customers running Falcon sensor for Windows version 7.11 and above, that were online between Friday, July 19, 2024 04:09 UTC and Friday, July 19, 2024 05:27 UTC, may be impacted. 

Systems running Falcon sensor for Windows 7.11 and above that downloaded the updated configuration from 04:09 UTC to 05:27 UTC – were susceptible to a system crash.

Technical details

On Windows systems, Channel Files reside in the following directory:

C:\Windows\System32\drivers\CrowdStrike\

and have a file name that starts with “C-”. Each channel file is assigned a number as a unique identifier. The impacted Channel File in this event is 291 and will have a filename that starts with “C-00000291-” and ends with a .sys extension. Although Channel Files end with the SYS extension, they are not kernel drivers.

Channel File 291 controls how Falcon evaluates named pipe execution on Windows systems. Named pipes are used for normal, interprocess or intersystem communication in Windows.

The update that occurred at 04:09 UTC was designed to target newly observed, malicious named pipes being used by common C2 frameworks in cyberattacks. The configuration update triggered a logic error that resulted in an operating system crash.