CrowdStrike internal research — 8.5 million computers crash caused by 40 KB file

CrowdStrike has published a post incident review (PIR) regarding a faulty update that disabled 8.5 million computers. The problem is blamed on a testing program.

Due to a bug, the software did not properly test the content update that was distributed to millions of machines on Friday. CrowdStrike promises to test updates to its products more thoroughly, improve error handling, and introduce a phased rollout to avoid a repeat of the disaster.

CrowdStrike’s Falcon software is used by companies around the world to fight malware and security breaches on millions of Windows computers. On Friday, CrowdStrike released a configuration update for its product that was supposed to «collect telemetry on possible new threat techniques». These updates are shipped regularly, but this particular one caused the Windows failure.

CrowdStrike typically releases configuration updates in two different ways. There is the so-called Sensor Content, which directly updates the CrowdStrike Falcon running at the Windows kernel level. Then there is Rapid Response Content, which updates the behavior to detect malware. A small 40 KB Rapid Response Content file caused Friday’s problem. Last week, CrowdStrike released two Rapid Response Updates — what the company calls pattern instances.

Although CrowdStrike conducts both automated and manual testing, it was still not thorough enough. The rollout of new template types in March provided «confidence in the checks performed in Content Validator», so CrowdStrike seemed to assume that the rollout would not cause problems.

To prevent this from happening again, CrowdStrike promises to improve Rapid Response Content testing through local developer testing, content updates and rollback testing, and stress testing. CrowdStrike will also conduct stability and UI testing of Rapid Response Content and update its cloud-based validation tool.

Source: The Verge