Impacts range from airlines grounding flights to hospitals canceling non-emergency procedures. Several states are reportedly losing 911 services as well, while people in Australia and New Zealand have said on social media that they can’t access online bank accounts. Some stores also warned customers they could not process credit cards.
State and local governments are addressing the crisis. In an update to residents, New York City Mayor Eric Adams said the city has been working on restoring services and security. He expected the city would see “cascading effects of the outage throughout the day,” but assured residents that 911, water systems and traffic signals were unaffected. Rhode Island Gov. Dan McKee said on X that “some state computer systems” were impacted, but 911 remained operational.
The trouble stems from cybersecurity company CrowdStrike issuing a faulty software update that crashed Windows machines that were running it.
According to Microsoft Azure’s website, “We have been made aware of an issue impacting Virtual Machines running Windows Client and Windows Server, running the CrowdStrike Falcon agent, which may encounter a bug check (BSOD) and get stuck in a restarting state.” New York City CTO Matthew Fraser said in a press release that the faulty patch was pushed out between midnight and 1:30 a.m., at which point CrowdStrike was notified of the issues and stopped deploying the patch.
CrowdStrike said the problem is related to its Falcon Sensorproduct, which is intended to detect and block cyber attacks. The product is cloud-based and receives automatic updates. But in this case, a “defect” in just one content update for Windows caused the problems, CrowdStrike said. Mac and Linux were unimpacted, as were machines running Windows 7/2008 R2.
“This is not a security incident or cyber attack,” CrowdStrike wrote.
Cybersecurity journalist Brian Krebsnoted that “like most security software, CrowdStrike requires deep hooks into the Windows operating system to fend off digital intruders, and in that environment a tiny coding error can quickly lead to catastrophic outcomes.”
New York CTO Fraser said that cybersecurity software needs to be able to update in real time to respond to continually evolving threats — but the side effect is that “if a patch goes wrong” it creates dramatic effects. But New York’s critical systems, like 911 and 311, were protected from the bad patch because the city keeps them isolated in a separate environment with automatic updates only allowed to go through during certain periods, after the city tests the updates in a sandbox environment.
CrowdStrike also tests its updates before sending them out, but the company found that apparently “something changed or something got corrupted” between this update’s testing and deployment, Fraser said. CrowdStrike has now revoked the bad update, Microsoft reported, and the cybersecurity company said it found the problem and issued a fix.
But the fix cannot be applied automatically, according to cybersecurity company CyberArk’s CIO, Omer Grossman, who spoke to CNBC. Because the glitch crashed endpoints, they cannot be remotely updated, and each individual endpoint needs to be worked on manually. As such, he said he expected the process to take days.
Microsoft is exploring options that Azure customers can take to mitigate the effects. It also recommends customers try to restore from backups predating the software update’s rollout. Microsoft provided additional advice here.
But customers may need to turn to CrowdStrike for further help. CrowdStrike posted advice here, recommending “organizations ensure they’re communicating with CrowdStrike representatives through official channels.”
