By Rob Foxx
On July 19th, 2024, CrowdStrike pushed out an update that disabled an estimated 8.5 million computer systems across the world to their customer base. This not only damaged their reputation, but it also sent fear and uncertainty into those businesses that utilize IT services. This single event cost about 5.5 billion in damages and business loss.
The real question is who can be affected by an event like this. The short answer is anyone who uses any products or services that can affect the system files of a computer. This includes Microsoft Windows automated updates, antivirus, remote configuration tools, and many others. For those who are not familiar with the previously mentioned list you have several of these on any given business computer.
So why do we not hear about events like this more often? The reality of the situation is these events happen every day. Windows updates pushed out by a managed service provider that removes all network drivers thus disabling the connection to the login service rendering a computer useless was my personal CrowdStrike. This happened several times over the course of a month. Anti-virus similar CrowdStrike that disables use of business-critical software or core access. An update to Windows that prevents backup software from running correctly. These have all happened. Most people have never heard of these events as they did not have a worldwide impact or cost billions of dollars in losses. CrowdStrike made news because of their wide adoption as they offer a service only a handful of other companies do.
So how do I protect my organization? The only 100% means of protecting your computer environment is to power it down, unplug it, and bury it in concrete. Business, however, runs on acceptable risk so let us look at what could help lower your risk and impact. During the course of onboarding or reviewing a vendor for contract renewal you can verify or request that they add in testing criteria before sending out updates. Comparing vendor services with an RFP (Request for proposal) can identify if a service fits your needs. Several similar companies allow for the designation of a test group which includes a sampling of computers across your organization. This group should fill all the key roles to running your enterprise and test for a week or more before the updates are pushed to the rest. Perhaps one of the best means to protect yourself is to have knowledgeable IT on staff or as an on-call vendor. The fix to restore CrowdStrike would take per computer less than 10 minutes at a keyboard. A little longer if the computer has BitLocker or other similar protections in place. A good recovery plan and continuity for running on alternate systems can also save quite a bit of recovery time.
I do believe there will be more events like the CrowdStrike outage. Especially given how large some services grow to global offerings. I believe vendors will be more cautious in their products. Will you be more cautious in your implementations?
Foxx is director – infosec and IT audit services for FIPCO. He can be reached at rfoxx@fipco.com or 608-441-1249.
For more information about FIPCO forms, software, or other products, visit fipco.com, call 800-723-3498, or email fipcosales@fipco.com
FIPCO is a WBA Gold Associate Member.