Key Lessons for Clinical Labs from the CrowdStrike Outage

From the fears surrounding the Y2K bug at the start of the millennium to those concerning the yet-to-be-addressed Year 2038 problem, worries about the potential disruption a widespread technological outage can cause are pervasive. Earlier this year, the world experienced a glimpse of this through the CrowdStrike outage, dubbed the “largest IT outage in history.”¹ Although the causative error was resolved within hours, the impacts were far-reaching. Millions of systems worldwide were incapacitated, leading to an estimated tens of billions of dollars in global damages² and bringing many sectors to a standstill—including laboratory medicine. But how do outages like this happen? How do they affect the clinical lab? And how can lab leaders prepare for—and navigate—such crises?

Every outage has its origin

“In my 30-year laboratory medicine career, I’ve come across many forms of outages that can leave clinical labs without access to digital services,” says Giuseppe Lippi, who chairs the European Federation of Clinical Chemistry and Laboratory Medicine (EFLM)’s task force on lab emergencies. In Lippi’s experience, although such outages can arise from a variety of sources, they are broadly divided into two groups: internal and external. Internal outages commonly occur when a lab’s computers, middleware, servers, network, or software fail. “Computer or middleware failures are the easiest to resolve; usually, replacing hardware and reinstalling programs will restore function,” he says. “They typically take up to a day or so to fix and only affect part of your activity.”

When an outage prevents the flow of information within a hospital or clinical lab, the issue can be more disruptive. “Physical servers can stop working, similar to a home PC. Because these servers typically handle vast volumes of information, their loss can be serious. Although most hospitals and laboratories have ‘backup’ servers, it can take hours before all functions can be reactivated—and sometimes backup servers may not start properly,” Lippi explains. Network interruptions, caused by router failures, bandwidth spikes, or physical damage to cables, can cause similar issues.

Most concerning are outages originating outside the clinical lab—like CrowdStrike’s distribution of a faulty software update. Because external outages often require external solutions, labs may be left stranded with few options while they wait for updates—something Lippi, who is also director of laboratory medicine at the University Hospital of Verona, has experienced firsthand. “My hospital was the victim of a disruptive cyberattack almost nine months ago,” he says. “Alongside cyberterrorism events like this—an increasingly common occurrence in healthcare—regional networks storing patient data can be disrupted. In such cases, you can only wait until external intervention solves the problem; there’s very little you or your hospital IT staff can do.” For labs affected by the CrowdStrike outage, this vulnerability may have been exacerbated by its relatively uncommon origin. “Outages that occur due to external software failures such as CrowdStrike are relatively new,” says Lippi. “They haven’t previously been considered a potential cause of healthcare failure, but are nonetheless comparable to the most disruptive cyberattacks.”

The outcome of an outage

Like the potential sources, the damages from a technological outage can vary. For clinical labs, the impact severity can depend on which systems the outage disables. “Hospital information system (HIS) failure is easier for the clinical lab to deal with; instrumentation should still be connected to the laboratory information system (LIS) and maintain function of query host if requests are manually entered into the LIS. Losing LIS access can cause more problems,” Lippi says. However, even if some systems remain accessible during an outage, Lippi is quick to caution that working without access to even one of its systems can greatly increase the lab’s risk of error.

In the most severe cases, labs’ only option to maintain operations during the outage and recovery is to revert to paper-based systems. “When all IT equipment fails, you’re basically forced to work the way things were done 40–50 years ago when everything was manual,” Lippi explains. “Blood tubes must be manually labeled and transported to the lab, instruments manually programmed, test results faxed (assuming you still have working fax machines in your facility) or hand-delivered on paper sheets, and urgent tests or critical results communicated by telephone.”

Prepare to go paper-based

What lessons can laboratorians learn from the CrowdStrike incident? For Lippi, outages like this are warnings to ensure that preparations are in place. “After the cyberattack on my hospital, five other Italian hospitals became cyberattack victims,” he explains. “We always think these incidents could only affect others until they affect us. If IT problems affect someone you know, be aware that your risk of being impacted is just as great.”

Regarding precautionary steps to take, establishing a policy for such IT disturbances is crucial. Lippi offers the following things to consider when formulating such a policy:

• • Train all hospital staff in cyberattack risk reduction. This includes applying diversified precautions for institutional access (changing passwords at least quarterly, using complex passwords, and enabling multifactor access) and installing efficient and updated firewalls and antivirus software.

• • Develop a parallel HIS/LIS/network. This can be kept offline and immediately reactivated when needed to prevent disruptive consequences such as those of a cyberattack or complete software failure.

• • Establish a response team to coordinate operations. “This is what we did in the four hours after the complete IT failure at my hospital. The orders must always come from a single (hierarchically superior) team to prevent chaos.”

• Creating a recovery plan. Recommendations from the EFLM Task Force include preparing emergency request forms stored in a special folder (accessible even when networks are down) on all hospital PCs, alongside several printed copies kept in each ward so that physicians can still order the tests their patients need.³ Similarly accessible should be emergency lab reports for transcribing patient information and test results to be sent back to the ordering physician. “In the event of a HIS/LIS/network failure, we always manually enter all test results performed offline into the LIS so that they aren’t lost and remain part of the patient’s medical history. This can take some time, but is essential. It goes without saying that this is much easier if the patient ID and test results are included in a well-designed emergency lab report.”

Having such a policy in place means that, if a technological outage does occur, labs can adhere to Lippi’s final piece of advice: stay calm and follow the plan. “Don’t panic! At the beginning of the cyberattack we experienced, I saw fear in my employees’ eyes. But we were able to successfully work offline for nearly four days.”

References:

1. 1. B Fung. We finally know what caused the global tech outage – and how much it cost. CNN. July 24, 2024. Available at: https://edition.cnn.com/2024/07/24/tech/crowdstrike-outage-cost-cause/index.html.

1. 1. LK Wee. Here come the wave of insurance claims for the CrowdStrike outage. Business Insider. July 22, 2024. Available at: https://www.businessinsider.com/businesses-claiming-losses-crowdstrike-outage-insurance-billions-losses-cyber-policies-2024-7.

1. 1. G Lippi et al. EFLM Task Force Preparation of Labs for Emergencies (TF-PLE) recommendations for reinforcing cyber-security and managing cyber-attacks in medical laboratories. Clin Chem Lab Med. 2024; online ahead of print. doi:10.1515/cclm-2024-0803.