How Do You Solve a Problem Like AI Regulation?

As with all tools indicated for medical use, those enabled by artificial intelligence (AI) and machine learning (ML) require oversight and regulation to ensure quality and patient safety. But regulating such a dynamic technological landscape is easier said than done, especially without quashing innovation or potential. With these technologies highlighting the constraints of current frameworks—and changes likely on the way, what do clinical lab professionals need to know about the FDA’s regulation of AI and ML to ensure that they, and the tools they use, are compliant both now and in the future?

New technologies, old frameworks

It’s often said that regulation lags behind innovation. There are perhaps few better examples of this than AI’s rapid evolution juxtaposed against slow-to-adapt regulatory frameworks.

“The regulations currently governing AI/ML-based medical devices, including those used in clinical laboratories, are the general medical device regulations,” explains Christine Bump, founder and principal regulatory attorney at Penn Avenue Law & Policy. “As such, AI/ML intended for medical purposes—such as in diagnosis, disease prediction/prognosis, or some types of clinical decision support—are considered devices by the FDA. The existing medical device regulations do not generally allow for devices to be frequently modified or improved. The FDA admits that the traditional paradigm of medical device regulation does not work for AI/ML, even if the agency needs to review these technologies.”

To illustrate, she points to two of the FDA’s premarket submission routes for medical devices, the 510(k) and the premarket approval (PMA) pathways. Manufacturers of 510(k)-cleared devices, demonstrated to be substantially equivalent to a predicate device,¹ must submit a new 510(k) for changes that alter the device’s safety, effectiveness, or intended use. Such changes can include those that control mechanisms or operating principles. Similarly, for PMA-cleared devices whose safety and effectiveness have been demonstrated through scientific evidence, manufacturers must submit supplements for modifications that impact safety or effectiveness. This can include changes to labeling, design specifications, or quality control. Many of these supplements must be reviewed and approved by the FDA before the manufacturer can implement the change to the device and, during this process, additional testing and evidence is sometimes required. 

The problem? These pathways are simply not equipped to deal with the evolving nature of AI or the sheer frequency and consistency of changes that devices powered by these technologies can undergo as a result. “This framework is ill-suited for software and AI/ML, which can make these types of changes constantly,” says Bump. “Unless locked, AI/ML, by its very nature, is constantly learning and changing. Such changes cannot be paused while a new FDA submission is prepared and reviewed. Thus, the FDA is adapting its review process to enable development, adoption, and use of AI/ML while still trying to ensure that these technologies are appropriate, safe, and effective for their intended uses.”

Concerns in clearing devices

Over the past decade, the FDA has cleared, authorized, or approved close to 950 AI/ML-enabled devices.² More than 80 percent of these have been in radiology, but the remainder include devices cleared for pathology, hematology, and clinical chemistry purposes. Currently, under these individual clearances, authorizations, and approvals, the agency works with the manufacturer to handle future changes and modifications—but, as Bump explains, change appears to be on the horizon.

“The FDA is now working generally to develop a new framework tailored to AI/ML,” she says. “In 2019, the agency sought stakeholder feedback on a proposed regulatory framework for modifications to AI/ML-based software as a medical device (SaMD),³ and in 2023, released draft guidance specific to predetermined change control plans (PCCPs) for AI/ML-enabled device software functions.⁴ That draft guidance proposes recommendations for ML-enabled functions that do not need to be explicitly programmed and for which modifications can be made automatically or by human input.”

Beyond this, Bump has other concerns regarding these technologies. “In addition to the number and rates of modifications inherent in AI/ML, these technologies present issues related to potential bias, lack of transparency, quantification of uncertainty, and performance assessments that are not common in other medical devices,” she explains. “The device regulations are not the most appropriate regulatory framework to address these unique concerns. Alongside not allowing for the number and rate of changes inherent in AI/ML, they cannot generally be applied to ‘black box’ algorithms that are not transparent. The FDA needs to develop new regulatory frameworks, but must do so within the authority granted by Congress. We will see more guidance documents before new regulations are promulgated.”

An additional concern is that FDA-approved AI/ML devices often lack clinical validation of their safety and efficacy, as highlighted by a recent study.⁵

The future of frameworks

So, what does the future of AI/ML-enabled device regulation look like and what should labs do to stay informed and compliant? The key to laboratory AI compliance is ensuring familiarity with the existing medical device regulations and monitoring the FDA’s guidance and discussion papers to stay up to date on the agency’s latest thinking about AI/ML technologies. “Laboratories can monitor updates from FDA’s Digital Health Center of Excellence⁶ and also receive alerts if FDA issues proposed rules, drafts, or final guidance regarding AI/ML by signing up for Federal Register notifications and email notifications from the FDA regarding medical devices, digital health, and news and events,” says Bump. She also advises that laboratories and other stakeholders submit comments to any draft guidance or proposed rules to share their thoughts on how the FDA should regulate AI/ML and provide information to the agency about the issues labs face.

But regulatory change is coming, and Bump believes that it will allow AI/ML technologies to realize their full potential. “The medical device regulations simply don’t work,” she says. “The FDA is seriously considering how to address transparency, explainability, bias, cybersecurity, and quality assurance for AI/ML. We will likely see a series of guidance documents related to these specific issues before the agency proposes new regulations. Most of my work with this technology has been in the genomics space, where AI/ML enables identification of hidden patterns in complex datasets. There is so much potential for using AI/ML to improve diagnosis, disease prediction, and laboratory efficiency, as well as to minimize laboratory errors, which is exciting. We’re at the point currently where AI/ML technology is ahead of regulations—so new regulatory frameworks are necessary to address the FDA’s concerns without impeding innovation.”

References:

1. 1. U.S. Food and Drug Administration. How to Study and Market Your Device. August 7, 2023. https://www.fda.gov/medical-devices/device-advice-comprehensive-regulatory-assistance/how-study-and-market-your-device
   2. U.S. Food and Drug Administration. Artificial Intelligence and Machine Learning (AI/ML)-Enabled Medical Devices. https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-aiml-enabled-medical-devices.
   3. U.S. Food and Drug Administration. Proposed Regulatory Framework for Modifications to Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD). April 2, 2019. https://www.fda.gov/media/122535/download.
   4. U.S. Food and Drug Administration. Marketing Submission Recommendations for a Predetermined Change Control Plan for Artificial Intelligence/Machine Learning (AI/ML)-Enabled Device Software Functions. April 3, 2023. https://www.fda.gov/media/166704/download.
   5. https://www.g2intelligence.com/has-your-ai-been-validated/
   6. U.S. Food and Drug Administration. Digital Health Center of Excellence. September 9, 2024. https://www.fda.gov/medical-devices/digital-health-center-excellence.