FDA wants public input before regulating AI-enabled medical tools

The Food and Drug Administration is mulling over the future of generative artificial intelligence-enabled medical devices.

In its discussion paper, released Tuesday, the FDA’s Center for Devices and Radiological Health says it’s not proposing or implementing policy changes, nor is it solidifying regulatory expectations, but is instead seeking early input to advance discussion among stakeholders.

Its ultimate goal is to “enable a nimble regulatory approach that employs least burdensome principles and allows patients timely access to safe and effective medical devices,” the paper reads.

Specifically, the agency is looking for feedback on risk assessment, premarket evaluation and postmarket monitoring of GenAI-enabled medical devices.

The FDA said that while genAI-enabled medical devices hold “transformative promise for patient care and the broader health ecosystem,” they may introduce “unique risks when compared to traditional software and AI-enabled medical devices.”

Its paper flagged multiple risks: confabulations or hallucinations that appear authentic, uncertainty about intended-use boundaries, performance degradation over time and limited insight into how the underlying AI models work.

The FDA floated a voluntary program where AI model makers could confidentially file model cards or system cards with the agency that device makers building on their models could reference. The agency explicitly asked whether such a voluntary program makes sense, given model developers “may have limited incentive to disclose safety-relevant information.”

Rather than testing every possible input, the FDA is also considering a different approach modeled on how doctors are licensed, including device benchmarking with standardized tests and realistic patient cases, before the AI model is trusted to work on its own.

Tension between democratizing patient access to clinical AI information and patients’ evaluation of whether AI is wrong is repeatedly flagged in the document.

The FDA draws a line between AI advice given directly to patients and AI advice given to doctors, noting patients don’t have the training to interpret it, spot its limits or catch a wrong answer before acting on it.

On the other hand, treating all patient-facing AI as automatically riskier could backfire by potentially blocking tools that help patients access useful health information and take a more active role in their own care.

This could work against the benefits of “democratization of clinical information” and “patient empowerment and engagement,” according to the discussion paper.

The paper, led by the FDA’s Digital Health Center of Excellence, aims to advance regulatory frameworks for AI and digital health technologies.

“Artificial intelligence is transforming medicine, and the United States must lead in shaping how this technology is developed and used safely and responsibly,” acting FDA Commissioner Kyle Diamantas said in a statement. “Today’s announcement reflects the FDA’s commitment to advancing innovation for health care professionals and leveraging AI to improve care and patient health outcomes.”

The director of the Digital Health Center of Excellence, Rick Abramson, said in a statement that because genAI-enabled medical devices are “poised to reshape the health technology landscape,” the FDA has a “responsibility to provide thoughtful leadership for this new era.