Provider groups push liability back to AI developers
Hospital, physician and telehealth organizations are drawing a hard line around generative AI medical devices: the clinicians using them should not be the ones carrying primary responsibility for safety, performance or ongoing monitoring. In comments to the Food and Drug Administration, the American Hospital Association, American Medical Association and American Telemedicine Association argued that the burden belongs with developers and manufacturers, not with hospitals, health systems or individual physicians.
The issue reaches far beyond legal housekeeping. Generative AI systems can accept open-ended prompts, generate different answers to similar inputs and produce recommendations that may change over time as models are updated. In healthcare, those traits create a different regulatory problem than traditional software tools, especially when the output may influence diagnosis, triage, documentation or treatment decisions. If a device can behave differently with each use, then post-market oversight cannot rely on the same assumptions used for static medical software.
A central concern for providers is practical capacity. Continuous monitoring of generative AI would add another surveillance layer to organizations already stretched by staffing shortages, quality reporting and existing technology oversight. Smaller and rural practices would feel that pressure first. For them, assigning liability to the clinician who happens to click “accept” on an AI recommendation would turn device adoption into a malpractice risk calculation rather than a patient-care decision.
Open-ended models create a different regulatory burden
The FDA’s proposed framework treats risk according to the action a device performs. A tool that simply supplies information with no immediate action, such as a prediction of a future cardiovascular event, would generally sit on the lower-risk end of the spectrum. A system that makes autonomous decisions, by contrast, carries far greater clinical danger because errors can propagate without human interruption.
Provider groups largely accepted that starting point, but they want the agency to go further. The American Hospital Association argued that activity alone does not capture the true danger of a generative AI tool. A system that recommends emergency care can create harm in two directions at once: it may send a patient with a non-emergency condition into the emergency department, or fail to flag a patient who truly needs urgent attention. A simple classification based only on whether the tool is informational, action-oriented or supervised by a clinician misses the probability of error and the consequences of that error.
That matters because risk in healthcare is rarely binary. A recommendation engine can be technically “informational” and still nudge clinical behavior in ways that change triage, referral patterns or treatment timing. The physiological stakes are high when a device affects whether a patient reaches the right level of care promptly or waits too long. The FDA framework will need to account for those downstream effects if it is going to describe real-world danger accurately.
Human oversight is not a complete safety net
The American Medical Association challenged another common assumption: a human reviewer does not automatically make an AI tool safe. Clinicians may miss subtle model errors, especially when outputs are presented with confidence and speed. Repeated exposure can also create automation bias, where users come to trust a system’s suggestions more than their own judgment. In that setting, “human in the loop” becomes a weak safeguard unless the human reviewer is also evaluated as part of the overall risk profile.
This point cuts to the core of how generative AI enters medical workflows. If a clinician is expected to supervise output from a system they cannot fully interrogate, the supervision model depends on transparency, training and vigilance that may not be realistic in busy settings. Hospitals told the FDA that governance structures already exist in many health systems for AI-backed devices, but those structures do not solve the problem of opaque models. Without understanding how a system arrives at its output, it becomes difficult to identify faults, track drift or determine whether performance is slipping in a particular patient population.
Manufacturers should remain responsible for the safety and performance of the devices they sell.
Calls for reporting channels and patient-facing labels
The Workgroup for Electronic Data Interchange proposed a more structured oversight pathway. Its comments urged the FDA to create a formal process for reporting adverse events linked to generative AI devices and to develop “nutrition” labels that would explain what a tool does and what risks it carries. The concept echoes prior efforts to make AI systems more legible to clinicians and patients by standardizing disclosures about function, limitations and intended use.
That idea is not new in health technology policy. A similar card-style disclosure approach had been proposed under the Biden administration before being rescinded by the Trump administration last year. Even so, the logic behind the label remains clinically relevant. A clear device summary can help clinicians distinguish between a documentation aid, a triage assistant and a tool that touches diagnostic reasoning. Patients, too, may be better positioned to understand when an AI system is contributing to their care and when it is merely handling administrative tasks.
Trust in health AI depends on more than novelty. Physicians have repeatedly said they want tools that are validated by a trusted entity and monitored continuously after deployment. They also want a clear liability framework and a regulator willing to police safety claims before problems reach the bedside. Without that structure, adoption will remain cautious, especially in settings where clinicians already shoulder the fallout from technology failures they did not design.
FDA guidance may arrive within months
Rick Abramson, director of the FDA’s Digital Health Center of Excellence, said at a conference earlier this week that the agency plans to post draft guidance on its approach to regulating generative AI-enabled medical devices in the coming months. That timing gives provider groups a narrow window to shape the final contours of the framework before it hardens into policy.
The policy question is no longer whether generative AI will enter medical practice. It already has. The real question is whether the rules will assign responsibility to the people building these systems, or whether clinicians will be expected to absorb the operational and legal risk of tools they did not create and cannot fully inspect. The answer will determine not just who gets sued after a failure, but how quickly hospitals feel comfortable using these devices in everyday care.