Why Old Medical Device Rules Keep Missing AI Risk
Artificial intelligence in clinical care is moving faster than the regulatory frameworks built to govern it. A legal analysis in the Vanderbilt Journal of Entertainment and Technology Law argues that two familiar twentieth-century models have dominated policy thinking for too long: the idea that software should simply be regulated like a medical device, and the idea that safety can be preserved by placing humans around the machine as a kind of buffer. Both approaches, the authors contend, leave major gaps when AI and machine learning medical software is used to support trained health professionals at the bedside.
The core problem is not whether the software can be approved. It is whether the broader clinical system can remain safe once the software starts shaping diagnosis, triage, treatment, and workflow. In that setting, safety no longer belongs to one regulator alone. It depends on physicians, nurses, administrators, hospital leaders, and the bodies that govern medical practice itself. A narrow device-centered model cannot fully capture that reality.
Medical Devices Are Not the Whole Story
Treating AI-enabled software as if it were just another device puts the spotlight on premarket safety and technical performance. That model can work when the main question is whether a product does what it claims under controlled conditions. AI in clinical care raises a different question: what happens when the tool is embedded in a living system with variable staffing, unequal expertise, workflow pressure, and clinical judgment that may be nudged or eroded over time?
A model built around device safety can miss the human architecture around the tool. A recommendation engine may be technically sound and still produce harm if physicians defer too readily, if nurses are denied time to challenge a suggestion, or if administrators deploy the tool without sufficient safeguards. Clinical risk can emerge from the interaction between software and institution, not just from an error in code.
One sentence captures the concern bluntly: software safety alone does not make clinical safety.
Slack at the Human-AI Interface Was Meant to Help
The second regulatory model described in the article treats AI-enabled health care as a sociotechnical system. In that framework, the challenge is to create “slack” at the human-AI interface through buffers, redundancies, and checks that give people room to catch errors before they reach patients. The logic is straightforward: if an algorithm recommends the wrong test, treatment, or triage path, there should be enough human capacity left in the system to intercept the mistake.
That approach influenced the policy thinking behind the 21st Century Cures Act of 2016, which favored a regulatory posture that relied on human oversight and system design rather than a pure device-law model. Yet the article argues that this vision has not been implemented successfully. In practice, many health systems operate under exactly the conditions that erode slack: time pressure, staffing shortages, alert fatigue, and workflow designs that encourage passive acceptance of machine recommendations.
When the margin for intervention disappears, “human oversight” can become a legal fiction rather than a real safeguard.
Corporate Practice Doctrine Reframes the Safety Problem
The article then moves beyond the usual AI policy vocabulary and returns to an older doctrine from health law: the corporate practice of medicine. Through that lens, the central risk is not just defective software. It is the possibility that nonphysician corporate actors, including software developers and potentially the institutions deploying their products, could override or corrupt a physician’s independent medical judgment.
That framing changes the regulatory target. Instead of asking only whether the software is accurate, regulators would also need to ask who controls clinical decision-making, who can pressure clinicians to follow algorithmic outputs, and whether institutional incentives are distorting bedside judgment. A tool that merely recommends care is not necessarily benign if it becomes a vehicle for commercial control over how medicine is practiced.
The concern is especially sharp in systems where software is not neutral support but an operational force that shapes what is visible, what is recommended, and what is hard to do. Clinical autonomy can be narrowed without any formal change in licensure or scope of practice. The pressure arrives through interfaces, defaults, and workflow constraints.
AI as an Intelligent Agent, Not Just a Product
The article’s final conceptualization is the most expansive. It compares AI and machine learning tools to intelligent agents that are “colonizing” the health care system and threatening the humans who live and work within it. In that framing, the relevant harms are not limited to false positives, false negatives, or deployment defects. The harms may also include disruption of professional culture, erosion of trust, displacement of judgment, and changes in institutional values that accumulate over time.
This is not a call for anti-technology panic. It is a warning that medical AI may operate less like a static product and more like an active participant in care delivery. Once software starts adapting, influencing behavior, and embedding itself inside clinical routines, the old boundaries between tool, workflow, and authority become harder to defend.
A system can be technically efficient and still become clinically corrosive.
What Safer Oversight Would Have to Include
The article argues that protecting the safety, culture, and values of health care will require institutional and regulatory reforms that go beyond repurposing old frameworks from the last century. That means safety cannot be treated as a narrow question for device regulators alone. It also cannot rest on the assumption that humans will always have enough time, authority, and institutional support to correct machine errors.
A more durable approach would have to examine how clinical judgment is protected inside hospitals, how software is deployed by institutions, and how legal doctrines constrain nonphysician control over medical decision-making. It would also have to recognize that AI is not merely another software category. In the clinical setting, it can function as a decision-shaping force with broad effects on workflow, accountability, and bedside authority.
For regulators, the uncomfortable implication is that familiar tools may be too small for the problem. For clinicians, the practical implication is even sharper: independent judgment remains a safety mechanism, not a ceremonial ideal. If AI systems are allowed to crowd out that judgment, the risk reaches well beyond a single erroneous recommendation and into the structure of care itself.