Large language models are moving into care faster than the rules can follow
Large language models are entering clinics, administrative workflows, and patient-facing digital tools with unusual speed, but the regulatory frame around them has not kept pace. A BMJ digital health & AI analysis argues that some LLM applications can influence medical decisions or clinical workflows without cleanly fitting the definitions that trigger formal oversight under either the Medical Device Regulation or the AI Act. The problem is not theoretical. When a model is built for general use, but later repurposed, misused, or allowed to drift into medical decision support, the legal status can become uncertain even as the patient safety stakes rise.
The article focuses on a narrow but important gap: LLM systems that may be used in healthcare yet do not carry a medical intended purpose from the manufacturer. Under that condition, they may fall outside the MDR’s device framework. At the same time, they may also avoid the AI Act’s meaningful high-risk obligations if they are not classified as high-risk systems. The result is a category of tools that can affect care without matching the regulatory expectations normally attached to software that touches diagnosis, treatment, triage, or other clinical functions.
Why the intended purpose of the manufacturer matters so much
Regulation in this area turns heavily on intended purpose. If a digital tool is designed and marketed for a medical function, the law can treat it as a medical device and attach corresponding obligations. LLMs complicate that structure because the same model may be used in very different ways depending on prompt design, workflow integration, and local deployment decisions. A general-purpose system may answer patient questions, summarize charts, draft letters, or support clinicians in ways that look medical in practice even when the manufacturer never framed the product that way.
That gap creates a difficult safety paradox. A system can be clinically influential without being formally recognized as medical software. In practical terms, this means the safeguards that usually follow medical device classification may not attach, even though users may rely on the output as if it were trustworthy clinical guidance. In a high-pressure care environment, a plausible but wrong response from an LLM can travel quickly from screen to bedside, especially when staff are seeking speed, automation, or help with documentation.
The manuscript treats that gap as a patient safety and liability issue, not only a legal classification problem. Once a tool is embedded in healthcare practice, errors are no longer abstract. They can shape triage, delay escalation, distort documentation, or influence how clinicians interpret complex information. LLMs are particularly sensitive because their value comes from flexibility, but that same flexibility makes boundaries hard to police.
The grey zone is defined by repurposing, malfunction, and inadvertent misuse
The analysis identifies a set of ambiguous applications that sit in a regulatory grey zone. These are not necessarily products sold as medical devices, yet they may be repurposed in medical settings, malfunction in clinically meaningful ways, or be used in ways the manufacturer never anticipated. In each case, the core concern is the same: a tool with real-world clinical impact may bypass the strongest layers of oversight because its formal classification never caught up with its actual use.
This is a familiar problem in software governance, but generative AI intensifies it. Traditional software tends to behave in relatively bounded ways. LLMs generate language adaptively, which makes their outputs more context-dependent and less predictable. A system that appears helpful in one exchange can fail in another, especially when prompts are ambiguous, the clinical context is complex, or users over-trust the output. The regulatory question therefore cannot stop at whether the product was sold as a medical device. It has to account for how the tool is being used and what kinds of harms it can plausibly create.
Four recommendations aimed at prevention rather than reclassification
The manuscript does not argue that every LLM should be reclassified as a medical device. Instead, it proposes four response areas centered on validation, mitigation, compliance, and alignment. The emphasis is preventive. Rather than forcing every application into the same legal bucket, the authors call for targeted safeguards that reduce risk in the settings where these tools are most likely to cause harm.
Validation speaks to whether an LLM behaves acceptably in the environment where it is actually used. A model that performs adequately in broad language tasks may still fail when the task is clinical summarization, advice generation, or workflow support. Mitigation refers to practical steps that reduce the chance of unsafe use, including limits on deployment context and controls on how outputs are presented or relied upon. Compliance addresses the need to fit healthcare use into existing legal and organizational requirements even when a product is not formally classed as a device. Alignment concerns the broader need to make the model’s design, deployment, and use patterns consistent with patient safety goals.
One sentence captures the central tension. Flexibility is the selling point, and flexibility is the hazard.
Patient safety now depends on how clinics govern use, not only on how companies label products
The article’s regulatory logic has a direct bedside consequence. If healthcare systems assume that a general-purpose LLM is outside oversight because it lacks a medical label, they may underestimate risk. If they assume that all responsibility sits with the manufacturer, they may miss the practical reality that local deployment choices can convert a general system into a clinically consequential one.
That makes governance a shared obligation. Hospitals, clinicians, developers, and regulators each influence whether a model is used safely. In a clinical setting, the most consequential failures may arise not from dramatic breakdowns but from ordinary misuse: overconfident reliance, poor prompt discipline, inadequate review of generated text, or using a system for a purpose beyond its tested scope. Those are the kinds of failures that can slip through when regulation waits too long for a formal label to catch up with the use case.
The paper leaves a clear message for healthcare organizations adopting LLMs. They cannot rely solely on whether a tool is officially designated as a medical device. They need to ask what the system can do, how it is being used, who is supervising it, and what safeguards exist when it makes a wrong or misleading output. The unresolved issue is not whether generative AI will enter healthcare. It already has. The remaining challenge is whether oversight can keep pace with tools whose clinical influence may be real even when their legal category is not.