EU AI Act Framework Draws a Hard Line on Language Models in Nursing
The rapid move of large language models into health care creates a familiar problem with a new surface: tools that can write fluently may still fail when the clinical task demands accuracy, ethical restraint, and structured reasoning. A new framework published in JMIR Medical Informatics addresses that gap by testing 15 large language models and 2 small language models through an evaluation system aligned with Regulation (EU) 2024/1689. The work was designed for nursing and specialized health care management, where an error can cascade from documentation into triage, staffing, education, or decision support.
The framework was not built around casual benchmark chatter or single-answer accuracy. It was built around clinical utility, safety, and resilience under pressure.
Seven Domains, 32 Prompts, and a Delphi Reference Standard
Investigators used a 7-domain methodological structure and stress-tested feasibility with 32 multiparametric clinical prompts engineered to generate 27 evaluation items. Delphi expert responses served as the ground truth, giving the framework a clinically anchored reference rather than a generic language benchmark. A double-blind interdisciplinary panel rated model outputs on a 7-point Likert scale, and interrater reliability averaged Krippendorff α=.759 across models, which supports a fairly stable shared judgment across evaluators.
The design matters because nursing use cases rarely involve a single neatly packaged question. They involve overlapping priorities: patient safety, taxonomy-based language, documentation fidelity, ethical boundaries, and the ability to preserve meaning when the prompt becomes messy or adversarial. The study therefore evaluated evidence-based alignment, reference classification, automated consistency, and NANDA-I terminology, while also testing ethical resilience through progressive jailbreaking.
Safety Screening Removed Most Models Before They Reached the Clinic
The first major result arrived quickly. A stringent “Safety-Gatekeeper” domain classified 11 of 17 language models as unsuitable because of critical failures in evidence-based alignment or ethical resilience. Only GPT-o1, GPT-4o, Gemini 2.0 Pro Experimental, and three Anthropic models moved past the minimum threshold and into deeper evaluation.
A model that cannot withstand harmful prompting has no place at the bedside.
Among the models that cleared the gate, only Anthropic Sonnet variants achieved a uniform “recommended” categorization. Claude 3.7 Sonnet with extended thinking stood out with 75.9% accurate, focused references and high average scores in both clinical safety and data security, with mean scores of 6.73 (SD 0.23) and 6.83 (SD 0.41), respectively. Those scores are not cosmetic. In a health care workflow, safe information retrieval and secure handling of content are inseparable from trust, because a model that cites poorly or behaves inconsistently can destabilize clinical documentation and teaching at the same time.
Harmful Prompt Resistance Split the Field
The adversarial testing exposed a sharper divide than the ordinary benchmark measures. DeepSeek-R1, Perplexity Sonar, Mistral Large 2, and Qwen2.5-14B-Instruct failed to resist even explicit harmful prompts. Claude 3 Opus resisted both explicit harmful prompts and all jailbreak attempts while showing null sycophancy, an especially relevant property in clinical settings where agreeable but incorrect answers can be dangerous.
This is the practical fault line in health care AI: a fluent response is not the same thing as a safe response. Models that echo user intent too readily may amplify a bad assumption, endorse an unsafe plan, or smooth over uncertainty instead of challenging it. In nursing education and management, where queries often involve staffing, escalation, and protocol interpretation, that kind of behavior can distort downstream action.
Local Deployment Showed Real Value in Staffing Optimization
The locally deployed Qwen2.5-14B-Instruct model deserves attention for a different reason. Although it did not perform well in the harmful-prompt resistance tests, it outperformed 4 of the 15 LLMs in multistep problems tied to nurse staffing optimization. That result is important because locally deployed systems may offer an operational route for hospitals that need more control over data flow, infrastructure, and access, especially where cloud dependence creates procurement or privacy barriers.
Local deployment is not automatically safer, but it can be strategically useful when the task is bounded and the organization can supervise the model closely. Staffing optimization depends on stepwise logic, scenario balancing, and resource allocation, all of which can be supported by a model that is designed for internal use rather than broad open-ended conversation. The study’s finding suggests that local models may still have a role in operational nursing work even when they fall short on broader safety benchmarks.
Taxonomy-Based Context Improved NANDA-I Translation
Performance improved when the models were given taxonomy-embedded contexts for NANDA-I diagnostic translation. Gemini reached adequate performance in this setting, with an F1-score of 0.59 and a mean absolute priority distance of 4.0. The result points to a familiar problem in clinical AI: terminology matters, and context scaffolding can materially improve structured output.
Nursing language is not interchangeable with generic medical prose. Diagnostic labels, priorities, and standardized taxonomies carry operational weight, especially in documentation systems and care planning. A model that recognizes those structures more reliably may support education and workflow standardization, but only when its output is constrained by expert oversight and validated against clinical references.
What the Framework Means for Universities and Hospital Systems
The framework was positioned for pilot university hospitals and specialized care environments, where educators and clinical leaders need a repeatable way to separate useful tools from risky ones before deployment. Because the evaluation is designed to be dynamically adaptable, it can be updated as model behavior changes, regulatory standards mature, or new clinical use cases appear. That adaptability is important in a field where model versions can shift quickly and yesterday’s acceptable output can become today’s liability.
The study also extends beyond model ranking. It argues for a governance model in which interdisciplinary experts evaluate language models before they influence educational platforms, care algorithms, or management support systems. The emphasis on evidence-based nursing, nondiscrimination policies, operational resilience, and data protection places the framework squarely inside the real constraints of health care implementation rather than abstract AI enthusiasm.
Future work should prioritize multimodal capability, locally functioning models, and protections that address resource disparities. The study also ties its implementation logic to Sustainable Development Goal 10, connecting model access and infrastructure to broader equity concerns. Even so, the central message is narrower and more concrete: in nursing and health care, a language model must prove it can stay aligned with evidence, resist manipulation, and preserve clinical meaning before it earns a role in practice.