Real-World ED Triage Exposes Wide Performance Gaps Across 16 Large Language Models
Emergency department triage is a high-stakes sorting problem. In practice, a nurse must translate a brief complaint, vital signs, and bedside judgment into a decision that shapes urgency, bed placement, and downstream resource use. This benchmarking study asked whether contemporary large language models can approximate that work when the input comes from a real-world triage environment rather than a curated test set.
The answer was uneven. Across 16,107 walk-in encounters from a tertiary academic emergency department in Germany, model performance varied sharply, with agreement ranging from poor to strong depending on configuration and evaluation set. A single model, GPT 5.1 with high reasoning effort, reached a quadratic-weighted Cohen’s κ of 0.5031 on the full cohort, a level that sits in the moderate range but still leaves substantial room for disagreement with nurse-assigned Emergency Severity Index levels. Sector assignment was even less convincing. Models rarely aligned well on whether a patient belonged in the emergency department or urgent care practice pathway.
A Benchmark Built From Routine Triage, Not a Curated Prompt Set
This was a retrospective cross-sectional study performed at a tertiary academic ED with an integrated central point of assessment. The dataset included all 16,107 walk-in encounters between October 18, 2023, and February 27, 2024. Inputs came from a structured clinical decision support system that captured presenting complaints, vital signs, and the triage decisions recorded by specialized nursing staff. That design matters because it reflects the information actually available at the front door of an emergency service, where ambiguity is common and decisions must be made quickly.
Sixteen configurations were compared on the first 2,000 encounters. GPT 5.1 with high reasoning effort was additionally tested on the full 16,107-encounter cohort. The primary endpoint was agreement between model-predicted and nurse-assigned Emergency Severity Index levels, measured with quadratic-weighted Cohen’s κ. Secondary measures included sectoral assignment, misclassification direction, confidence-versus-agreement patterns, and reproducibility.
Agreement Rose and Fell Sharply Depending on the Model Strategy
Performance spanned a wide band. On the shared 2,000-encounter subset, quadratic-weighted κ ranged from 0.180 to 0.860 across configurations. The strongest result came from the Stepwise approach, which combined structured model assessments with deterministic ESI mapping and achieved κ_qw = 0.860, with 82.70% exact agreement. Most other models clustered far lower, often in moderate or weaker agreement territory.
The difference between those results is not cosmetic. Triage accuracy determines whether a patient waits in a chair, moves to a monitored area, or receives immediate clinical attention. A model that drifts from the nurse’s assignment on a substantial fraction of patients would be a poor fit for direct operational use, especially in a system where misclassification can delay stabilization or consume scarce acute-care capacity. A separate challenge emerges from the fact that the models were not merely imprecise; they were also inconsistent with one another.
Sector Assignment Remained Weak Even When Triage Grading Improved
The sectoral task asked something simpler on the surface: emergency department or urgent care practice. Even there, agreement stayed uniformly low, with κ values at or below 0.304. In clinical terms, that suggests these systems struggled with a fundamental routing decision that depends on symptom severity, acuity, and local workflow rules.
Front-door medicine depends on more than language fluency. The model must interpret a symptom, infer risk, and map that risk onto an operational pathway. In an ED with an integrated central point of assessment, that routing judgment can affect throughput, staffing, and patient safety. Low sector agreement signals that the models may be especially fragile when asked to make decisions tied to institutional organization rather than pure text classification.
Non-Deterministic Output Undermined Reproducibility
Reproducibility testing added another concern. Substantial variability appeared in 23% of cases, indicating non-deterministic output behavior for clinically relevant decisions. In other words, the same clinical input did not always lead to the same recommendation.
That kind of instability is difficult to reconcile with emergency workflows. Triage systems need consistency because the same patient presentation should not be routed differently simply because the model was queried again. Variability also complicates validation: a model that performs acceptably on one run but changes its answer on the next is harder to audit, harder to regulate, and harder to trust when placed near bedside decisions.
Why the Stepwise Approach Performed Better
The Stepwise configuration likely benefited from an explicit bridge between model reasoning and deterministic ESI mapping. By separating assessment from final category assignment, the approach may have reduced the free-form drift that can arise when a model tries to infer too much from limited clinical context. In a triage environment, rules matter. Structured logic can preserve the link between the available data and the operational endpoint, especially when the model is handling complaints, vitals, and acuity clues that need to be weighed in a fixed order.
Still, even the best-performing method here should not be confused with deployable clinical automation. The authors set out to benchmark performance, not to validate autonomous triage. The gap between a high κ on retrospective data and safe use in a live emergency department is wide, particularly when reliability, calibration, and workflow integration remain unresolved.
What the Full-Cohort Result Means for Clinical Use
GPT 5.1 with high reasoning effort produced the strongest large-sample estimate reported in the abstract, but κ = 0.5031 does not support direct substitution for trained nursing triage. The result suggests some useful signal, yet not enough consistency for independent clinical action. In emergency care, moderate agreement may still translate into clinically meaningful discordance, especially around edge cases where acuity is not obvious at first glance.
The real-world character of this dataset makes the finding more relevant, not less. Curated benchmarks often smooth away the messy features of frontline care: incomplete histories, competing symptoms, and the need to triage before the full picture is known. A model that cannot keep pace with that environment faces a steep path to implementation, no matter how fluent its language output appears.
Regulatory Validation Will Need More Than Accuracy Alone
Before any clinical implementation, this work points to three requirements: confidence estimates, reliability, and workflow integration. Prospective validation in regulatory-compliant clinical settings will be essential. Accuracy alone will not answer whether a model can support triage without introducing instability or hidden error patterns.
The study was approved by the ethics committee of the University of Freiburg, and informed consent was waived because the data were anonymized at collection. One author disclosed a potential conflict of interest tied to the company that manufactured the decision support system used to record triage data; the remaining authors reported no conflicts. The manuscript was revised to correct a minor reporting error.
For now, the most defensible reading is straightforward. Current large language models can sometimes approximate emergency triage decisions, but their performance is uneven, their routing judgments are weak, and their output can shift from one run to the next. In a setting where seconds and seconds of hesitation both matter, that instability remains a central barrier.