Microsoft’s DxO beat generalist physicians on diagnosis, but only inside a tightly controlled benchmark
Microsoft researchers have put a large language model-based diagnostic system through a test designed to mimic how physicians actually reason through a case, and the result was stark. The system, called Diagnostic Orchestrator or DxO, reached the correct diagnosis in 83.2% of New England Journal of Medicine clinicopathological conference cases and 80.3% of routine emergency department admissions. Generalist physicians performed far lower on the same material, even when they had access to search and reference tools: 36.1% on the clinicopathological cases and 46.8% on the emergency department cases.
The gap was not just numerical. DxO also operated differently at the bedside logic level, asking far more questions before ordering its first test and showing a more deliberate sequence of history-taking, hypothesis refinement and test selection. Physicians asked 5.6 questions on average before the first test. DxO asked 25.5.
A stepwise design makes the comparison more clinically meaningful
Earlier AI diagnostic experiments often gave a model the full case in one prompt, then asked for an answer. That approach can inflate apparent performance because real patients do not arrive as complete vignettes with every clue preloaded. In this study, both the AI system and the human physicians had to work through each case in stages. Each presentation began with a short clinical summary, then the diagnostician had to choose what to ask next, what to order, and when enough evidence had accumulated to commit to a diagnosis.
A separate AI gatekeeper controlled access to case details and released information only when requested. Every test carried an estimated cost based on CPT pricing from a hospital chargemaster. In practical terms, this means the benchmark tried to measure not only final answer accuracy but also the path taken to get there. A diagnosis reached after indiscriminate testing is not the same as one reached through focused reasoning.
The study used 304 NEJM clinicopathological conference cases and 325 emergency department–to-admission encounters from the MIMIC-IV-ED database. DxO ran on GPT-5 in the main analysis and was framed as a panel of physicians with different roles who challenged one another’s assumptions, tracked a differential diagnosis and weighed test cost before ordering more data.
The model’s errors looked different from human errors
When DxO was wrong, the miss usually stayed close to the correct answer. Its errors tended to involve a related diagnosis at the wrong level of specificity. Human errors were more often farther from the target diagnosis. That pattern matters clinically because a near miss can still keep the right disease family in view, while a distant miss can send the workup down an entirely different and potentially harmful path.
This is the kind of difference that can affect everything from downstream testing to whether a patient is admitted, observed or discharged. A model that stays within the right diagnostic neighborhood may be easier to correct than a clinician who has anchored on the wrong syndrome entirely. The study suggests that large language models can keep more of the relevant differential in play for longer, which may help them avoid premature closure.
The researchers attributed part of the advantage to the breadth of medical knowledge encoded in large language models and to consistent application of that knowledge under pressure. Fatigue, time constraints and cognitive bias can all affect human decision-making. DxO has none of those constraints, at least not in the form faced by clinicians in daily practice.
Cost fell when the system was budgeted, but accuracy also moved
Without budget limits, DxO’s average testing costs were higher than those of physicians. On the clinicopathological cases, the AI averaged $5,431 per case versus $4,725 for physicians. On the emergency department cases, the figures were $4,979 for DxO and $3,400 for physicians.
When budget constraints were imposed, the system became cheaper and remained ahead of the physician average. On the clinicopathological cases, DxO reached 80.9% accuracy at $2,241 per case and 79.6% at $1,303 per case. On the emergency department cases, budget-limited versions scored 73.2% at $2,634 and 69.5% at $2,037. The tradeoff is familiar to anyone who manages diagnostic uncertainty in real practice: more testing can buy confidence, but it also brings cost, delay and potential patient burden.
In this benchmark, the model’s ability to regulate its own investigation suggests a path toward diagnostic systems that do not simply generate answers, but also manage resource use with some discipline. Whether that translates into fewer unnecessary scans, fewer redundant labs or faster definitive workups in routine care remains unresolved.
Why the benchmark still falls short of real clinical practice
The study’s own limitations are substantial. The comparison was against individual generalist physicians, not specialists or multidisciplinary teams. The authors said they would expect the performance gap to narrow when a case is matched to the right expert. The NEJM cases also come with a teaching value that makes them unusually rich and often already include specialists’ conclusions.
The benchmark was text-only. It did not include imaging, waveforms or pathology slides, and it excluded the noise of ordinary clinical life, including patients who forget, omit or misunderstand details. Every case had a pathological diagnosis, so healthy or benign visits were left out entirely. Cost estimates relied on standardized U.S. pricing and did not capture invasiveness, discomfort or time to result.
Just as important, the study examined only the diagnostic phase. It did not assess treatment choice, safety, follow-through or patient outcomes. A system can appear strong at naming a disease and still be untested at deciding what to do next, when to escalate care, or how to avoid harm from an overly aggressive workup.
Microsoft’s disclosure and the next question for clinical deployment
All authors were Microsoft employees and held equity in the company, which sells AI and health care services. Eight authors were named inventors on a Microsoft patent application tied to the evaluation framework. The study had no outside funding.
The immediate clinical question is not whether DxO can win a benchmark. It is how a tool like this would behave in a live workflow where a clinician, a patient and a set of messy real-world constraints all shape the final decision. The authors pointed toward collaborative workflows in which clinicians and AI systems work together, and they also see possible use in medical education and continuing professional development.
For now, the study makes one point very clearly: under controlled, text-based conditions, a stepwise AI diagnostic system can outperform generalist physicians on accuracy and cost-aware reasoning. Whether that advantage survives contact with imaging, bedside uncertainty, specialist input and treatment decisions is still unanswered.