Five large language models held up well on infectious disease exam items, but not equally across every topic
Five contemporary large language models performed at a high level on infectious disease coursework, yet the pattern of errors was uneven and clinically revealing. Investigators compared ChatGPT-4.1, ChatGPT-4o-mini, Claude, Copilot, and Gemini against 227 exam items drawn from an infectious diseases course in a Doctor of Pharmacy program. Overall accuracy ranged from 89.4% to 91.2%, a narrow band that suggests strong baseline competence across the group. ChatGPT-4o-mini posted the top score, while Claude and Copilot landed at the bottom of the range.
The result is encouraging for educators who are already seeing these systems in study workflows. It is also a reminder that broad language fluency is not the same as dependable clinical reasoning. Infectious diseases education demands more than memorized facts: antimicrobial selection, pharmacologic nuance, organism recognition, and the ability to interpret images all pull on different cognitive machinery. The models handled some of those demands well and others much less consistently.
Pharmacology and therapeutics were strongest; chemistry and microbiology exposed the weak spots
Performance varied sharply by subject area. Accuracy was highest for pharmacology questions at 93.6%, followed closely by therapeutics at 93.0% and PK/PD at 91.1%. Microbiology fell to 87.5%, and medicinal chemistry came in lowest at 84%. Statistical testing showed medicinal chemistry and microbiology performed significantly worse than therapeutics, with p values below 0.05.
The pattern makes biological and educational sense. Pharmacology and therapeutics often reward structured reasoning, pattern recognition, and standard treatment logic, all of which large language models handle relatively well. Medicinal chemistry asks for more granular structural relationships, and microbiology requires detailed organism-specific distinctions that may be harder to infer from language alone. In infectious diseases practice, those gaps matter because the line between an effective regimen and an ill-suited one often depends on mechanisms rather than surface-level familiarity.
One sentence captures the core tension: high average accuracy can mask clinically meaningful misses.
Higher-order reasoning was harder than recall
Questions classified as Evaluate/Create performed worse than questions in the Remember/Understand category, with accuracy of 85.6% versus 92.0%, and the difference was statistically significant. This gap matters because lower-order recall is only a small slice of real clinical decision-making. Infectious diseases clinicians routinely weigh competing data, balance toxicity against benefit, and adapt recommendations to local resistance patterns, comorbidities, and drug interactions. A system that answers memorization items well may still struggle when the task shifts toward judgment.
In practical terms, the performance drop at the Evaluate/Create level points to limits in synthesis. Many medical students can recite first-line treatment choices; fewer can justify exceptions, compare regimens, or anticipate downstream complications. Large language models are trained on vast text corpora and can reproduce plausible medical language quickly, but plausibility is not the same as disciplined inference. The more a question demands prioritization, tradeoffs, and nonstandard reasoning, the more room there is for drift.
Image-based questions produced the sharpest decline
The most pronounced weakness appeared when questions included a picture. Accuracy fell to 69% for image-based items, compared with 91% for questions without a picture. That drop is clinically important because infectious diseases care often relies on visual information: microscopic morphology, radiographic patterns, dermatologic lesions, and other image-driven clues can alter diagnosis and management within minutes.
The image penalty also signals a broader limitation in multimodal clinical education. Text-only competence does not guarantee reliable interpretation of visual evidence, and visual evidence is often the hinge point in real cases. A model that can discuss antifungal therapy fluently may still misread the clue that should have led to the diagnosis in the first place. In a classroom, that may translate into a wrong answer. At the bedside, it can distort the differential diagnosis.
Moderate agreement shows the models often converged, but not perfectly
Agreement across model responses was moderate, with Cohen’s kappa values ranging from 0.497 to 0.613. Moderate agreement suggests these systems frequently reached similar conclusions, yet not with enough consistency to treat them as interchangeable. For educators and trainees, the finding argues against blind reliance on one model’s response, especially when the question draws on chemistry detail, image interpretation, or higher-order reasoning.
A moderate kappa also hints at a larger operational issue: consensus among models does not automatically confer correctness. Multiple systems can be wrong in similar ways when they draw from overlapping training data or propagate the same pattern-based assumptions. In medical education, that creates a specific risk. A learner may encounter concordant answers from several tools and mistakenly treat agreement as validation, even when the shared answer is incomplete or incorrect.
What this means for infectious diseases teaching and clinical training
The findings support a measured role for large language models in infectious diseases education rather than unrestricted use. These tools appear capable of assisting with review, self-testing, and rapid reinforcement of core content, especially in pharmacology and therapeutics. They are less dependable when the curriculum shifts toward medicinal chemistry, microbiology, image interpretation, or questions requiring deeper reasoning.
Careful integration is the operative word. Faculty oversight, answer verification, and explicit instruction about model limitations will be essential if these systems are incorporated into training. Without guardrails, a model’s polished delivery may obscure uncertainty, and in medicine polished delivery is not a substitute for correctness. The study did not provide evidence for bedside patient outcomes, and it did not test the models in live clinical decision support. What it does show is narrower and more immediate: even when performance is high overall, the failures cluster in exactly the areas where infectious diseases clinicians need precision most.
No funding was received for the project, and the authors reported no competing interests. The study’s value lies in its practical realism: it tested models against actual course material rather than abstract prompts, making the weaknesses easier to translate into educational policy and faculty guidance.