LLMs Matched Experts Best When the Evidence Was Settled
A multimodel comparison published in Arthroscopy, Sports Medicine, and Rehabilitation tested whether current large language models could approximate a human expert Delphi panel on posterolateral corner treatment statements. The design was straightforward but clinically pointed: four commercially available models — Gemini-2.5-Pro, GPT-4o, o4-mini, and o3 — were asked to rate consensus statements on a 5-point Likert scale and to estimate how much expert agreement each statement would generate. Their performance was then measured against the concurrently run human modified Delphi process.
The central pattern was not surprising to anyone who has watched these systems perform in medicine, but the details matter. LLM-generated ratings correlated only moderately with average expert ratings, and some models leaned toward overly positive scoring. In a field where nuance often separates standard practice from disputed technique, a consistently optimistic model can flatten the distinctions between well-supported care and weakly supported opinion. The gap widened when the models tried to infer consensus from their own internal self-consistency. Agreement among repeated runs did not reliably track actual expert consensus, which means repetition alone did not make the models more faithful to the panel.
Consensus Prediction Worked Better Than Self-Consistency
A separate task asked each model to estimate the level of expert agreement behind each statement. That part of the exercise proved more informative than simple rating replication. Some models aligned more closely with the true expert consensus when asked to forecast the degree of panel agreement, suggesting that these systems can sometimes recognize the difference between settled knowledge and contested territory even when their own Likert ratings blur the line.
In practical terms, this matters because modified Delphi work depends on more than a yes-or-no answer. It depends on judging where consensus is robust, where it is fragile, and where the literature remains fragmented enough to justify further discussion. For a musculoskeletal topic such as posterolateral corner treatment, where operative indications, technique selection, and interpretation of supporting studies can be debated, the ability to estimate consensus may be more useful than blindly generating a single clinical judgment.
The models were not interchangeable. Qualitative review showed distinct patterns in how they justified answers, how much certainty they expressed, and how they handled conflicting evidence. Some offered more assertive language than the underlying evidence could support, while others appeared more cautious. Those differences are not cosmetic. They affect whether a user interprets an answer as a carefully weighed synthesis or as an overconfident summary built on incomplete inference.
Web Search Improved Alignment With Recent Evidence
A more encouraging finding came from the models’ deep research or web-search-enabled mode. Access to current literature improved both rating accuracy and consensus prediction. That result fits the clinical logic of Delphi work, where the relevant evidence base can change quickly and older training data may leave a model stranded behind the current discussion. When a system can retrieve recent, relevant studies, it becomes less dependent on static memory and more able to reflect the literature actually driving expert opinion.
Still, the improvement was not absolute. The analysis also identified evidence of hallucination in model reasoning, a reminder that fluent prose can still rest on unstable inference. In a consensus environment, hallucination is not just a language problem. It can distort how certainty is assigned, how opposing studies are weighted, and how a controversial statement is ultimately classified. For clinicians and researchers using LLMs to support consensus generation, the issue is not whether the model can sound persuasive. It is whether it can remain tethered to the evidence when the evidence is incomplete, contradictory, or evolving.
Why Established Topics Travel Better Through AI
The study’s broader message is clinically intuitive. LLMs performed better on established topics than on emerging or debated ones because established topics offer clearer pattern matching, more stable phrasing in the literature, and less ambiguity in expert interpretation. Once a subject becomes contested, however, the model has to do something closer to scholarly synthesis: compare competing claims, infer the weight of evidence, and decide how much agreement is truly present. That is a harder task than reproducing familiar orthopedic doctrine.
This distinction has direct implications for specialty education and consensus building. LLMs may be most reliable as assistants for topics where the literature already supports a relatively stable expert position. They are less dependable when the field is still sorting out technique variations, indications, or outcome definitions. In those settings, a model that produces a polished answer too quickly may be less useful than one that explicitly signals uncertainty and can cite recent sources.
A single model’s confidence should not be mistaken for expert agreement.
What the Delphi Comparison Means for Orthopedic Practice
The investigators framed the work as an artificial intelligence-augmented Delphi study, and that framing is apt. Delphi methodology depends on iterative judgment, careful comparison of opinions, and deliberate handling of disagreement. LLMs can contribute to that process, but they do not reproduce it cleanly. Their strengths lie in rapid synthesis, broad exposure to published language, and the ability to estimate consensus when prompted to think beyond a single rating. Their weaknesses appear when the task requires calibrated uncertainty, disciplined handling of conflicting evidence, or restraint in the face of incomplete data.
For clinicians working in sports medicine and orthopedics, the takeaway is concrete. LLMs may help organize discussion, screen statements, or suggest where consensus is likely to hold. They should not be treated as stand-ins for domain experts, especially in unsettled areas such as posterolateral corner management. The models also appear sensitive to the way they are prompted and to whether they can search current literature, which makes deployment choices part of the clinical methodology rather than a technical afterthought.
The study was classified as Level V, expert opinion, so it should be read as a methodological comparison rather than a clinical outcomes trial. Even so, it raises a practical question for future consensus work: if AI is used to support Delphi processes, which model, which retrieval method, and which guardrails are needed to keep the output aligned with expert reasoning rather than merely fluent approximation? The answer will depend on more validation, especially on disputed clinical topics where confidence is easiest to display and hardest to justify.