Large Language Models Mostly Stayed Within Choosing Wisely Boundaries, But Two Scenarios Drove Most Overuse
Commercial large language models were generally cautious when confronted with scenarios built from Choosing Wisely recommendations, but they were not uniformly disciplined. In a benchmark of 225 clinical vignettes drawn from 15 recommendations, the models rarely withheld an appropriate action and only sometimes pushed clinicians toward testing or treatment that should have been deferred. The pattern was uneven rather than random. A small number of scenarios accounted for most of the inappropriate escalation.
The evaluation used six commercial models accessed through production application programming interfaces from OpenAI, Anthropic, and Google between February and March 2026. Investigators generated 4,050 responses across three runs per scenario, using prompts written from a clinician’s perspective. The study did not involve patients; it tested whether the systems could distinguish when restraint was appropriate from when an action should still be taken despite a recommendation that often cautions against overuse.
Overuse Was Uncommon Overall, But Not Uniformly Distributed
Across all responses, the rate of model-recommended overuse was 5.4%, with a 95% confidence interval from 0.6% to 12.2%. In plain clinical terms, the models usually resisted recommending care that Choosing Wisely would discourage. Yet that headline number concealed sharp variation by scenario. Eight of the 15 recommendations never triggered overuse at all, while two scenarios accounted for most of the inappropriate recommendations that did appear.
Those two scenarios were especially vulnerable to model drift toward intervention: thyroid ultrasound for an incidental subcentimeter nodule and routine electrocardiography in low-risk adults. In the thyroid ultrasound scenario, the models recommended overuse in 42.8% of responses. In the routine electrocardiography scenario, overuse appeared in 28.3% of responses. Both are familiar traps in everyday practice, where the instinct to “check one more thing” can override the low-yield nature of the test. Here, the models reproduced that tendency instead of consistently resisting it.
The rest of the recommendations behaved differently. For the remaining five recommendations in which overuse sometimes appeared, the rate ranged from 0.6% to 3.3% of responses. One sentence captures the split: the models were usually conservative, but they were not reliably selective when the clinical situation was one of the classic gray zones in overtesting.
False Restraint Was Rare, But Prostate Screening Stood Out
The converse error occurred far less often. False restraint, defined as withholding an appropriate action, was 0.6% overall, with a 95% confidence interval from 0.1% to 1.2%. In 11 of the 15 recommendations, false restraint did not occur at all. When it did emerge, the highest rate was seen with prostate-specific antigen screening, where false restraint reached 3.3%.
That result matters because it suggests the models were more comfortable avoiding unnecessary care than denying care that remained clinically justified. In a physician-facing workflow, that asymmetry could cut both ways. A cautious system may reduce noise and help curb low-value testing, yet even a low rate of false restraint can be consequential when the withheld action is one that should have been discussed or offered to the patient. Screening recommendations are especially sensitive to this balance because they often depend on age, risk, preferences, and the clinical context around shared decision-making.
The study’s co-primary outcomes—overuse and false restraint—frame an important tension for medical AI. A model that errs toward action may reinforce the very overuse Choosing Wisely was built to reduce. A model that errs toward restraint may block care that remains appropriate in the right patient. These benchmark results suggest the commercial systems tested were more likely to drift toward the first problem than the second, but the imbalance was driven by a limited set of scenarios rather than a broad failure across all recommendations.
Benchmark Performance Does Not Equal Real-World Clinical Reliability
The design was intentionally controlled, and that control is both a strength and a limitation. Single prompts were used, with no follow-up clarification or iterative dialogue. That means the models were not tested in the conversational setting where clinicians often refine uncertainty, add details, or challenge an initial answer. The evaluation was also a benchmark rather than a real-world deployment study, so it cannot show how these systems behave inside actual clinical workflows, during time pressure, or in response to incomplete histories.
Responses were classified by an LLM judge and then reviewed by physician adjudication. That approach allows large-scale assessment, but it also introduces dependence on the same class of technology being evaluated. The findings may also fail to extend to consumer-facing versions of these tools, where prompt quality, guardrails, and interface design differ from the production APIs used in the study. In other words, this was a test of a specific technical environment, not a universal verdict on all medical chatbot behavior.
Even with those boundaries, the signal is clinically useful. The models rarely withheld appropriate recommendations when action was needed, and their overuse was concentrated in two familiar high-risk areas: incidental thyroid imaging and routine electrocardiography in low-risk adults. Those are not obscure edge cases; they are common settings where low-value testing can spread quickly if a tool treats every abnormality as a reason to escalate. The results suggest that current commercial models can approximate restraint in many Choosing Wisely situations, but they still need tighter conditioning around scenarios where the default human impulse is to investigate, even when the evidence argues against it.
What Clinicians Should Take From the Pattern
For bedside use, the practical message is narrow. These systems cannot be assumed to reproduce Choosing Wisely logic consistently across all low-value care scenarios, and their performance is uneven enough to justify close oversight in any deployment meant to support clinical decisions. The study’s most important finding is not that the models failed everywhere; it is that their errors clustered where clinicians already struggle with overtesting.
The absence of external funding does not change the interpretation. It does, however, make the benchmark easier to read as an independent assessment rather than a vendor-driven demonstration. What remains unresolved is how these models behave when embedded in real clinical conversations, where they can be queried, corrected, and pressured toward nuance. Until that is tested directly, the safest reading is simple: current commercial LLMs are often cautious, sometimes overly so, and still vulnerable to overuse in a few highly recognizable scenarios.