Large Language Models Misfire on Choosing Wisely Scenarios in Both Directions
Large language models are increasingly being asked to serve as clinical helpers, but their behavior in Choosing Wisely scenarios raises a familiar problem in a new form: they can push care too far, or hold it back too aggressively. The framing alone is important. Choosing Wisely recommendations are built to reduce low-value testing and treatment, yet an algorithmic system that is too eager to act can drive overuse, while a system that is too conservative can create false restraint and leave patients without appropriate evaluation or therapy.
That tension sits at the center of this ACP journal review. The title signals a dual failure mode rather than a single one, and that distinction matters at the bedside. A model that recommends unnecessary care in one scenario and withholds needed care in another cannot be treated as a neutral decision aid. It is making judgment calls with consequences in both directions.
Why Choosing Wisely Scenarios Are a Difficult Test for AI
Choosing Wisely recommendations are not simple yes-or-no prompts. They often require an understanding of pretest probability, clinical context, symptom severity, and the difference between low-value and necessary care. A large language model can sound confident while still missing those distinctions. In practice, that can mean recommending an imaging study, laboratory panel, or intervention when evidence-based restraint would be more appropriate, or urging restraint when the patient’s presentation justifies further workup.
That is a hard task for human clinicians already. The model is being asked to reason across probability, nuance, and risk tolerance without direct access to the patient, the physical exam, or the clinical setting in which the question is being asked. A system trained on broad text patterns may reproduce the language of prudence without reliably applying it. It may also mirror the language of action without recognizing when action is unnecessary.
The phrase false restraint is especially useful here because it captures more than simple omission. A tool that overcorrects toward doing less can appear safety-minded while still creating clinical harm through delay, under-testing, or failure to escalate care. In low-signal scenarios, restraint may be appropriate. In uncertain or high-risk presentations, restraint can become a mistake.
Overuse and Underuse Are Two Sides of the Same Error
The title of the review places overuse and false restraint side by side, and that is not just a stylistic choice. It reflects a deeper problem in how large language models may handle medical recommendations. A system tuned toward avoiding unnecessary care may become reluctant to recommend services that are actually indicated. A system that leans toward responsiveness may do the opposite, generating excessive testing or intervention with little clinical justification.
That duality is clinically uncomfortable because it undermines the usual assumption that more AI support automatically means better decision support. In real practice, good recommendations are not defined by how often they say yes or no. They are defined by whether the recommendation matches the patient’s situation. A model that is inconsistent across Choosing Wisely scenarios cannot be trusted simply because it is fluent, polished, or fast.
One sentence captures the bedside concern plainly: fluent recommendations are not the same as correct ones.
What This Means for Clinicians Using AI Tools
For clinicians, the takeaway is not to abandon large language models but to treat them as unverified assistants rather than authorities. The more a tool is used to support decisions around testing and treatment, the more important it becomes to check whether it can distinguish low-value care from appropriate care in context-rich situations. Choosing Wisely use cases are particularly revealing because they sit at the intersection of safety, resource stewardship, and diagnostic uncertainty.
A model that can identify broad stewardship principles may still fail when the question becomes specific. It may recommend avoiding a test that would be appropriate in a patient with red flags, or it may suggest a workup where observation would be better. Either error can distort care, but the mechanism differs. One inflates utilization. The other suppresses needed action.
This matters because clinical decision support is often judged on convenience before it is judged on accuracy. A tool that is easy to query and quick to answer can seem useful even when it is unreliable in exactly the cases where precision matters most. In Choosing Wisely scenarios, precision is the entire point.
The Clinical Burden of Confident Error
Large language models have a particular liability in medicine: they can sound decisive even when the underlying recommendation is unstable. In a Choosing Wisely framework, that confidence may encourage overuse if the model gives permission for more testing than needed, or false restraint if it discourages action where escalation is warranted. Either way, the recommendation may be presented with the same calm tone.
That makes clinical oversight nonnegotiable. Human review remains necessary because the costs of error are asymmetric and patient-specific. An unnecessary test may lead to cascades, incidental findings, anxiety, and downstream procedures. An overly restrained recommendation may delay diagnosis, reduce access to beneficial care, or reinforce premature closure. The same interface can produce both harms.
The most practical implication is blunt: AI output should be treated as a draft, not a directive.
What the Title Suggests About the Next Phase of Evaluation
The focus on both overuse and false restraint suggests that future evaluation of large language models in medicine cannot rely on a single accuracy metric. A model should not be judged only on whether it agrees with a reference answer. It also needs to be assessed for the direction of its mistakes, especially in scenarios where the difference between action and restraint has real clinical consequences.
Choosing Wisely scenarios are a useful proving ground because they expose whether a model can distinguish between interventions that are unnecessary and interventions that are justified. If a system repeatedly errs toward one side, that bias becomes a safety issue. If it fluctuates across cases, that inconsistency becomes a reliability issue. Both matter in outpatient care, inpatient consults, and triage settings where diagnostic and treatment decisions are made quickly.
For now, the title of this ACP review captures the central warning with unusual clarity. Large language models can fail by recommending too much care or too little of it, and both patterns threaten thoughtful practice. The open question is not whether these systems can produce recommendations. It is whether they can produce recommendations that are context-sensitive enough to avoid replacing one kind of low-value care with another.