Automated metrics tracked edits, not clinical judgment, in patient portal messaging
A staged rollout of AI-generated clinical messages exposed a fundamental weakness in common text-similarity scores: they can measure how closely a physician edits a draft, but not whether the finished message is as effective as one written from scratch. Thirty primary care physicians used GPT-generated draft responses for patient portal messages between September 2023 and August 2024, creating a natural comparison between two very different workflows. In one arm, AI drafts were shown to physicians, who could edit them before sending; in the other, drafts were generated silently and never displayed, leaving physicians to compose replies independently.
The study design captured two separate questions that are often blurred together in conversations about clinical generative AI. One asks whether a physician stays close to the machine-generated draft. The other asks whether the resulting message is actually good clinical communication. Those are not the same problem, and the data separated them cleanly.
Similarity scores rose when physicians edited AI drafts
Among messages in the intervention group, 2,264 of 21,073 drafts, or 11%, were incorporated into final patient-facing replies. Those edited messages showed strong semantic alignment, with BERTScore F1 reaching 0.93. In plain terms, when physicians used the draft as a starting point, the final text remained very close to the original machine-generated version. That is exactly the kind of workflow similarity metrics are built to detect.
The non-displayed group tells a different story. Physicians in that arm wrote independently, yet their final messages still showed moderate similarity to the unseen drafts, with BERTScore F1 of 0.83 across 45,127 messages. The score looks impressive at first glance, but the practical meaning shifts once the draft has never entered the physician’s field of view. Similarity in that setting does not prove that the AI draft shaped care; it only shows that human clinicians, writing about common portal issues, can arrive at text that overlaps with what a model would have produced.
Physicians preferred human-authored replies despite comparable semantic scores
A single number can hide the most consequential result in the study. Even when semantic scores looked favorable, physicians preferred human-authored messages, rating them 4.24 out of 5. That preference matters because patient portal communication is not a neutral text-generation task. It is a clinical act that must balance empathy, clarity, liability, and the limits of what can be safely implied in writing.
A portal reply often needs to do several things at once: answer the question, avoid overpromising, reflect the clinician’s judgment, and remain legible to a patient reading on a phone screen late at night. Automated overlap measures do not assess tone, reassurance, or the risk of subtle misunderstanding. A message can be linguistically close to a draft and still feel more trustworthy when a physician revises it in a way that captures nuance rather than raw phrasing.
Metric validity changed with the workflow context
The most clinically important finding was not the raw similarity values themselves, but where those values did and did not track physician quality judgments. BERTScore correlated with physician ratings in the intervention group, where AI drafts were visible and edited. In the non-displayed group, the same metric failed to correlate with physician quality ratings. The metric could assess editing fidelity in a workflow where the draft was part of the human writing process. It could not evaluate whether AI-generated content matched independent human performance when the draft never entered view.
That distinction has immediate consequences for how hospitals and researchers validate generative AI. A metric that looks strong in one deployment setting can become misleading in another. If an organization uses automated text scores to judge whether an AI assistant improves clinical communication, it may confuse resemblance with value. In patient messaging, resemblance is not the endpoint. Clarity, accuracy, and usefulness to the patient matter more than how closely a sentence mirrors a model’s suggestion.
BLEU, ROUGE, and reading ease were not enough to settle the question
The investigators evaluated messages with BLEU, ROUGE, BERTScore, and Flesch Reading Ease, then compared those automated scores with blinded physician ratings of effectiveness, misunderstanding potential, and harm. That range of tools reflects a common assumption in AI evaluation: if several metrics move together, they may collectively approximate quality. The study shows why that assumption can fail in clinical messaging.
Text overlap measures are strongest when the research question is narrow and mechanical. They perform much less reliably when the goal is to judge whether a communication artifact supports care. A short, efficient reply may score well on readability and similarity while still missing the patient’s concern or introducing ambiguity. A longer, more careful reply may score worse on overlap but better on clinical communication. The measured dimensions do not collapse neatly into one another.
Clinical evaluation needs context-specific validation
The practical lesson for health systems is straightforward: automated metrics should be validated against the actual use case before they are used as proxies for quality. Here, the metrics were useful for measuring how closely physicians followed an AI draft when the draft was visible. They were not sufficient for judging independent human performance when the draft remained hidden. In other words, the same score can mean two different things depending on whether the model is shaping the message or merely mirroring what a clinician would have written anyway.
This matters for any deployment of generative AI in patient communication, especially in high-volume portal environments where small changes in wording can alter how patients interpret reassurance, uncertainty, or urgency. A hospital cannot assume that a high semantic score proves safer communication. It only proves that text is semantically close. Clinical communication demands more than closeness.
The study’s design, centered on a staged rollout and blinded physician assessment, gives the findings unusual force. It shows that the evaluation framework itself has to be tested as carefully as the model. If the metric cannot distinguish edited AI-assisted writing from independently composed clinical prose, then the metric may be answering the wrong question altogether.