ChatGPT Delivered the Most Reliable Public Answers on Active Charcot Neuro-Osteoarthropathy
Active Charcot neuro-osteoarthropathy is a race against time. When patient-facing information is incomplete or wrong, offloading can be delayed, clinical assessment can be postponed, and imaging may arrive too late to prevent avoidable collapse of the foot and ankle. This study examined whether public large language models could answer standardized questions on active CNO with enough accuracy, safety, and source transparency to support disease-related health literacy.
Investigators built 20 standardized questions spanning 10 clinical domains from six guideline or consensus sources and locked the question set before testing began. Four public-facing systems were evaluated in their recorded native configurations: ChatGPT, Claude, DeepSeek, and Doubao. One first-complete response was retained for each question, producing 80 responses in total. Two clinicians independently scored masked answers using DISCERN, EQIP, GQS, the JAMA benchmark, a question-specific guideline-concordance scale, and a prespecified safety-error taxonomy. Sources were manually checked, and both response length and six readability indices were measured.
The most direct signal came from the safety analysis.
ChatGPT had 0/20 responses with any safety-critical error. Claude showed such errors in 14/20 responses, DeepSeek in 16/20, and Doubao in all 20 responses. Moderate or major errors were absent in ChatGPT, but appeared in 10.0% of Claude responses, 45.0% of DeepSeek responses, and 60.0% of Doubao responses. Major errors occurred only with Doubao, affecting 3/20 responses. In a condition where missed offloading or delayed review can accelerate deformity, these differences are not cosmetic; they map directly onto bedside risk.
Guideline Concordance and Verifiable Sources Diverged Sharply
Across the quality scales, all informational outcomes differed between platforms, with global Friedman testing yielding P < 0.001. ChatGPT achieved the highest median scores on DISCERN, EQIP, GQS, JAMA, and the guideline-concordance scale. Median guideline-concordance scores were 5.0 for ChatGPT, 3.5 for Claude, 3.0 for DeepSeek, and 2.0 for Doubao.
The source-verifiability findings were especially stark. Authentic sources with auditable claim-source linkage appeared in every ChatGPT response and in none of the other platforms. Unsupported clinically meaningful numerical claims appeared in 0% of ChatGPT responses, compared with 60% of Claude responses, 75% of DeepSeek responses, and 65% of Doubao responses. For a patient asking whether weight-bearing should stop immediately, whether imaging is urgent, or how long offloading usually lasts, a model that supplies polished but unverifiable numbers can create false confidence in the wrong decision.
The pattern suggests a familiar failure mode in public-facing generative AI: fluent language is not the same as clinically grounded language. A response can read cleanly while still omitting the practical next step that matters most in active CNO, such as urgent evaluation, immobilization, or prompt specialist review. In a neuropathic foot, delay is not neutral. It can translate into more swelling, more bony destruction, and a harder reconstruction problem later.
Length Was Not a Substitute for Clinical Value
Response length varied substantially across platforms, and pooled analyses found inverse associations between longer responses and all five quality-related measures. Even so, within-platform and adjusted sensitivity analyses did not show a consistent negative relationship. The simplest reading is not that brevity is always better or that length automatically degrades quality. Rather, the study shows that verbosity alone does not rescue a response that is poorly grounded, incomplete, or unsafe.
That point matters for patient-facing care. A lengthy answer may feel reassuring, but if it dilutes the urgency of offloading or fails to name the need for assessment and imaging, it can mislead the user at the exact moment when speed matters. Conversely, a shorter answer can still be adequate if it is specific, guideline-concordant, and careful about uncertainty.
Readability was also assessed, but the core clinical story did not depend on a single readability score. The more consequential issue was whether the systems could anchor advice to verifiable guidance and avoid harmful omissions. In active CNO, the relevant question is not whether the prose sounds accessible; it is whether the patient leaves with the right actions and the right urgency.
What the Benchmark Does and Does Not Prove
Under the prespecified conditions of this benchmark, the tested ChatGPT public-facing configuration showed relative advantages in informational quality, guideline concordance, source verifiability, and safety. Even so, the authors framed the result correctly: this was a single-response, time-specific snapshot of recorded public-product configurations, not a stable ranking of the underlying models.
That distinction is important because public-facing model behavior changes over time, and commercial systems are frequently updated without warning. A future version may perform better, worse, or differently on the same benchmark. The study therefore speaks most strongly to governance: public medical AI should be evaluated in the exact configuration patients encounter, not only in abstract model comparisons.
The findings also support a narrower clinical conclusion. Well-governed public LLMs may have a role in disease-related health literacy and timely care-seeking if they can reliably preserve guideline-concordant actions, identify legitimate sources, and avoid unsupported numerical claims. For active Charcot neuro-osteoarthropathy, where the harm from delay is tangible and irreversible, that standard is not optional.
The remaining problem is implementation. Public systems that generate confident but unverified guidance can still obstruct care by softening urgency, omitting escalation, or obscuring the need for expert assessment. Until those risks are controlled prospectively, patient-facing AI in CNO should be treated as a tool to be audited, not trusted by default.