Large language models gave uneven answers on femoral head avascular necrosis questions
Artificial intelligence chatbots are already being used as a shortcut to health information, but this head-to-head comparison shows how uneven those answers can be when patients ask about avascular necrosis of the femoral head. Five widely used large language models—ChatGPT, Gemini, Copilot, Claude, and DeepSeek—were each asked the same 15 frequently asked questions, generating 75 total responses for review. Four orthopedic surgeons then judged those answers using the JAMA benchmark criteria, DISCERN, an adapted five-point clinical accuracy scale, and a binary safety score. The design is simple, yet the implications are immediate: patients may receive answers that sound confident while differing materially in transparency, quality, and readability.
Claude came out on top on the JAMA and DISCERN measures, while DeepSeek scored better than ChatGPT, Gemini, and Copilot on DISCERN. Clinical accuracy differed significantly across models, with statistical separation reaching p < 0.001. No significant difference emerged between Claude and DeepSeek, suggesting that both produced comparatively stronger patient-facing answers than the other three systems in this topic area. One sentence stands out because it cuts through the promise of conversational AI: the platforms did not perform uniformly on accuracy, even when asked exactly the same questions under the same review framework.
Accuracy, transparency, and clinical usefulness did not travel together
The JAMA benchmark and DISCERN instrument probe different dimensions of educational quality. JAMA-style scoring emphasizes transparency, authorship, attribution, and currency, while DISCERN weighs the reliability of information about treatment choices. In this study, the spread in scores shows that a model can appear polished while still falling short on the details that matter most to a patient weighing symptoms, diagnosis, and treatment options for femoral head osteonecrosis. The variation also matters because avascular necrosis is not a condition where vague language is harmless; missing key context about progression, imaging, joint preservation, or collapse can distort the patient’s next decision.
The orthopedists rated every response as safe across all four reviewers, which is reassuring but not the same as clinically complete. Safety scoring in this setting likely captures the absence of overtly dangerous advice, yet a model can remain “safe” while still omitting nuance, overgeneralizing, or failing to explain uncertainty. In bedside terms, a response that avoids immediate harm may still leave the patient without enough information to seek timely orthopedic evaluation. That gap is especially relevant in femoral head disease, where delay can turn a potentially joint-preserving window into a joint-replacement conversation.
Readability remained too demanding for patient education
The clearest weakness in the comparison was readability. Model performance differed significantly on both Flesch reading ease score and Flesch-Kincaid grade level, with p = 0.043 for FRES and p < 0.001 for FKGL. Across the systems, the average FKGL ranged from 10.85 to 13.93, which sits above the grade levels generally recommended for patient education materials. In plain clinical terms, the answers were often written at a level that would challenge many patients, especially those already coping with pain, limited mobility, or anxiety about the future of their hip.
Readability is not a cosmetic issue. If a patient cannot comfortably process the explanation, then even an accurate answer may fail at the point of care. For a disease such as femoral head avascular necrosis, where patients often need to understand risk factors, staging, prognosis, and possible intervention pathways, dense prose can become a barrier to action. The models may have produced fluent text, but fluency did not translate into accessibility.
Why osteonecrosis questions may expose a weakness in generative AI
Femoral head avascular necrosis is a condition in which blood supply failure leads to bone death and structural compromise of the hip. Because the disorder can progress from subtle pain to collapse of the femoral head, educational content has to balance simplicity with precision. An AI system that flattens those details risks missing the biological urgency embedded in the diagnosis. A patient asking about symptoms, imaging, or treatment is not just seeking a definition; the question often reflects a fear of losing joint function.
This is where large language models can fail in a clinically meaningful way. They are built to generate plausible language, not to verify every sentence against orthopedic nuance unless specifically constrained to do so. In a topic like osteonecrosis, the ideal answer should clarify uncertainty, separate conservative care from surgical options, and explain why timing matters in preserving the hip. The variation across models in this study suggests that some systems are better at approximating that structure than others, but none solved the readability problem.
What clinicians should take from the comparison
The study used a focused, repeatable design: 15 common patient questions, five models, one response per question from each model, and two structured quality instruments plus safety and readability measures. That makes the findings relevant for clinicians who are already encountering AI-generated text in patient portals, educational handouts, and informal online searches. The practical lesson is not that these systems should be ignored. It is that they should be treated as uneven drafting tools rather than reliable substitutes for clinically reviewed education.
Orthopedic teams and hospital educators will likely need to verify any AI-generated explanation before it reaches patients with hip pain or a diagnosis of osteonecrosis. Shorter sentences, lower grade-level writing, and explicit discussion of when to seek care could improve usefulness more than polished phrasing alone. A model that can answer without obvious danger still needs human editing to achieve the clarity expected in patient education.
Patient comprehension was not evaluated, which leaves an important gap. A response can score well on expert review and still fail the person who has to read it at home, under stress, and without a clinician beside them. The next step is not simply more AI output; it is testing whether patients actually understand and act on what these systems say.