Large Language Models Give Uneven Answers on Functional Hypothalamic Amenorrhea
A head-to-head comparison of four large language models found that functional hypothalamic amenorrhea questions did not reliably produce the same quality of answer across systems or across audiences. Patient-facing questions and clinician-facing questions triggered different strengths and weaknesses, with no model delivering consistently superior performance in every setting. The practical issue is simple: a model that performs well on one type of query can still produce incomplete or inaccurate guidance on another.
Investigators entered 11 patient-level and 15 clinician-level FHA questions into ChatGPT 3.5, ChatGPT 4.0, Gemini, and OpenEvidence. OpenEvidence was limited to clinician-level questions. Three expert reviewers, blinded to the model source, graded each response as accurate and complete, accurate but incomplete, or inaccurate, while a fourth reviewer resolved disagreements. For patient-facing content, readability was also measured with the Flesch Reading Ease Score and word count.
Patient Questions Favored ChatGPT 4.0, But Not by a Wide Margin
On patient-level questions, ChatGPT 4.0 produced the highest accuracy, with 9 of 11 answers rated accurate, or 82%. ChatGPT 3.5 and Gemini each reached 8 of 11, or 73%. Those differences did not reach statistical significance, which means the small numerical spread should not be mistaken for a decisive separation in performance. Even the better-performing model still missed nearly one in five patient questions.
Completeness tracked in a similar direction. ChatGPT 4.0 gave the most complete responses for patient questions, with 89% completeness, while the other models were less complete. For a condition like FHA, where patients often want to understand menstrual loss, exercise, nutrition, stress, fertility, and bone health in the same conversation, incomplete explanations can leave important gaps in care. A model may sound confident and still omit the next clinical step a patient actually needs.
The readability findings were more striking than the accuracy differences. Gemini generated the easiest-to-read patient content, with a median Flesch Reading Ease Score of 43.5, compared with 30.6 for ChatGPT 3.5 and 28.8 for ChatGPT 4.0. Lower scores indicate more difficult reading, so both ChatGPT versions produced text that was harder for lay users to process. Word count did not differ significantly across the models, suggesting that the issue was not brevity versus length, but sentence structure and linguistic complexity.
A patient can receive a medically correct answer and still struggle to use it.
Clinician-Facing Answers Were Strongest With OpenEvidence
The clinician-level results shifted the ranking. OpenEvidence delivered perfect accuracy, with 15 of 15 answers rated correct. ChatGPT 4.0 followed at 93%, while Gemini reached 80%. On completeness, OpenEvidence again led the field with 93%, and the other systems lagged behind. For clinicians asking about diagnostic criteria, evaluation strategy, or management priorities, that gap matters because partial answers can distort decision-making in a condition that often overlaps with eating disorders, overexercise, weight loss, psychological stress, and hypothalamic suppression.
The performance pattern suggests that model design and training emphasis matter. A tool optimized for clinician use may be better at presenting evidence-based content in a compact and operational form, while a general-purpose system may be more variable when asked for medical nuance. In a disorder like FHA, a sound response needs more than a definition. It has to connect reproductive endocrinology, nutritional status, energy availability, and long-term consequences such as impaired bone health.
Even in the clinician arm, no model served as a flawless substitute for judgment. The difference between 100% and 93% accuracy may look small on paper, but in a narrow clinical domain it can represent a missed caution, an incomplete differential diagnosis, or an omitted follow-up step.
Why Functional Hypothalamic Amenorrhea Is a Difficult Test Case
FHA is not a single-problem disorder. It sits at the intersection of endocrinology, nutrition, exercise physiology, and mental health, which makes it a demanding topic for any language model. The condition can present with absent or irregular menses, low energy availability, stress-related hypothalamic suppression, and downstream reproductive and skeletal effects. Good counseling must be tailored, because patients may need different advice depending on whether the main driver is inadequate intake, excessive exercise, psychological stress, or a combination of all three.
That complexity helps explain why generic medical language models can stumble. They may generate plausible text without reliably prioritizing the most clinically urgent information. Inpatient-facing settings, that can translate into explanations that are too technical. In clinician-facing settings, it can translate into answers that sound polished but fail to distinguish among related endocrine causes of amenorrhea or fail to stress the need for individualized workup.
The study also makes clear that readability and clinical accuracy are not the same problem. A response can be easy to read and still be wrong. A response can be accurate and still be too dense for a patient trying to understand whether she needs evaluation for an eating disorder, changes in training load, or assessment of bone risk.
Human Oversight Still Sits at the Center of Safe Use
The findings point toward a practical path rather than a wholesale rejection of language models. Fine-tuning on specialty-specific data, prompt engineering, and human-in-the-loop review could improve performance. Those steps are especially important in conditions where the right answer depends on context, patient goals, and the tradeoff between reassurance and escalation.
For clinicians, the main takeaway is not that generative tools are unusable. It is that they need supervision when they are asked to explain reproductive endocrine disorders, especially to patients who may already be vulnerable, anxious, or receiving fragmented advice from multiple sources. In this study, the best systems still produced incorrect or incomplete information often enough to justify caution.
For patients, the study argues for a narrower message. A chatbot may help with basic orientation, but it should not be treated as a final authority on menstrual suppression, fertility concerns, or the health effects of chronic low energy availability. Careful review by a clinician remains essential when symptoms suggest functional hypothalamic amenorrhea, particularly because the condition can be missed if the conversation stops at a general explanation.
The central limitation is built into the exercise itself: scripted questions are not the same as real-world counseling, and real-world patients rarely ask one question at a time. The next step is to determine whether specialty-tuned systems can keep accuracy high while making patient counseling simpler, safer, and more usable in actual clinical encounters.