Chatbot answers on pulmonary aspergillosis varied sharply across platforms
Large language model chatbots are already being used as a first stop for medical questions, but pulmonary aspergillosis is a poor test case for casual summarization. The condition demands nuance: diagnostic reasoning changes with host immune status, imaging patterns, microbiology, and antifungal selection, and those moving parts are exactly where a fluent but shallow chatbot can become risky. In this comparative cross-sectional study, four widely used platforms were challenged with 31 search terms drawn from five years of worldwide Google Trends data, then judged on information quality, transparency, and readability. The resulting picture was blunt: the systems did not perform evenly, and none produced text that was simple enough for a sixth-grade reading threshold.
On 6 August 2026, investigators submitted the same query set to Microsoft Copilot, ChatGPT (GPT-5.5), Perplexity, and Gemini 3.6 Flash. Those prompts generated 124 first-response texts, creating a matched comparison across platforms rather than a loose collection of unrelated answers. Quality and transparency were assessed with JAMA, GQS, EQIP, and DISCERN, while readability was measured using SMOG, Coleman-Liau, FKGL, GFI, FRES, and ARI. The design matters because it captures the default response a patient, caregiver, or trainee would actually see on first contact, before any refinement through follow-up prompting or clinician correction.
One result stood out immediately. None of the 124 default responses met the prespecified sixth-grade threshold on any of the six readability measures.
That finding has direct bedside implications. In a disease where delays can be fatal and where the terms themselves are often unfamiliar even to medically literate readers, information that is too dense can function like misinformation by omission. If a patient cannot decode what the chatbot says about aspergillosis, the answer may still look polished while remaining unusable for real-world decision-making. Readability was not a cosmetic issue here; it was a core patient-safety variable.
Friedman testing, used to account for matched queries, showed significant platform differences across all four quality and transparency metrics. Median DISCERN scores were 43 for Copilot and Perplexity, 36 for ChatGPT, and 33 for Gemini. Copilot and Perplexity also achieved higher scores on several other measures of written-information quality and transparency, suggesting that these two systems handled the underlying medical material with somewhat greater structure and disclosure than the others. Even so, their relative advantage did not translate into plain-language output, and the entire cohort of chatbot responses remained textually complex.
The imbalance between apparent fluency and real communicative utility is clinically important. Aspergillosis is not a condition where vague reassurance or generic antifungal language is enough. Good medical writing in this setting must separate colonization from invasive disease, describe when imaging raises suspicion, and avoid turning uncertainty into false certainty. A chatbot that omits those distinctions can sound confident while quietly flattening the clinical pathway from symptom to diagnosis to treatment.
The study’s scoring framework reflects that concern. JAMA criteria capture basic authorship and attribution signals, GQS gauges overall quality, EQIP assesses information quality, and DISCERN focuses on treatment-related information. When platforms diverge across all four domains, the variation is not merely stylistic. It implies that some systems are better at organizing medically relevant content, signaling limits, and retaining enough structure to support further professional review. Even the best-performing platforms, however, remained far from acceptable as stand-alone patient education tools for a complex pulmonary fungal disease.
The design also carried a second message: default behavior matters. Chatbots are often judged on what they can produce under idealized prompting, yet real users usually ask a single question and accept the first answer they receive. In that setting, response quality and transparency become part of the diagnostic environment itself. For pulmonology, infectious diseases, and general medicine alike, the risk is not only incorrect content; it is content that is technically elaborate, superficially credible, and still too hard for ordinary readers to use safely.
No specific funding was received, and ethical review was not required because the study analyzed publicly accessible AI-generated information and involved no human participants, human biological specimens, or identifiable personal data. The authors declared no competing interests. Those details do not change the clinical message: chatbot-generated medical information should be treated as a supplementary resource only, and every answer on pulmonary aspergillosis still needs professional verification before it is used to guide care, education, or triage.