Public Chatbots Gave Mixed-Quality Advice on Diabetic Kidney Disease
Large language models are moving into patient education faster than most clinics can write guardrails around them, and diabetic kidney disease is exactly the sort of condition that exposes their strengths and their blind spots. In a cross-sectional comparative study, six publicly accessible chatbots—ChatGPT, Gemini, DeepSeek, Doubao, Kimi, and Qwen—answered standardized patient-facing questions about diabetic kidney disease, with safety set as the primary outcome. The design was straightforward and clinically relevant: 42 English-language questions were submitted once to each model in independent single-turn sessions, producing 252 responses for review.
The point of the exercise was not to ask whether these systems can sound helpful. It was to determine whether they can give advice that is safe enough for patients who may already be navigating chronic kidney injury, medication changes, dietary confusion, and fear of progression.
Safety Was Similar Across Models, But Unsafe Answers Were Not Rare
Safety was the first signal clinicians should care about, and it did not cleanly separate one chatbot from another. Agreement among raters was substantial, with Fleiss’ κ reaching 0.854, and the safe-response rates clustered within a fairly narrow band, from 83.3% to 92.9%. Statistical testing did not show a significant difference between models for safety, with P = 0.726.
Thirty of 252 responses, or 11.9%, were judged potentially unsafe. Most of those problems came from two predictable failure modes: incomplete medication-related information and fluid advice that was too generic to be safely individualized. In nephrology, those are not minor editorial lapses. Medication instructions can collide with renin-angiotensin system blockade, diuretics, sodium-glucose cotransporter 2 inhibitors, or other therapies that need exact context, while fluid guidance in CKD cannot be copied from one patient to the next without risking volume overload or dehydration.
A chatbot that sounds reassuring can still be clinically imprecise.
Accuracy, Empathy, and Information Quality Pulled in Different Directions
Performance became more uneven once the analysis moved beyond safety. Accuracy differed significantly across models, with ChatGPT ranking highest on clinician assessment. Empathy also separated the systems, but not in the same way: Gemini received the strongest clinician-rated empathy scores. Information quality showed another pattern, with ChatGPT performing best overall and Doubao ranking lowest.
Those separations matter because patient-facing education is not a single skill. Accuracy requires correct pathophysiology and correct recommendations. Empathy requires language that patients can tolerate when they are anxious or discouraged. Information quality depends on structure, completeness, and relevance, especially when a patient asks about progression risk, blood pressure control, protein intake, or whether a medication change is dangerous. A model can sound warm and still omit a key caveat. It can also be precise and still feel brittle or dismissive. In clinical practice, both failures reduce trust, but they do so in different ways.
The study also used DISCERN, EQIP, JAMA benchmarks, and the Global Quality Scale, which gives the comparison a broader lens than simple impression scoring. Even with those tools, the models did not converge on a uniformly strong profile. ChatGPT emerged as the strongest overall performer in information quality, while Doubao sat at the bottom of that category.
Readability Stayed Above the Level Most Patients Can Reliably Use
Readability was the quietest but perhaps most consequential result. All six readability indices differed across models, with P values below 0.001 and Kendall’s W ranging from 0.139 to 0.255, showing only modest concordance across the text-complexity measures. DeepSeek had the lowest mean scores on all five grade-level indices and the highest mean Flesch Reading Ease score, yet even its text still exceeded the sixth-grade reference level. Every model did.
For a chronic disease such as diabetic kidney disease, that matters at the bedside. Patients are often asked to understand laboratory monitoring, blood pressure targets, glucose control, renal risk, medication tradeoffs, and dietary restrictions while also managing other diabetes complications. If a chatbot writes above a sixth-grade level, the burden shifts to the patient to decode it. That can widen the gap between information and action, especially for people who already face health literacy barriers.
Why the Failure Modes Make Sense Clinically
The pattern of errors fits the clinical context. Kidney disease advice is rarely one-size-fits-all because the same symptom can mean different things depending on eGFR, albuminuria, blood pressure, diabetes control, diuretic use, and comorbidity burden. Generative models can reproduce the broad outline of guideline language, but they are not inherently grounded in a patient’s medication list or volume status unless the prompt contains those details and the system handles them reliably. In real practice, missing context is enough to turn an otherwise reasonable sentence into a dangerous one.
Fluid advice is a good example. Too little nuance can lead to overgeneralized recommendations that ignore edema, heart failure, hyponatremia, or dehydration risk. Medication counseling has a similar problem: a broadly correct statement can still fail if it does not specify when to contact a clinician, when to hold a drug, or which combinations need monitoring. The study’s unsafe-response tally shows that this is not a theoretical weakness.
What Clinicians Can Use These Models For Now
Publicly accessible LLMs can support patient education in diabetic kidney disease, but the data here argue against using them as stand-alone educators. Their output was often plausible, and several models produced acceptable responses much of the time, yet none was consistently safe, fully accurate, highly readable, and clinically complete across all questions. The most defensible role is supervised support: a draft explanation, a patient handout companion, or a starting point for a clinician to revise.
The practical guardrails are clear from the results. Responses need guideline anchoring, source verification, and clinician oversight before they reach patients. Readability also needs deliberate attention, because even otherwise sound information may fail if the language sits above the patient’s comprehension level. For diabetic kidney disease, where delays and misunderstandings can change blood pressure control, medication adherence, and renal trajectory, those guardrails are not optional.
The study leaves one unresolved issue in plain view: whether model performance improves when prompts become more clinically specific, or when outputs are constrained by validated renal content and real-time citations. Until then, the safest use of these systems remains behind a clinician’s review, not in front of the patient alone.