NLP Comorbidity Extraction Finds Promise and Gaps in Dementia Records
In dementia care, comorbid disease is not background noise. It shapes prognosis, medication burden, hospitalization risk, and the interpretation of every cognitive assessment. A preprint from researchers at the University of Leicester and King’s College London evaluates whether a natural language processing tool can recover that broader clinical picture from unstructured electronic health record text with enough fidelity to support research in people with dementia.
The study examined the MedCAT Comorbidity app in a cohort of 3,543 people with dementia drawn from South London and Maudsley NHS Mental Health Trust records and linked primary care and hospital data. The design was deliberately layered: internal validity was tested against manual review of clinical text, external validity was checked against structured data from linked sources, and the resulting comorbidity counts were then tested for their ability to predict mortality. That sequence matters because a tool can look accurate inside one record system and still fail when it is asked to represent disease burden across fragmented care settings.
One result stood out immediately. Internal precision reached 100% for atrial fibrillation, yet it fell to 78% for diabetes. Precision is only part of the story, though. Some comorbidities were documented clearly enough in free text to be extracted with high confidence, while others were more likely to be mentioned inconsistently, coded differently, or spread across multiple encounters in ways that reduced the algorithm’s consistency.
Wide Variation Across Conditions Exposed the Limits of a Single Data Stream
External validation painted a less uniform picture. Across conditions, precision ranged from 96% to 59%, and recall ranged from 77% to 4%. Those are not small differences at the margins; they indicate that performance depended heavily on which comorbidity was being measured and which comparator source was used. When recall falls to 4%, many true cases are simply not being captured, which has direct consequences for comorbidity prevalence estimates and downstream modeling.
The pattern is clinically intuitive. Dementia patients often receive care across psychiatry, primary care, acute hospitals, and long-term community services, and no single record system reliably holds the full history. Free text may contain rich narrative detail, but it can also omit a condition that is clinically active yet not repeated at every visit. Structured hospital codes may be cleaner for some diagnoses and blind to others. Primary care data can be more complete for chronic disease, but not every system is linked or accessible in real time.
This is where the study becomes more than a software benchmark. It is a reminder that comorbidity ascertainment is a data-integration problem as much as a language problem. An NLP app can only extract what clinicians actually wrote, and what they wrote often reflects the purpose of the encounter rather than a complete catalog of chronic illness.
Mortality Modeling Favored Hospital Data Over NLP Alone
The comorbidity burden identified by the MedCAT app still carried prognostic information. Mortality risk rose by HR 1.11 per condition in the model using NLP-derived comorbidity counts. In plain clinical terms, each additional comorbidity identified through the text-mining approach was associated with a higher hazard of death, consistent with the well-established impact of multimorbidity in dementia.
Yet the model fit improved when comorbidity was drawn from hospital admission data. That result is important because it suggests the NLP output was informative but not the strongest available representation of disease burden once structured acute-care data were included. Hospital records may capture severe events and diagnoses with greater specificity, while the NLP app may be better at detecting broader context in settings where linked data are missing.
A median of two comorbidities emerged when any single source was used. When sources were combined, the median rose to three. The increase is small in absolute terms, but it shows how much disease burden can be hidden when researchers rely on one view of the record. In dementia research, where frailty, vascular disease, diabetes, atrial fibrillation, and other long-term conditions all influence outcomes, undercounting comorbidity can distort both risk adjustment and cohort characterization.
Documentation Gaps, Not Just Algorithm Error, Shaped the Results
The preprint’s central message is not that NLP failed. It is that performance is uneven when a general-purpose extraction app is applied to a specific clinical subgroup. High precision in internal validation means the tool often identified the right condition when it said a condition was present. The drop in external performance shows that documentation practices and source completeness exert a powerful influence on apparent accuracy.
Incomplete documentation likely explains part of the mismatch between free text and linked structured data. A condition may be present in one source and absent in another for reasons unrelated to true clinical status: a diagnosis may have been managed elsewhere, omitted from a note, or recorded in a format the extraction pipeline did not capture well. In dementia populations, where care is frequently distributed across multiple services, those gaps are likely to be amplified.
The practical consequence is straightforward. Researchers using NLP-derived comorbidity data in dementia cohorts should not assume that one extraction pass is enough. Internal review against original text and external comparison against linked records both add value, especially before the output is used to define study populations, adjust models, or compare outcomes across groups.
Why This Matters for Dementia Research Infrastructure
The work also speaks to a broader infrastructure problem in health data science. General data extraction tools are often designed to be transferable across populations, but transferability can break down when the clinical subgroup has unusual documentation patterns or complex service use. Dementia care is one of those settings. Psychiatric notes, memory clinic assessments, general practice records, and hospital admissions each hold different pieces of the clinical history, and no single source guarantees completeness.
For investigators, the strongest implication is methodological rather than diagnostic. Quality assurance needs to be built into NLP workflows, not added afterward as an afterthought. Internal precision, external validity against linked sources, and a check against an outcome such as mortality provide three distinct lenses on whether a tool is actually fit for purpose in the intended cohort.
The authors also position the MedCAT Comorbidity app as potentially useful where primary care or hospital linkage is unavailable. That is a reasonable use case, but the study makes clear that the output should be interpreted as an estimate shaped by documentation density and source availability, not a perfect census of disease burden.
This is a preprint, and it has not been peer reviewed. The underlying lesson is concrete: in dementia cohorts, NLP can enrich comorbidity ascertainment, but its output should be tested against the clinical record it is meant to summarize, and against linked data when those data exist. The biggest failure mode is not a dramatic false positive signal; it is silent under-ascertainment of chronic illness in a population where every omitted condition can change the analytic picture.