Large language models strengthen multimodal EHR prediction across 57 studies
Electronic health records hold a mixture of structured fields and free text, and the clinical value of that unstructured material has long been difficult to capture at scale. A systematic review in npj Digital Medicine examined how large language models are being used to combine those data streams for predictive modeling, moving beyond traditional workflows that rely mainly on coded variables. The review screened 641 studies and included 57 that met eligibility criteria, offering one of the clearest snapshots yet of how language models are being folded into EHR-based prediction.
The central finding is straightforward. Across the included studies, multimodal approaches that incorporated language models outperformed single-modality models, with an average 6.4% increase in performance. In practical terms, the signal suggests that clinical text is not just a narrative companion to laboratory values, medications, and diagnosis codes. It is a source of predictive information in its own right, capturing subtle temporal cues, symptom framing, clinician concern, and disease context that structured fields often miss.
One sentence captures the shift better than any methodological gloss: free text changes the predictive floor.
Three integration strategies dominate current model design
The review identified three broad strategies for LLM-based multimodal integration: embedding, extraction, and text fusion. Each represents a different answer to the same problem—how to make clinical language computable without stripping away too much meaning.
Embedding approaches use language models to convert notes into dense numerical representations that can be fed into downstream prediction systems. This method is attractive because it preserves a broad slice of the semantic content in a compact form, making it useful when the target outcome depends on language patterns rather than discrete keywords alone. Extraction strategies take a more structured route, pulling clinically informative variables from text and converting them into features that sit alongside traditional EHR data. Text fusion, by contrast, summarizes or condenses long documents before integration, which can reduce noise and help align long-form narrative with model inputs that have practical size limits.
These are not interchangeable techniques. Each one reflects a different tradeoff between fidelity, interpretability, and computational burden, and the review places those tradeoffs at the center of current model development. A model that embeds every sentence may be powerful but opaque; a model that extracts only named variables may be easier to explain but miss contextual signals such as uncertainty, negation, or evolving clinical trajectory.
Why unstructured notes can improve prediction
The attraction of multimodal clinical predictive modeling lies in what text contributes that structured data cannot fully express. Progress notes, discharge summaries, consultation notes, and other narrative documents often contain clinician assessments of severity, response to treatment, symptom evolution, and social or behavioral details that rarely fit cleanly into billing codes or point-in-time fields. Language models can encode those nuances into representations that a predictive algorithm can use.
That matters biologically and operationally. Disease is rarely static, and the note often captures the clinical moment when risk becomes visible before it is formally coded. For a deterioration model, a phrase indicating rising work of breathing, poor oral intake, or worsening confusion may precede a charted diagnosis change. For a readmission model, the note may reflect discharge readiness, unresolved concern, or medication adherence barriers. The performance gains reported across the included studies are consistent with that logic: the model performs better when it sees more of the clinical picture.
Sample size, interpretability, and compute shape real-world adoption
The review does not present these systems as turnkey replacements for conventional predictive analytics. It instead identifies several design considerations that should shape future work, including sample size, interpretability, and computational costs. Those concerns are not peripheral. They determine whether a model can be trusted, reproduced, and deployed in a real health system rather than confined to a retrospective benchmark.
Interpretability remains especially important in medicine because better discrimination alone does not settle the question of clinical usefulness. A model can rank risk well and still be difficult to audit if its inputs are hidden inside a complex embedding space. Clinicians need to know whether a prediction is being driven by a transient documentation artifact, a meaningful symptom trajectory, or a feature that reflects documentation habits rather than patient biology.
Computational cost also matters because LLM-enabled pipelines can demand far more memory, processing time, and infrastructure than smaller classical models. In a hospital environment, those requirements affect latency, maintenance, and whether deployment is feasible across multiple care settings. Sample size, meanwhile, remains a persistent constraint because the more expressive the model, the more vulnerable it may be to overfitting when training data are sparse or inconsistently labeled.
What the review supports, and what it does not yet settle
This synthesis supports the use of language models as a meaningful addition to EHR prediction, but it does not imply that all multimodal architectures are clinically ready. The included studies were diverse in design, data sources, and integration strategy, which is exactly what makes the average performance gain informative and the next steps necessary. Heterogeneity in outcome definition, text source, and evaluation design limits the ability to claim a single best method for all settings.
The review also points toward a broader methodological reality: predictive performance gains are easiest to obtain when a model is allowed to absorb richer clinical context, yet the same richness makes validation harder. A model that draws on long-form narrative may capture more true signal, but it also becomes more sensitive to institutional documentation style, specialty-specific phrasing, and differences in note structure across sites.
The practical implication is clear for informatics teams and clinical investigators alike. Future LLM-enhanced predictive modeling will need to balance accuracy with transparency, calibration, and resource demands if it is to move from promising retrospective performance to dependable clinical use. The review supplies a framework for that work, but it also leaves a difficult implementation question on the table: which integration strategy can preserve the clinical value of unstructured data without making deployment too costly or too opaque for routine care?