EHR-based deep learning model reaches beyond single-system data for opioid risk prediction
A new deep learning model trained to predict opioid use disorder at one year aims at one of the hardest problems in clinical informatics: turning fragmented longitudinal records into a risk signal that can travel across health systems. The model, called OUD-REY for OUD Risk Evaluator at 1 Year, was developed from longitudinal electronic health record data in the All of Us research program and then tested in temporally separated All of Us data as well as in a large Veterans Affairs dataset spanning 20 years.
The clinical use case is straightforward but consequential. Patients receiving opioid prescriptions are not a uniform group, and the record often contains the warning signs in pieces — prior diagnoses, medication histories, utilization patterns, and other coded events that accumulate over time. The challenge is that these signals rarely line up neatly across institutions. OUD-REY was built to read that kind of history at scale and convert it into a probability of future disorder.
On internal validation, the model achieved an AUROC of 91.2% and an AUPRC of 77.0%. Those figures suggest strong discrimination in the development environment, where the model had access to the patterns it had learned from the All of Us cohort. The more demanding question was whether the same architecture would preserve performance when moved into different data streams with different coding habits, different populations, and different time periods.
It did.
Language modeling helped the model handle unfamiliar clinical codes
The technical core of the work is a pretrained language model designed to absorb longitudinal EHR sequences and accommodate thousands of unseen, out-of-vocabulary clinical features. In practical terms, that matters because health records are messy. Different systems may encode similar conditions differently, and new or rare features can break more rigid models that depend on a fixed vocabulary. By using language modeling for cross-EHR harmonization, the investigators created a system intended to recognize clinical meaning even when the exact data token had not appeared in training.
This approach is especially relevant for OUD prediction because the exposure history is often heterogeneous. Opioid prescribing, pain conditions, mental health diagnoses, substance use codes, and care patterns may all carry risk information, yet none of those elements is guaranteed to be captured consistently across health networks. A model that can tolerate unfamiliar inputs without collapsing is better suited to the real-world EHR environment than a narrow classifier tuned to one institution’s coding structure.
External validation preserved discrimination in two independent settings
Performance fell from the internal validation set to the external cohorts, but the model remained strong enough to be clinically interesting. In the temporally separated All of Us validation set, OUD-REY reached an AUROC of 86.4% and an AUPRC of 23.1%, with OUD prevalence of 2.9%. In the VA dataset, performance was similar, with an AUROC of 85.5% and an AUPRC of 19.5%, alongside OUD prevalence of 3.5%.
Those external results carry the most weight. AUROC in the mid-80s across two large, distinct datasets indicates that the model retained useful ranking ability outside its development environment. The lower AUPRC values are not surprising given the low prevalence of OUD in the validation sets; precision-recall performance becomes much harder to sustain when the outcome is uncommon. Even so, the model continued to separate higher-risk from lower-risk patients in settings that differed by data source, era, and clinical context.
The temporal split inside All of Us is particularly important because it tests whether the model is merely echoing a static pattern from one snapshot in time or whether it can withstand temporal drift. A tool that fails when the calendar changes will not last long in routine care. Here, the decline in performance was real but not catastrophic, suggesting the architecture captured durable clinical structure rather than a fragile set of system-specific quirks.
Fairness auditing found broadly comparable performance across demographic groups
The investigators also audited the model for fairness and found it to be generally fair across patients of different genders, races, and ethnic backgrounds. In a task with direct implications for triage and prevention, that kind of evaluation cannot be an afterthought. Risk algorithms in substance use care can amplify existing disparities if they rely too heavily on access patterns, utilization intensity, or coding practices that differ across groups.
A model that performs unevenly across demographic categories can quietly redirect attention away from patients who need help or over-flag groups already burdened by surveillance. The fairness audit does not remove the need for local evaluation, but it does indicate that the model did not show an obvious demographic skew in the tested datasets. In a domain where mistrust of algorithms is justified, that finding is clinically meaningful.
Prevention workflows may be the clearest near-term use
The most immediate value of OUD-REY is not as a diagnostic substitute but as a prevention aid. A one-year risk estimate could support clinicians, care managers, and health systems trying to identify patients who may benefit from closer follow-up, safer prescribing decisions, counseling, or earlier referral pathways. Because the model operates on longitudinal EHR data, it is designed to work with information already present in the chart rather than requiring a new testing workflow.
The burden of opioid-related harm unfolds over time, and the value of prediction is in the interval before a disorder becomes entrenched. If a model can surface patients at higher risk during the prescribing window, it may help redirect attention toward intervention before escalation. The biological logic is not mysterious: repeated exposure, comorbid pain and psychiatric conditions, and patterns of healthcare use often cluster before formal OUD emerges. A model that detects those clusters early can give clinicians a way to act while there is still room to change the trajectory.
What the data can and cannot yet prove
Several limits remain visible in the data presented here. The article provides discrimination metrics and fairness results, but it does not establish whether using OUD-REY in practice improves prescribing behavior, increases treatment uptake, or reduces overdose or new OUD diagnoses. No prospective clinical implementation study is presented, and no bedside workflow is tested.
The external validation results are reassuring, yet they are still retrospective. Real-world deployment would need calibration review, local adaptation, and prospective monitoring for unintended consequences. Predictive tools in addiction medicine can easily become another layer of screening without changing care unless they are embedded into actionable pathways.
Even with those caveats, the central message is clear: a deep learning model trained on longitudinal EHR data can predict opioid use disorder with strong discrimination and hold up across two large external datasets. The next question is not whether the model can score risk, but whether health systems can use that signal in time to change what happens next.