Creatinine-Based AKI Models Travel, but Not Cleanly, Between ICU Databases
Clinical teams want machine-learning tools that can predict acute kidney injury before creatinine rises become obvious at the bedside. In intensive care, that goal runs into a stubborn reality: the same laboratory signal can behave differently once it moves from one electronic health record system to another. This study tested whether a creatinine-based AKI prediction model could be transported between two large ICU datasets, MIMIC-IV v3.1 and eICU-CRD v2.0, without losing basic clinical usefulness.
The answer was mixed. Prediction was feasible in both databases, but the data pipeline itself exposed major differences in endpoint computability, lab coverage, measurement density, and calibration behavior.
The Endpoint Was a Post-24-Hour Creatinine Rise, Not Full KDIGO AKI
The investigators used retrospective secondary analysis of adult ICU stays and built predictions at approximately 24 hours after ICU admission using only information observed during that first day. This was not an admission-time model, and it was not designed to diagnose AKI at the moment of ICU entry. The primary outcome was a post-24-hour day-7 serum-creatinine-based operational endpoint, which required first-24-hour creatinine plus follow-up creatinine values through day 7.
That distinction matters at the bedside. Creatinine-based AI models often appear straightforward until missing follow-up labs, delayed sampling, and uneven measurement frequency start to shape the label itself. Here, the endpoint was intentionally narrower than full KDIGO staging. It was an operational creatinine-rise construct rather than proof of incident AKI in the strictest clinical sense.
Data Readiness Varied Sharply Between MIMIC-IV and eICU-CRD
The feasibility audit included 94,458 MIMIC-IV ICU stays and 200,234 eICU adult ICU stays. First-24-hour creatinine coverage reached 96.8% in MIMIC-IV, but fell to 84.6% in eICU. Endpoint computability diverged even more: the day-7 post-24-hour creatinine endpoint could be computed in 89.2% of MIMIC-IV stays, compared with 65.6% in eICU.
Those differences are not cosmetic. They tell you whether a model can even be trained on comparable labels before transport is attempted. A risk score built on one hospital’s dense laboratory culture may face a very different reality in a system where follow-up creatinine is less consistently measured. In practical terms, the model may not fail because the algorithm is weak; it may fail because the data ecology is unstable.
Predictive Performance Was Moderate and Similar Across Internal Validation
The modeling cohorts included 84,237 stays from 58,782 MIMIC-IV patients and 131,429 stays from 102,013 eICU patients. Primary endpoint event rates were 26.9% in MIMIC-IV and 18.5% in eICU. Investigators trained L2-regularized logistic regression models in each database using age, sex, first-24-hour renal and electrolyte laboratories, missingness indicators, and first-24-hour creatinine count.
Internal AUROC was 0.732 in MIMIC-IV and 0.723 in eICU. Those values suggest moderate discrimination rather than bedside-ready precision. The model could separate higher- from lower-risk patients better than chance, but not with the confidence needed for unguarded deployment. In ICU populations where creatinine changes can reflect hemodynamics, nephrotoxins, sepsis physiology, or fluid shifts, moderate discrimination is useful only if calibration and transportability remain stable.
External Validation Exposed Calibration Drift in Both Directions
When the models crossed databases, discrimination held up reasonably well. External AUROC was 0.703 for MIMIC-IV to eICU and 0.722 for eICU to MIMIC-IV, with confidence intervals reported in the main table. The more troubling signal came from calibration. The MIMIC-IV model overpredicted risk when applied to eICU, while the eICU model underpredicted higher-risk strata when transported to MIMIC-IV.
That pattern is clinically important because clinicians act on predicted probabilities, not AUROC alone. A score that ranks patients correctly but systematically misstates absolute risk can push thresholds in the wrong direction, especially if the ICU team uses a cutoff to trigger nephrology review, closer monitoring, or medication adjustment. Calibration shift usually reflects differences in case mix, laboratory ordering patterns, and measurement density as much as it reflects algorithm design.
Measurement Density Is a Model Variable, Not a Background Detail
This study treated missingness indicators and creatinine count as predictors, which is an important methodological choice. In ICU data, how often a lab is drawn can carry information about illness severity and clinician concern. A patient with repeated creatinine checks is not simply a patient with more data; that frequency may signal instability, kidney concern, or a treatment pathway that itself changes future risk.
The authors also centered the evaluation on measurement-density failure modes, which is where many real-world AI projects break. A model can appear portable if it sees the same variables in both systems, yet still lose validity when the timing and density of those variables differ. In kidney-risk prediction, the presence of a creatinine value is not enough. The rhythm of sampling matters too.
What This Means for ICU AI Deployment
The main contribution of this work is not a dramatic performance gain. It is a structured warning about data readiness before deployment claims are made. The study supports the idea of an EHR data passport for renal-risk AI, one that documents lab coverage, label computability, missingness structure, and calibration behavior before a model is moved across institutions.
It also reinforces a more conservative view of algorithmovigilance in nephrology. Cross-database validation cannot stop at AUROC. Transportability should be judged by whether the target system can support the endpoint at a clinically acceptable rate, whether the predictor distribution resembles the source setting, and whether predicted probabilities remain usable after transport.
No prospective deployment, no intervention assignment, and no patient-level outcome trial followed from these retrospective analyses. The endpoint did not establish full KDIGO staging or prove incident AKI at ICU admission. What the data did show, with unusual clarity, is that a creatinine-based ICU model can survive the journey between two major databases and still arrive altered by the measurement environment. Before renal AI enters routine use, the calibration drift and endpoint gaps need to be treated as first-order clinical constraints, not technical footnotes.