Machine learning models separate mortality classes with strong discrimination but imperfect calibration
Internal validation of mortality classifiers has become more sophisticated, yet the clinical bar remains high: a model must do more than sort cases into buckets. It must be reliable enough to guide decisions, and its probabilities must mean something at the bedside. In this study, investigators evaluated all-cause mortality, cardiovascular mortality, and stroke mortality using nested cross-validation across several machine learning approaches, and the central pattern was clear. Discrimination was often strong. Calibration was not.
The analysis included 4,137 sampled participants with linked binary mortality labels: 1,473 all-cause deaths, including 821 cardiovascular deaths and 223 stroke deaths. Four single-task models were compared — Elastic Net, Random Forest, XGBoost, and support vector machine — alongside an exploratory joint multinomial Elastic Net. Model development used 5-fold outer and 3-fold inner nested stratified cross-validation, with imputation, encoding, normalization, and class up-sampling confined to the training folds. That design matters because it limits leakage, a common failure point in predictive modeling that can inflate apparent performance and make a model look cleaner than it really is.
Elastic Net led the all-cause and cardiovascular tasks
For all-cause mortality, Elastic Net and the support vector machine reached similar AUROC values of 0.925 after rounding. Elastic Net produced the highest AUPRC at 0.886, which is particularly relevant in imbalanced data where precision-recall performance often tells the more honest story. In practical terms, the model was not simply separating survivors from decedents in a broad sense; it was doing so with enough consistency that false-positive burden could remain manageable in a classification setting.
Elastic Net performed best for cardiovascular mortality as well, with an AUROC of 0.873 and an AUPRC of 0.550. This is the clearest result in the study. Cardiovascular death carries a biological structure that may be more predictable from baseline features than many other fatal outcomes because it reflects cumulative vascular injury, cardiometabolic burden, and systemic inflammation. In machine learning terms, those signals can be captured by sparse regularized models when the relationship between predictors and outcome is nonlinear but still partly ordered by known physiology.
A single sentence captures the key tension in the dataset: high discrimination did not mean trustworthy absolute risk.
Stroke mortality remained harder to classify
Stroke mortality behaved differently. The support vector machine achieved the best AUROC at 0.837 and the highest AUPRC at 0.189. Those values are respectable in a difficult task, but they also show how much harder stroke death was to predict than all-cause or cardiovascular death. Stroke mortality usually sits at the intersection of vascular risk, neurologic severity, competing causes of death, and timing effects that static baseline labels cannot fully absorb.
The low AUPRC is especially important here. Even a model with a reasonable AUROC can have limited practical utility when positive cases are sparse and the decision threshold must contend with many false alarms. In stroke prediction, the cost of misclassification is not abstract. A poorly calibrated classifier can lead clinicians or researchers to overestimate the likelihood of death in patients who are in fact lower risk, or underestimate the danger in those with a very narrow physiologic reserve.
The exploratory joint multinomial Elastic Net did not improve discrimination and performed worse than single-task Elastic Net for cardiovascular mortality. Multi-outcome modeling can be attractive because real patients do not present as isolated labels, but shared structure does not guarantee better prediction. If outcomes differ in biological drivers, event frequency, and feature-response relationships, forcing them into one multitask framework can dilute the strongest signal rather than strengthen it.
Calibration gaps limit clinical translation
The study did not stop at discrimination metrics. Calibration curves and statistics showed meaningful deviations from ideal calibration, and that result sharply constrains interpretation. Rebalanced probabilities should not be treated as validated absolute-risk estimates. A model can rank-order patients well and still systematically overstate or understate their true event probabilities. For decision support, that difference is decisive. Clinicians need more than “high risk” labels; they need probabilities that correspond to actual event rates if those outputs are going to shape prevention strategies, follow-up intensity, or resource allocation.
Nested cross-validation strengthens confidence in internal performance, but it does not solve the calibration problem. Fold-contained preprocessing reduces contamination, yet the absence of outcome dates also limits the meaning of the prediction task. These models classify mortality status, not time-to-event trajectories. Without the temporal structure of death, the models cannot answer when risk emerges, how quickly it changes, or whether the signal is stable across follow-up windows. That is a fundamental boundary, not a minor statistical footnote.
Age and lactate dehydrogenase emerged as prominent features
SHAP beeswarm and dependence plots identified age and lactate dehydrogenase as prominent nonlinear attribution features. Age is unsurprising, but its prominence still matters because it anchors the models in a biologically plausible gradient of vulnerability: older patients accumulate vascular injury, frailty, and comorbidity, all of which raise the probability of death across categories. Lactate dehydrogenase is more interesting. Elevated levels often reflect tissue injury, hypoxia, hemolysis, or broader systemic stress, so its appearance as an important feature suggests the models may be capturing a nonspecific but clinically meaningful burden of illness rather than a single organ lesion.
This pattern is consistent with what machine learning often finds in mortality work: the strongest predictors are not always disease-specific biomarkers but markers of physiologic reserve and multisystem strain. In a hospital or screening database, those features can carry substantial signal, even when they do not point to one single diagnosis.
What this means for mortality modeling now
The study offers a solid internal-validation framework for mortality-status classification, but it stops short of providing deployable risk prediction. External validation is still required. Recalibration within each fold is also needed if the goal is to estimate probabilities that clinicians can trust. Outcome dates would allow the next step: moving from static classification to genuine prognostic modeling with time-aware endpoints.
For now, the practical value lies in methodological discipline. The models were trained with leakage controls, tested with nested cross-validation, and interpreted with explainability tools, which makes the reported discrimination credible within the sampled cohort. The ceiling is still visible. In cardiovascular mortality, a sparse regularized model did well. In stroke mortality, performance fell. And in every outcome, calibration remained too uneven to support absolute-risk use without further work. The next iteration has to connect classification with time, probability, and external transportability before these tools can influence routine care.