Random-Forest HFNC Prediction Falters Outside the Derivation Cohort
A machine-learning model built to predict high-flow nasal cannula failure in older adults with COVID-19 performed well in its internal testing set, then lost that footing when moved to a separate hospital cohort. The contrast is clinically important because bedside escalation decisions in hypoxemic COVID-19 often hinge on the narrow window before respiratory collapse, when delays in intubation or other rescue therapy can worsen outcomes. Here, the model’s apparent internal strength did not translate into dependable external discrimination.
Investigators retrospectively analyzed 238 patients aged 65 years and older who received HFNC. Failure occurred in 161 patients, a rate of 67.6%, which signals a heavily enriched high-risk population and helps explain why prediction modeling looked attractive in the first place. Candidate algorithms were trained on 70% of the derivation cohort and then screened on the remaining 30% hold-out set, which also served as the algorithm-selection set for the final performance estimate. The selected 13-predictor random-forest model, RF-13, reached an AUC of 0.812 in that internal hold-out sample, with a 95% bootstrap CI of 0.700 to 0.908.
Then came the cross-site check.
Transportability Broke Down in the External Cohort
When the same RF-13 model was applied without modification to a 26-patient cross-site cohort, its AUC fell to 0.527, with a very wide confidence interval of 0.273 to 0.767. In practical terms, that interval spans both poor performance and potentially useful discrimination, which makes the transportability question unresolved rather than answered. The model did not provide a stable basis for bedside decision-making across institutions, especially when the sample was this small and the uncertainty this large.
That gap between internal and external performance is a familiar failure mode in clinical machine learning. A model can capture local patterns tied to documentation style, case mix, treatment culture, or hospital-specific thresholds for HFNC initiation and intubation, yet still fail when those signals shift. In older adults with COVID-19, where physiologic reserve varies widely and escalation thresholds are often individualized, overfitting to one center’s practice environment can be enough to erode generalizability.
The study also evaluated ROX-based indices and performed a post-hoc severity-excluded sensitivity analysis. Those additional checks did not restore confidence in the main model’s portability. Decision-curve estimates for RF-13 and the exploratory RF-5 model were driven by very few reclassified patients, limiting the clinical meaning of the net-benefit curves. A prediction tool can look statistically elegant and still leave the clinician with too little information to change management safely.
The Smaller Post-Hoc Model Looked Better, But the Signal Was Fragile
A five-predictor model, RF-5, was retained only as a post-hoc exploratory analysis. It posted an AUC of 0.681 in the algorithm-selection set and 0.788 in the cross-site cohort. On paper, the external estimate appears stronger than RF-13. In context, the reversal is not persuasive evidence of improved generalizability because the estimate was imprecise, the model was selected after the fact, and the cross-site sample was tiny.
The authors also compared performance with APACHE II, which achieved a cross-site AUC of 0.700, with a 95% CI of 0.466 to 0.908. Even that established severity score remained imprecise in the external cohort, which is a reminder that small validation samples can make several tools look statistically plausible while none is truly settled. In older adults with COVID-19, a broad acute-physiology score may still be easier to interpret at the bedside than a bespoke machine-learning construct, especially when the latter has not survived a rigorous multicenter test.
A biologically plausible prediction model for HFNC failure would need to reflect the same processes clinicians track in real time: oxygenation trajectory, inflammatory burden, respiratory muscle fatigue, and the point at which gas exchange can no longer be rescued without more intensive support. The 13 predictors used in RF-13 may have captured some of those dynamics in the derivation center, but the cross-site result shows that predictive biology and predictive portability are not the same thing. One hospital’s signal can become another hospital’s noise.
What This Means for Bedside Escalation in Older Adults
Older adults with COVID-19 are a particularly difficult group for HFNC triage because frailty, comorbidity, and baseline pulmonary reserve vary so widely. Clinicians need models that are not just accurate in retrospect but stable under transfer to a new setting with different staffing, referral patterns, and treatment thresholds. This study does not provide that assurance for either RF-13 or RF-5.
The most actionable finding is negative: the external data were too sparse to establish or exclude transportability with confidence. The apparently higher RF-5 cross-site AUC should not be read as a rescue signal, because the analysis was post hoc and imprecise. Likewise, the RF-13 confidence interval was too wide to support a clear judgment about whether the model truly fails outside its home environment or simply needs a larger validation sample.
The study was retrospective and used de-identified clinical data, with ethics approval from the relevant committees and a waiver of written informed consent. Funding came from provincial Chinese sources, with no role in study design, data collection, analysis, interpretation, manuscript preparation, or submission. Those are standard safeguards, but they do not change the central limitation: the cross-site cohort was only 26 patients, far too small for a dependable assessment of real-world model portability.
For now, the model belongs in the research queue, not the bedside workflow. New multicenter validation with adequate sample size is needed before either random-forest approach can be considered for clinical use.