Machine Learning Edges Ahead in Predicting Hospitalization and High Cost, But Only Slightly
A retrospective study from the Johns Hopkins Health System found that machine learning can improve prediction of next-year hospitalization and elevated total cost, but the gains over traditional regression were narrow. Investigators tested the Johns Hopkins Adjusted Clinical Group ACG System against logistic regression and three machine learning approaches, then tuned each model for either area under the receiver operating characteristic curve AUROC or F1 performance. The result was not a dramatic replacement of established methods. It was a measured advance, with XGBoost producing the strongest discrimination in most comparisons and logistic regression remaining highly competitive.
The most important signal in the study is how little room there was between the models. In a large cohort of 350,463 patients selected in 2019, logistic regression already performed well for both outcomes, reaching average AUROC values of 0.886 for cost and 0.841 for hospitalization. XGBoost improved those figures to 0.891 and 0.849, respectively. The differences were statistically significant, but they were small enough to matter mainly at scale, where even fractional gains can alter triage lists, case management queues, and resource planning across a health system.
The same pattern appeared when the models were optimized for F1 instead of AUROC. For hospitalization, logistic regression reached an average F1 score of 0.367, while XGBoost produced 0.328. For high cost, logistic regression averaged 0.341 and XGBoost reached 0.411. One model therefore did not dominate every task. Prediction of utilization and prediction of cost appear to behave differently, likely because the underlying signals are not identical: hospitalization often reflects acute destabilization, while high cost may capture chronic complexity, repeated encounters, pharmacy burden, and persistent multimorbidity.
Why the ACG System Remains a Strong Base for Risk Prediction
The ACG System already organizes clinical information into a structured framework that captures comorbidity burden and expected utilization. That structure gives conventional regression a natural advantage because the feature set is already clinically informed, curated, and statistically meaningful. When a model begins with such strong inputs, machine learning has less opportunity to create dramatic separation. In other words, the value of ML here is not that it rescued a weak data environment. It nudged an already strong system a little farther.
This helps explain why logistic regression continued to hold up so well. In settings where interpretability matters, the model’s transparency is not a minor convenience. Care managers, analysts, and physicians often need to understand why a patient was flagged, not simply that a score crossed a threshold. Logistic models also tend to remain stable under imbalance, especially when the goal is identifying a relatively uncommon outcome. For a health system, a method that is easier to explain and operationalize can be just as valuable as one that posts a marginally higher metric.
The study also speaks to the practical tension between discrimination and deployment. AUROC can make a model look broadly strong across thresholds, while F1 focuses more directly on the balance between precision and recall. Those are different lenses, and each one can favor a different algorithm depending on the outcome definition, the prevalence of the event, and the operational cost of false positives. In this analysis, the machine learning advantage was not uniform across the two optimization strategies, which suggests that model selection should follow the use case rather than the reverse.
One sentence captures the bedside consequence plainly: a slightly better score does not automatically mean a better workflow.
Class Imbalance Shaped the Comparison Between Models
The investigators also examined sensitivity, positive predictive value, and F-beta statistics, which is important because heavy class imbalance can distort how performance is perceived. Elevated cost was defined as spending above the 95th percentile of total amounts billed, including pharmacy costs. That creates a small target class by design, and small target classes can reward models that are better calibrated to rare events or more conservative in their positive predictions. Under those conditions, a model may appear superior on one metric while looking less impressive on another.
This is where the calibration problem becomes clinically relevant. If a model overcalls high-risk patients, care teams absorb more false alarms and waste attention on patients who may not need intensive intervention. If it is too conservative, the health system misses the very patients it hoped to identify. The study’s findings suggest that logistic regression often produced high-precision estimates for the outnumbered class, even when machine learning gained ground on some aggregate metrics. In a live population management program, that can be the difference between a manageable worklist and a noisy one.
XGBoost’s modest edge for cost prediction may reflect its ability to capture nonlinear relationships and higher-order interactions among ACG-derived features. Random forest and elastic net were also tested, but the summary emphasized XGBoost as the leading machine learning approach. The fact that the gains were modest, not sweeping, is itself clinically useful. It suggests that careful model choice can refine a mature prediction system, yet it does not justify discarding a familiar statistical method when interpretation, simplicity, and implementation speed are priorities.
What Health Systems Should Take From the Comparison
The study does not argue that machine learning should replace regression in utilization prediction. It shows that a well-tuned algorithm can slightly outperform a conventional baseline, but only under specific optimization settings and outcome definitions. For hospitals and integrated delivery systems, that distinction matters because predictive models are rarely built for elegance alone. They are built to steer staffing, outreach, care management, and financial planning.
The practical question is not whether a machine learning model can win on a benchmark. It is whether the gain is large enough to justify the added complexity of training, monitoring, explanation, and integration into existing ACG-related workflows. For a system already grounded in a robust risk stratification framework, the answer may often be yes for some use cases and no for others. The results favor a tailored approach: use the model that fits the operational burden, the available data structure, and the cost of a wrong prediction.
The authors also point to the need to consider how imbalance affects model calibration. In datasets like this one, where high-cost patients are a small minority and hospitalization risk varies across a broad clinical spectrum, apparent performance can shift depending on the metric chosen for tuning. That leaves health systems with a narrow but important mandate: define the clinical decision first, then choose the model that serves it best. A single winner is unlikely to exist across all prediction tasks, and this cohort makes that clear without overstating the reach of machine learning beyond its actual gains.