BRII Exposes What High-Risk Alerts Actually Contain
Clinical prediction tools often look strong when judged by AUROC alone, yet that score can hide what happens inside the group of patients flagged as high risk. In intensive care, where death can preempt the target event and follow-up can end early, a model may appear accurate while its alert cohort is dominated by censored patients or by competing outcomes rather than the event clinicians want to prevent. The Balanced Risk Integrity Index, or BRII, was designed to address that gap by measuring the proportion of target events among target events, competing deaths, and early-censored observations within a flagged stratum.
The metric is intentionally narrow in scope. It does not replace discrimination measures such as AUROC or calibration tools; instead, it quantifies the composition of the patients who actually get pushed into the highest-risk alert tier. For bedside workflows, that matters because an alert cohort made up largely of patients who will be censored early is very different from one enriched for the event of interest. In acute kidney injury prediction, where timing, attrition, and mortality all interact, that distinction can change how an alert should be trusted or operationalized.
Seven AKI Models Across Three ICU Databases
Investigators evaluated BRII using seven acute kidney injury prediction models built in MIMIC-IV and externally validated in MIMIC-III and eICU-CRD. The combined analysis included 116,010 patients. The model families covered common prediction approaches, including logistic regression, XGBoost, LightGBM, and survival-based methods such as cause-specific Cox proportional hazards and cause-specific random survival forest, alongside inverse-probability-of-censoring-weighted variants. The clinical target was AKI prediction over defined horizons, including a 48-hour prediction task examined in the top 10% risk stratum.
A single ranking metric was not enough to describe how these models behaved once they moved across datasets. In MIMIC-III, BRII for 48-hour AKI prediction in the top 10% stratum ranged from 0.90 to 0.96. In eICU-CRD, the same measure stayed below 0.26 across all models. The contrast was driven by the structure of the validation cohort rather than by any one model alone: 61.5% of evaluated patients in eICU-CRD were censored before the prediction horizon.
Censoring Swamped the Flagged Stratum in eICU-CRD
The flagged eICU-CRD patients tell the story more plainly than the aggregate score does. Within the top 10% risk strata for the 48-hour AKI target, 37.0% to 50.2% of patients were censored early, 12.2% to 15.5% developed AKI, and 3.4% to 8.1% died before AKI. In practical terms, the alert population was not simply a cleaner slice of impending AKI. It was a mixed bag of incomplete observation, competing mortality, and a relatively small target-event fraction.
That composition has biological and operational consequences. When critical illness progresses rapidly, death can remove a patient from the risk set before AKI is ever observed, and heavy censoring can erase the chance to verify the target outcome within the horizon. A model that marks many such patients as high risk may still be useful for triage, but it is not offering the same kind of signal as one that captures a tighter concentration of true AKI events. BRII makes that difference visible instead of burying it inside a single binary endpoint.
BRII and PPV Ranked Models Differently
BRII is algebraically related to positive predictive value, yet the two measures did not behave identically across the evaluated settings. Across 48 cohort-outcome-top-k groups, BRII and PPV ordered at least one pair of models differently in 34 groups. This means that a model favored by PPV could look less coherent once early censoring and competing death were folded into the composition of the alert stratum, or the reverse could happen.
The divergence is not a statistical curiosity. It reflects a real tension in prediction-alert design: whether the highest-risk patients are truly accumulating the event of interest, or whether the alert is being diluted by patients who cannot be fully observed or who experience another terminal outcome first. For hospital teams building escalation pathways, those are different populations, and they should not be treated as interchangeable.
BRII also carries a decomposition that separates target events, competing deaths, and early-censored observations. That structure allows a clinician or model evaluator to see why a stratum performs poorly, not just that it performs poorly. A low BRII can arise because a model overflags patients who die first, because follow-up ends too soon, or because the target event is genuinely rare in the flagged group. Those are not equivalent failure modes.
Random Censoring Altered BRII Far More Than AUROC
Simulation results made the sensitivity of BRII even more explicit. Under simulated random censoring for 48-hour AKI prediction, BRII declined by as much as 0.80. Over the same simulation conditions, binary-label AUROC changed by less than 0.02. The gap between those responses shows why AUROC can remain nearly stable while the clinical meaning of the alert cohort shifts dramatically.
AUROC ranks patients by score, but it does not tell you whether the top-ranked patients are available for observation, whether they die before the endpoint, or whether the endpoint is even interpretable in the same way across cohorts. BRII was built to capture exactly those omissions. In a setting like ICU AKI prediction, where early death and early discharge are constant threats to endpoint clarity, a metric that ignores censoring can preserve mathematical elegance while losing clinical relevance.
A Stratum-Level Metric for Alert Governance
BRII does not claim to solve every metric problem in predictive medicine. It is best viewed as a companion measure for situations where competing risks and incomplete follow-up reshape the meaning of a positive alert. Used alongside PPV, BRII can show whether a model is finding a cohort dense in the target event or merely a cohort that looks high risk because the alternative outcomes are stacking up.
For external validation, the findings also argue against treating one cohort as a universal benchmark. The same model family can behave very differently across ICU databases because case mix, observation windows, and censoring patterns are not interchangeable. In that setting, a score that explicitly reports the fraction of target events, competing deaths, and early-censored cases offers a more disciplined way to compare models than discrimination metrics alone.
The preprint is an important methodological contribution, but it remains a preprint. The next step is not simply wider use of BRII; it is testing whether the metric improves real-world alert governance, threshold selection, and workflow design in clinical environments where AKI prediction is deployed under active competing risk and incomplete follow-up.