The authors introduced the Balanced Risk Integrity Index (BRII) as a stratum-level measure that isolates the composition of outcomes among patients flagged by a clinical prediction model. BRII is defined as the proportion of target events among the sum of target events, competing deaths, and early-censored observations within a flagged high-risk stratum. The study team noted that BRII is algebraically related to PPV and recommended that BRII be reported together with a component decomposition that separately lists the shares of target events, competing deaths, and early censoring. This decomposition is intended to make explicit what the flagged cohort actually contains, rather than relying on discrimination metrics alone.
BRII addresses a concrete shortcoming of conventional metrics like AUROC: those metrics quantify discrimination but do not describe what proportion of flagged patients actually experience the outcome versus die beforehand or leave observation early. The authors framed BRII as a complementary, stratum-focused metric for settings—such as critical care—where competing risks and incomplete follow-up are common and materially affect clinical alert interpretation.
The evaluation used seven acute kidney injury (AKI) prediction models developed in MIMIC‑IV and externally validated in MIMIC‑III and eICU‑CRD, encompassing 116,010 patients in total. For 48-hour AKI prediction in the top 10% risk stratum, BRII values in MIMIC‑III ranged from 0.90 to 0.96, indicating that most flagged patients were classified as target events rather than competing deaths or early censored observations in that cohort. By contrast, BRII was less than 0.26 across all models in eICU‑CRD.
The study team reported that eICU‑CRD had a large share of patients censored before the prediction horizon—61.5% of evaluated patients—affecting the composition of high-risk strata. In the eICU‑CRD top 10% strata for 48-hour AKI prediction, the component decomposition showed 37.0% to 50.2% of patients were early-censored, 12.2% to 15.5% experienced AKI, and 3.4% to 8.1% died before AKI. Those component proportions explain the low BRII values in eICU‑CRD despite model discrimination measured elsewhere.
The authors compared BRII with PPV and observed that the two metrics can rank models differently. Across 48 cohort–outcome–top-k groupings, BRII and PPV ordered at least one model pair differently in 34 instances, indicating that BRII captures information about competing events and censoring that PPV alone does not.
To probe sensitivity to observation processes, the investigators ran simulations of random censoring for 48-hour AKI prediction. Under increased random censoring, BRII dropped substantially—by up to 0.80 in the reported experiments—while binary-label AUROC changed by less than 0.02. The simulation results indicate that BRII is far more responsive to changing censoring patterns and competing-risk prevalence than standard binary-label discrimination metrics.
These findings suggest that two models with similar AUROC values might produce markedly different clinical alert compositions when applied in settings with different censoring or competing-death dynamics. The authors therefore positioned BRII as a complement to discrimination metrics for evaluating real-world alert behavior across observation environments.
The study team recommended that model evaluations present BRII together with its component decomposition so clinicians and implementers can see what fraction of flagged patients are true target events, how many are censored early, and how many experienced competing deaths. Presenting those components allows interpreters to assess whether a high-risk alert population is actionable: whether alerts mostly capture the intended outcome, or instead reflect patients lost to follow-up or who die before the outcome could occur.
The authors argued for standardized reporting of BRII across models and cohorts to enable transparent comparisons. They provided examples from their external validations where BRII clarified why a high-risk stratum in one ICU dataset might be clinically meaningful while the same-risk stratum in another dataset contains few observable target events because of censoring or competing deaths.
The analysis used publicly available critical-care datasets (MIMIC‑IV, MIMIC‑III, eICU‑CRD) accessible through PhysioNet. Source data for figures and tables were provided with the paper, according to the authors. The manuscript states that all databases contain de‑identified data and that use of these data is exempt from institutional review board approval under the cited regulation; access was obtained through PhysioNet Credentialed Health Data Use Agreements.
Limitations and open questions remain tied to the specifics of real-world deployment: the authors demonstrated BRII’s sensitivity to censoring and competing events, but the paper did not prescribe concrete operational thresholds for acceptable BRII values or list regulatory milestones for adoption. The study team did not report additional implementation guidelines or prospective clinical evaluation results in this preprint. The authors concluded that BRII yields complementary, stratum-level insight when interpreted alongside PPV and its component decomposition, and that this combined reporting provides a standardized framework for evaluating clinical prediction alerts across models and observation environments.