The study team proposed three simple, case-level metrics to describe when AI and radiologists disagree and in which direction that disagreement falls. Positive Non-Agreement (PNA) captures cases where the AI was correct but the radiologist was incorrect. Negative Non-Agreement (NNA) captures the opposite — radiologist correct, AI incorrect. Net Non-Agreement (NENA) is their arithmetic difference and expresses the overall direction of non-agreement across a dataset. The authors reported these definitions and stressed that the measures complement, rather than replace, conventional performance metrics because they register who is right on the same cases, not just aggregate accuracy rates.
Investigators illustrated the metrics in three hypothetical examples, each containing 1,000 examinations. These toy scenarios showed how NENA can be positive in some settings and negative in others depending on the balance of PNA and NNA. The illustrative examples also served to clarify interpretation: a positive NENA indicates net cases where AI corrected radiologist error; a negative NENA indicates a net advantage for the radiologist. The source did not report additional synthetic data beyond the broad outcomes described in the abstract.
The authors applied the directional metrics to three previously published comparisons between AI platforms and radiologists involving pulmonary embolism and mammography. Data indicated that across those studies, NENA values ranged from -4.18% to +10.74%. The paper reported a consistent trend with radiologist experience: junior residents had a higher positive NENA (+6.27%) while consultants showed a smaller positive NENA (+1.45%). The study team attributed this gradient mainly to a reduction in PNA as radiologist experience increased, coupled with only modest rises in NNA. The source did not supply further patient- or image-level details beyond these comparative results.
To quantify uncertainty the investigators calculated bootstrap confidence intervals for the directional metrics. Bootstrap-derived intervals framed the reported NENA estimates and determined whether observed directions differed reliably from zero. One of the study comparisons produced a confidence interval for NENA that included zero; the investigators reported that for this weakest comparison no meaningful reclassification fragility index could be defined because the direction lacked statistical support.
The team adapted a fragility-based framework — the reclassification fragility index (FI) — to test the statistical robustness of observed NENA directions. The FI measures the minimum proportion of case reclassifications required to remove statistical support for NENA's direction. In the three illustrative 1,000-case scenarios the reclassification FI ranged from 0.10% to 1.70%. When applied to the published comparisons, six of seven pairwise comparisons yielded an FI between 0.34% and 4.31%; one comparison did not have an FI because its NENA confidence interval included zero. The authors presented the FI as a complementary gauge of how stable a directional result is to small changes in case classifications.
Data indicated a consistent decline in PNA as radiologist experience rose. Junior residents showed a larger positive NENA (+6.27%), suggesting the AI corrected more radiologist errors in that subgroup. Consultants showed a smaller positive NENA (+1.45%). The authors noted that this pattern was driven primarily by falling PNA with increasing experience, while NNA increased only modestly. The source text did not provide subgroup sample sizes, case-mix details, or additional stratified diagnostics beyond the percentages quoted in the abstract.
The article states that all data produced are available on reasonable request to the authors. The paper is posted on medRxiv as a preprint and has not undergone peer review; medRxiv explicitly cautions that findings from the preprint should not be used to guide clinical practice. The authors declared no competing interests and confirmed adherence to ethical guidelines.
Limitations and open questions directly documented in the source include the preprint status and the lack of peer review; the study did not report additional methodological specifics or clinical consequences beyond the directional metrics, their bootstrap intervals, and the reclassification fragility index. The source did not document regulatory milestones, prospective clinical validation plans, or detailed image- or patient-level results within the text provided.