The investigators assembled three distinct datasets to test how human-derived reference standards behave under different group conditions. They described one cohort of 50 transbronchial lung cryobiopsy cases from patients with interstitial lung disease, reviewed by 27 pathologists; a second cohort of 95 resected lung adenocarcinomas, reviewed by 11 pathologists; and a third dataset of 596 images of pulmonary granulomatous lesions, reviewed by 12 pathologists. The authors reported that larger panels produced more stable, reproducible collective judgments. Simple statement: as panel size rose, consensus classifications converged. Yet the study team also made clear that stability in classification is not identical to clinical usefulness. Agreement metrics improved with larger groups, but that reproducibility was a property distinct from outcome association.
Panel composition mattered as well. The investigators did not rely solely on overall panel size; they examined how groups of pathologists identified by diagnostic behavior could form separate clusters. Those clusters generated consensus labels that sometimes differed from the all-rater consensus. The source did not report details of the clustering algorithm or exact cluster thresholds in this summary; the manuscript contains the analytic specifics. Data availability is restricted to requests to the corresponding author under ethical guidelines.
The study team evaluated whether stronger consensus translated to stronger links with patient outcomes. They observed that, in the cryobiopsy and lung adenocarcinoma cohorts, increasing panel size and the resulting convergence toward a single consensus did not increase, and in some cases weakened, associations with disease progression or survival. In other words, more unanimous labels were not necessarily more predictive of clinically meaningful endpoints. The authors reported that collective judgments produced by larger groups became more reproducible but did not improve associations with progression or survival in those two cohorts.
Investigators therefore separated two concepts: reproducibility, measured as interrater agreement and consensus stability, and clinical relevance, measured as the strength of association between a reference label and downstream outcomes. Data indicated these properties can diverge. The manuscript does not provide additional trial-like endpoints, effect sizes, or p-values in the abstract; those statistical details are available in the full preprint.
The study team examined the informational content of disagreement rather than treating it as mere noise. In the lung adenocarcinoma cohort, they compared cases graded differently by the all-rater consensus and a distinct pathologist cluster. That comparison exposed specific morphological patterns that the authors propose could contribute to the differing labels. The source text states these patterns may carry clinically informative signals, but it does not list the exact morphologic features in the abstract; readers should consult the full paper for the granular examples and image-level annotations.
In the granuloma image cohort, investigators applied a modified Delphi process to the same set of cases. The Delphi sequence increased agreement: Krippendorff’s α rose from 0.351 to 0.539. Agreement improved numerically, and the panel achieved a higher reproducibility metric after iterative adjudication. Despite that increase, the authors observed only modest changes in disease–morphology associations following consensus formation. This result reinforced the central claim that raising agreement does not automatically heighten clinical relevance.
The team argued for a different framing of expert disagreement. Instead of treating discordant annotations purely as error to be removed, they recommended viewing disagreement as a potential source of clinically meaningful information. They proposed a pathway: identify morphological features that associate independently with outcomes, validate those features externally, and then use them to refine consensus frameworks for specific clinical tasks. The abstract states this approach could support development of more clinically relevant, task-specific ground truth for pathology AI. The authors did not claim that any single consensus method or panel size is universally optimal; they described exploratory comparisons and outcome-linked assessments across the three cohorts.
Limitations, next steps, and open questions
The abstract notes ethical approval and restricted data availability, but it does not report granular statistical values or the full methodology in the summary. The study did not claim definitive optimal panel sizes or provide a single procedural prescription for constructing ground truth. Unanswered clinical questions documented by the authors include which morphological features, when independently validated, will improve outcome prediction and how to operationalize task-specific reference standards for AI. The manuscript therefore leaves concrete next steps: external validation of outcome-associated morphologies, specification of reproducible cluster-identification methods, and determination of when consensus refinement genuinely alters clinical performance rather than merely increasing agreement.