Investigators sampled 300 intraoperative transesophageal echocardiography (TEE) reports from a pool of 925 at two institutions, allocating 150 reports per site. Four trained abstractors applied a shared 39-variable V1 codebook to the corpus. The study team performed independent paired comparisons on 297 reports; three historical reports were included only in the final reference standard and did not undergo pairwise comparison. The authors reported random sampling with enrichment for challenging documentation, but the source did not provide additional sampling weights or the precise enrichment criteria beyond that description.
Abstractors resolved most items by reviewer agreement. The dataset produced 11,583 formal reviewer-comparison cells. Investigators documented that 10,293 cells (88.9%) were resolved without external adjudication. The remaining 1,290 cells (11.1%) required targeted expert adjudication; the manuscript records 1,289 adjudication tasks were conducted. This workflow established a layered, auditable process: initial abstraction, paired comparison, and targeted expert adjudication where disagreement persisted.
The study team used exact agreement and unweighted Cohen's κ to quantify categorical variable concordance. For numeric left-ventricular ejection fraction (LVEF), they used exact agreement and mean absolute differences to assess numeric concordance. The source reported a broad spread of categorical κ values across the 39 variables, with the lowest at 0.364 and the highest at 0.960. Those κ estimates reflect variable-specific heterogeneity in interrater reliability, as investigators reported.
Investigators also examined the relationship between agreement metrics and adjudication burden. They calculated a Spearman correlation between categorical κ and targeted-adjudication burden and reported a strong inverse relationship (Spearman ρ = −0.884; 95% CI, −0.955 to −0.734), indicating variables with lower κ consumed more adjudication resources.
Adjudication was not evenly distributed across topics. The study team observed concentrated adjudication activity in items related to aortic dissection, valve type, and post-intervention findings. When adjudication reasons were classified, the authors noted that codebook or schema ambiguity accounted for 40.8% of targeted adjudication tasks. Interpretation or classification differences accounted for 25.2% of tasks. Together, these two categories comprised 66.0% of targeted adjudication workload, indicating systemic schema and interpretation issues rather than isolated abstraction errors.
Other categories of adjudication reasons appeared less dominant, but the source did not enumerate every minor adjudication cause or provide per-variable counts beyond the high-level percentages and the named domains with concentrated adjudication.
Categorical agreement (κ) varied by variable, spanning from 0.364 to 0.960. The authors reported that sampling-weighted κ changed by no more than 0.05 for 32 of 35 categorical variables when they applied sampling-weight adjustments. That finding suggests relative stability of agreement estimates under the sampling frame used, according to the investigators. The study team also performed reviewer-pair and sampling-stratified analyses; however, the source did not provide full tabular detail for every reviewer pairing in the body text available.
Structured adjudication surfaced recurring schema issues beyond individual abstraction mistakes. Investigators reported these adjudication outcomes as specific targets for information-model refinement — for example, clarifying variable definitions where codebook ambiguity drove a large fraction of adjudication work.
Additional analyses included reviewer-pair agreement assessments and population-weighted agreement calculations. The authors stated these sensitivity checks produced limited change in overall agreement metrics: sampling-weighted κ moved by at most 0.05 for most categorical items (32 of 35). The manuscript did not provide granular numeric results for every sensitivity test within the source text, but the reported summary indicates robustness of primary agreement estimates to the sampling approach and reviewer composition.
The authors declared the clinical reports and patient-level adjudication data are not publicly available because of institutional data-use agreements and patient privacy protections. A public repository for analysis code exists at https://github.com/TEECompass/tee-reference-standard-v1, but the source noted the code release is pending institutional review. Both institutions determined the study to be exempt from institutional review board review, and the work was conducted under a fully executed data-use agreement between the University of Pennsylvania and the University of Washington.
Limitations documented by the authors include the restricted public availability of underlying reports and adjudication-level data, which constrains independent reanalysis. The manuscript emphasizes the need for independent validation of the reference standard and specifies that future work should refine the information model to address codebook ambiguity and interpretation differences identified during structured adjudication. The authors conclude that reference standards intended to evaluate NLP and LLM-based abstraction tools require comparable rigor in development, adjudication, and independent validation, and they stop at those next steps rather than proposing specific regulatory milestones.