---
title: "Auditable Reference Standard for extracting clinical data from transesophageal echocardiography re"
id: "medrxiv-19-building-an-auditable-reference-standard-for-clinical-information-extraction"
canonical_url: "https://medichelpline.com/clinical-feed/medrxiv-19-building-an-auditable-reference-standard-for-clinical-information-extraction"
content_type: "clinical_feed_article"
specialty: "Critical Care"
source_name: "medRxiv (Clinical Preprints)"
source_url: "https://www.medrxiv.org/content/10.64898/2026.11.19.26368500v1?rss=1"
published_at: "2026-11-20T12:00:00.000Z"
evidence_level: "Verified Feed"
license: "CC-BY-NC-4.0 / Informational Use"
---
# Auditable Reference Standard for extracting clinical data from transesophageal echocardiography re
## Provenance & Clinical Metadata
- **Canonical URL:** https://medichelpline.com/clinical-feed/medrxiv-19-building-an-auditable-reference-standard-for-clinical-information-extraction
- **Specialty:** [Critical Care](https://medichelpline.com/clinical-feed/critical-care.md)
- **Primary Source:** medRxiv (Clinical Preprints)
- **Source URL:** [Original Journal Publication](https://www.medrxiv.org/content/10.64898/2026.11.19.26368500v1?rss=1)
- **Published At:** 2026-11-20T12:00:00.000Z
- **Evidence Rating:** Verified Feed
## Executive GIST (TL;DR)
- Investigators built a two-institution reference standard for intraoperative **transesophageal echocardiography (TEE)** reports using 300 reports (150 per site) sampled from 925 available reports. - Four trained abstractors applied a shared 39-variable V1 codebook across selected reports; independent paired comparisons were performed on 297 reports while three historical reports contributed only to the final reference standard. - Categorical agreement was measured by exact agreement and unweighted Cohen's κ; numeric **left-ventricular ejection fraction (LVEF)** was assessed by exact agreement and mean absolute differences, as reported by the authors. - Of 11,583 reviewer-comparison cells, 10,293 (88.9%) were resolved by reviewer agreement; 1,290 (11.1%) required targeted expert adjudication (1,289 adjudication tasks performed). - The adjudication workload concentrated on aortic dissection, valve type, and post-intervention findings, according to the study team. - Adjudication reasons were classified; codebook/schema ambiguity accounted for 40.8% of targeted tasks and interpretation/classification differences for 25.2%, together comprising 66.0% of targeted adjudications. - Categorical κ values ranged from 0.364 to 0.960 across variables; the authors reported an inverse correlation between κ and targeted-adjudication burden (Spearman ρ=−0.884; 95% CI, −0.955 to −0.734). - Sampling-weighted κ changed by no more than 0.05 for 32 of 35 categorical variables, per the investigators' additional analyses that included reviewer-pair, sampling-stratified, and population-weighted agreement. - Authors concluded structured adjudication revealed recurring schema and interpretation challenges beyond individual abstraction errors and recommended rigorous development, adjudication, and independent validation for reference standards used to evaluate **NLP** and **LLM**-based abstraction tools. - Data are not publicly available because of institutional data-use agreements and patient privacy protections; associated analysis code is in a public repository but release is pending institutional review.
## Clinical Analysis & Structured Key Points
Building an Auditable Reference Standard for Clinical Information Extraction from Transesophageal Echocardiography Reports | medRxiv Skip to main content Building an Auditable Reference Standard for Clinical Information Extraction from Transesophageal Echocardiography Reports View ORCID Profile Emily J MacKay , Michael S Kaplan , Mingyang Yuan , Annie L Chen , Yujia Ge , Layla Canales , Emily Yu , Rohan Shad , Bo Zhang , Alison M Pouch doi: https://doi.org/10.64898/2026.11.19.26368500 Emily J MacKay 1 Department of Anesthesiology and Critical Care, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, PA; Find this author on Google Scholar Find this author on PubMed Search for this author on this site ORCID record for Emily J MacKay For correspondence: emily.mackay{at}pennmedicine.upenn.edu Michael S Kaplan 2 Department of Anesthesiology and Pain Medicine, University of Washington, Seattle, WA; Find this author on Google Scholar Find this author on PubMed Search for this author on this site Mingyang Yuan 3 Lewis Katz School of Medicine, Temple University, Philadelphia, PA; Find this author on Google Scholar Find this author on PubMed Search for this author on this site Annie L Chen 4 Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA; Find this author on Google Scholar Find this author on PubMed Search for this author on this site Yujia Ge 5 Tufts University School of Medicine, Boston MA; Find this author on Google Scholar Find this author on PubMed Search for this author on this site Layla Canales 6 Villanova University, Villanova, PA; Find this author on Google Scholar Find this author on PubMed Search for this author on this site Emily Yu 7 State University of New York Downstate Health Sciences University, New York City, NY; Find this author on Google Scholar Find this author on PubMed Search for this author on this site Rohan Shad 8 Division of Cardiovascular Surgery, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, PA; Find this author on Google Scholar Find this author on PubMed Search for this author on this site Bo Zhang 9 Vaccine and Infectious Disease Division, Fred Hutchinson Cancer Center, Seattle, WA; Find this author on Google Scholar Find this author on PubMed Search for this author on this site Alison M Pouch 10 Department of Radiology & Department of Bioengineering, University of Pennsylvania, Philadelphia, PA Find this author on Google Scholar Find this author on PubMed Search for this author on this site Abstract Info/History Metrics Supplementary material Data/Code Preview PDF Abstract Objectives To develop an auditable, two-institution reference standard for intraoperative transesophageal echocardiography (TEE) reports and evaluate whether reviewer disagreement identifies limitations in the underlying clinical information model. Materials and Methods We selected 300 reports (150 per institution) from 925 at the University of Pennsylvania and University of Washington using random sampling and enrichment for challenging documentation. Four trained abstractors applied a shared 39-variable V1 codebook. Independent paired comparisons included 297 reports; three historical reports contributed only to the final reference standard. Categorical agreement used exact agreement and unweighted Cohen's κ; numeric left-ventricular ejection fraction (LVEF) used exact agreement and mean absolute differences. Disagreements underwent structured expert adjudication and reason classification. Additional analyses assessed reviewer-pair, sampling-stratified, and population-weighted agreement. Results Among 11,583 formal reviewer-comparison cells, 10,293 (88.9%) were resolved by reviewer agreement and 1,290 (11.1%) required targeted expert adjudication (1,289 tasks). Codebook/schema ambiguity (40.8%) and interpretation/classification differences (25.2%) together accounted for 66.0% of targeted tasks. Adjudication was concentrated in aortic dissection, valve type, and post-intervention findings. Categorical κ ranged from 0.364 to 0.960 and was inversely correlated with targeted-adjudication burden (Spearman ρ=−0.884; 95% CI, −0.955 to −0.734). Sampling-weighted κ changed by no more than 0.05 for 32 of 35 categorical variables. Discussion Structured adjudication identified recurring schema and interpretation challenges beyond individual abstraction errors, providing specific targets for information-model refinement. Conclusion As NLP and LLM-based abstraction tools move toward clinical use, the reference standards used to evaluate them require comparably rigorous development, adjudication, and independent validation. Competing Interest Statement The authors have declared no competing interest. Author Declarations I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained. Yes The details of the IRB/oversight body that provided approval or exemption for the research described are given below: The study was determined to be exempt from institutional review board (IRB) review by both the University of Pennsylvania (UPenn) and the University of Washington (UW), and was conducted under a fully executed data use agreement between both institutions. I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals. Yes I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance). Yes I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable. Yes Data Availability The clinical reports and patient-level adjudication data used in this study are not publicly available due to institutional data-use agreements and patient privacy protections. The study was conducted using data from the University of Pennsylvania and the University of Washington. A public repository for associated analysis code has been established at https://github.com/TEECompass/tee-reference-standard-v1; code release is pending institutional review. https://github.com/TEECompass/tee-reference-standard-v1 Funder Information Declared National Institutes of Health , K23HL166964 Copyright The copyright holder for this preprint is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. It is made available under a CC-BY 4.0 International license . Back to top Previous Next Posted November 20, 2026. Download PDF Supplementary Material Data/Code Email Thank you for your interest in spreading the word about medRxiv. NOTE: Your email address is requested solely to identify you as the sender of this article. Your Email * Your Name * Send To * Enter multiple addresses on separate lines or separate them with commas. You are going to email the following Building an Auditable Reference Standard for Clinical Information Extraction from Transesophageal Echocardiography Reports Message Subject (Your Name) has forwarded a page to you from medRxiv Message Body (Your Name) thought you would like to see this page from the medRxiv website. Your Personal Message CAPTCHA This question is for testing whether or not you are a human visitor and to prevent automated spam submissions. Share Building an Auditable Reference Standard for Clinical Information Extraction from Transesophageal Echocardiography Reports Emily J MacKay , Michael S Kaplan , Mingyang Yuan , Annie L Chen , Yujia Ge , Layla Canales , Emily Yu , Rohan Shad , Bo Zhang , Alison M Pouch medRxiv 2026.11.19.26368500; doi: https://doi.org/10.64898/2026.11.19.26368500 Share This Article: Copy Citation Tools Building an Auditable Reference Standard for Clinical Information Extraction from Transesophageal Echocardiography Reports Emily J MacKay , Michael S Kaplan , Mingyang Yuan , Annie L Chen , Yujia Ge , Layla Canales , Emily Yu , Rohan Shad , Bo Zhang , Alison M Pouch medRxiv 2026.11.19.26368500; doi: https://doi.org/10.64898/2026.11.19.26368500 Citation Manager Formats BibTeX Bookends EasyBib EndNote (tagged) EndNote 8 (xml) Medlars Mendeley Papers RefWorks Tagged Ref Manager RIS Zotero Tweet Widget Facebook Like Google Plus One Subject Areas All Articles Addiction Medicine (642) Allergy and Immunology (928) Anesthesia (355) Cardiovascular Medicine (5059) Dentistry and Oral Medicine (496) Dermatology (432) Emergency Medicine (681) Endocrinology (including Diabetes Mellitus and Metabolic Disease) (1703) Epidemiology (16352) Forensic Medicine (37) Gastroenterology (1249) Genetic and Genomic Medicine (7333) Geriatric Medicine (761) Health Economics (1101) Health Informatics (5350) Health Policy (1480) Health Systems and Quality Improvement (1847) Hematology (604) HIV/AIDS (1389) Infectious Diseases (except HIV/AIDS) (16537) Intensive Care and Critical Care Medicine (1210) Medical Education (702) Medical Ethics (157) Nephrology (757) Neurology (7664) Nursing (380) Nutrition (1144) Obstetrics and Gynecology (1308) Occupational and Environmental Health (1046) Oncology (3769) Ophthalmology (1105) Orthopedics (414) Otolaryngology (474) Pain Medicine (508) Palliative Medicine (144) Pathology (727) Pediatrics (1877) Pharmacology and Therapeutics (759) Primary Care Research (793) Psychiatry and Clinical Psychology (6186) Public and Global Health (10051) Radiology and Imaging (2562) Rehabilitation Medicine and Physical Therapy (1540) Respiratory Medicine (1280) Rheumatology (666) Sexual and Reproductive Health (814) Sports Medicine (601) Surgery (809) Toxicology (109) Transplantation (321) Urology (309)
## Related Clinical Research

- [Triglyceride-Glucose Index Linked to Airflow Limitation Severity in Older Nondiabetic, Normoxemic](https://medichelpline.com/clinical-feed/pubmed-43067697.md) (DOI: 10.12659/MSM.954698)
- [Blockchain-Enforced Federated Ensemble for Privacy-Preserving In-Hospital Mortality Prediction](https://medichelpline.com/clinical-feed/medrxiv-18-a-blockchain-enabled-federated-ensemble-framework-for-privacy-preserving.md)
- [Machine learning model predicts ICU delirium in adults with acute kidney injury](https://medichelpline.com/clinical-feed/pubmed-43063111.md) (DOI: 10.1097/MD.0000000000051672)
- [High blood pressure in Ugandan children and adolescents with Type 1 diabetes: multicenter prevalen](https://medichelpline.com/clinical-feed/plos-one-11-high-blood-pressure-among-children-and-adolescents-with-type-1-diabetes.md)
- [Leveraging MAHA and Kennedy to Combat Measles Vaccine Hesitancy](https://medichelpline.com/clinical-feed/stat-news-0-opinion-how-kennedy-and-maha-could-help-stop-measles.md)

## Navigation
- [← Back to Critical Care Feed](https://medichelpline.com/clinical-feed/critical-care.md)
- [← All Clinical Specialties](https://medichelpline.com/clinical-feed.md)
## Medical & Regulatory Disclaimer

> [!CAUTION]
> MedicHelpline content is structured for research, educational, and professional discovery purposes. It does not constitute individual medical advice, clinical diagnosis, or treatment recommendations.
> Always verify dosing, contraindications, and regulatory alerts against official product labeling and primary regulatory sources before clinical decision-making.