TrialCalibre1 is an agentic system designed to connect clinical evidence, study design, and executable analyses within a single workflow. The authors described three core modes: reference-trial replication, direct target-trial emulation, and a combined benchmarking–extension–calibration workflow the team calls BenchExCal. The source reported that the system integrates ASCET-based benchmarking logic with executable code intended to automate large parts of trial emulation and validation. Two reviewers participated in assessing extraction, protocol decisions, and implementation; the study team reported both automated and human-in-the-loop checks as part of the evaluation.
Assessors reviewed a set of automated checks applied to the benchmark extraction and implementation. The authors reported SQL-level verification results: after adjudication, one reviewer rated 53 of 62 assessable SQL checks as met, while the other rated 50 of 62 as met. For R-based checks, both reviewers rated 12 of 13 assessable items as met. The report noted remaining disagreements around protocol items and proxy decisions; those disagreements persisted after discussion. The source did not provide further granular item-level descriptions beyond the aggregated counts of checks met and unresolved protocol/proxy disagreements.
The team evaluated a routing development set consisting of ten questions. The system produced 10 of 10 route matches in that set, and the authors reported 7 of 7 applicable reference trial matches. The paper presented these results as internal performance metrics for the routing and selection components of TrialCalibre1, indicating the routing logic yielded consistent match outcomes for that development set. The source did not report additional performance metrics such as sensitivity, specificity, or external validation on independent queries.
Two cohorts were reported in the case application comparing clopidogrel with aspirin among adults aged 81–90 years using synthetic PCORnet-style claims data. The agent benchmark cohort included 6,069 patients; the extension cohort included 1,602 patients. The authors emphasized that the human implementation provided descriptive comparisons rather than a same-data validation, and they made cohort counts and executable protocols available for review on request. The report did not supply details on dosing regimens, exposure definitions beyond the agent labels, or specific covariate lists when describing cohort construction.
For the extension analysis comparing clopidogrel versus aspirin, the reported point estimate was a risk ratio of 0.848 with a 95% confidence interval of 0.569–1.184. The calibration sensitivity analysis produced a calibrated sensitivity estimate of 1.274 (0.439–3.700). The source framed the calibrated estimate as a sensitivity probe to potential biases rather than as definitive correction. The authors noted that the human implementation was descriptive and not intended as a same-data validation; consequently, the reported effect estimates should be interpreted in the context of implementation choices and available synthetic data.
The report listed several concrete limitations that restricted interpretation. Dose proxies were used and reported as imperfect surrogates; some covariates remained unpopulated in the available data. Residual imbalance persisted after implemented adjustments, and the authors documented deviations between the agentic implementation and the human implementation. Cross-cohort covariance was not addressed in the analysis and was explicitly ignored, the source stated. The paper warned that these factors—dose proxies, missing covariate data, residual imbalance, implementation deviations, and ignored cross-cohort covariance—limit causal interpretation and preclude treating the human implementation as a validation on the same data.
The authors made protocols, code, and results available for review and stated that all data produced in the study are available upon reasonable request. The paper declared no competing interests. The source did not report trial registration identifiers, patient-level demographic breakdowns beyond the age band used, or specific imputation and modeling details for the benchmarks and extensions. Those methodological specifics were not present in the preprint and therefore are not reported here.
Limitations noted in the preprint remain the final concrete item in this report: available documentation did not include dosing definitions, full covariate lists, or granular implementation scripts within the manuscript text; readers seeking those materials must request the code and protocols from the authors per the data availability statement.