The study team built ClinDataBench to measure how well large language model (LLM) agents can produce complete clinical analysis packages. Agents received 22 SDTM domains, a statistical analysis plan (SAP) and dataset specifications and were required to deliver 13 ADaM datasets, Define-XML metadata and an offline program that can rebuild the outputs. The authors reported that prior work on LLMs and CDISC data had not evaluated a full, conformant and reproducible package; ClinDataBench aims to fill that evaluative gap.
Agents worked from publicly available, de-identified trial inputs provided by the CDISC SDTM/ADaM Pilot Project. The benchmark therefore tests an end-to-end translation pipeline: raw SDTM-like inputs plus written specifications into machine-readable ADaM deliverables, rather than isolated code snippets or single-file conversions.
The architecture restricted agents by running each inside a container whose only network route was the model API. That design limited external lookups during execution; agents could contact published reference outputs only through their model. The evaluation pipeline ran on the host and scored multiple, distinct outcomes.
The host-only evaluator reported four separate axes: conformance under the Pinnacle 21 Community (P21) validator, a set of selected derivation correctness checks, whether the package could be cleanly reconstructed from the bundled program (semantic rebuild), and human effort required. The P21 validator focuses on structural and rule-based checks. The authors noted that validator outputs test structure more than whether each derived value matches the specification’s intended semantics.
Three closed workflows framed the experiments. The Base workflow submitted each agent’s first generated package without further revision. Self-Refine allowed the agent to iteratively revise its package without receiving external validator feedback. CRITIC provided the agent with its own validator report and permitted revision informed by those findings. The paper described these settings as a way to probe whether self-guided or validator-informed iteration improves structural conformance, derivation correctness, or rebuildability.
The exploratory pilot included six models. Three closed-weight models and a model named DeepSeek delivered all 13 required ADaM datasets in every workflow setting tested. Those top agents produced twelve packages that, in eleven cases, rebuilt semantically when the authors ran the included program offline. Two open-weight models failed to deliver any complete package in the benchmark runs.
The authors observed variability in how models approached dataset derivations. Some agents produced outputs that satisfied many structural rules yet missed specification-defined summary records; others avoided structural violations but failed particular derivations. The study team emphasized that producing the files alone does not equal correct analytic derivations.
Validator findings per package varied substantially, with the reported range from 15 to 64 dataset-rule pairs. In the CRITIC workflows, four of six produced usable validator comparisons and each of those reduced its validator finding count after revision. The study team also reported that none of the workflows decreased the number of failed derivations, a separate measure tracking whether required derivation logic produced the expected values.
A concrete mismatch illustrated the problem: the package with the fewest P21 findings still did not include the 234 efficacy summary records in the form the specifications required. That single observation shows that fewer structural findings do not necessarily indicate more correct derivations. The authors therefore separated validator counts from semantic rebuild and derivation checks in their scoring.
The de-identified source data come from the CDISC SDTM/ADaM Pilot Project and are publicly available at the project repository the authors cite. The ClinDataBench benchmark harness, evaluator and aggregate experimental results are available in the ClinDataBench repository the authors provide. The authors stated that full run artifacts are not publicly distributed.
Readers should note this work is a medRxiv preprint and has not been certified by peer review. The authors declared no competing interests and confirmed ethical oversight because only de-identified public data were used.
Limitations and immediate unanswered questions recorded by the authors include the disconnect between validator finding counts and derivation correctness, the variable performance across open- and closed-weight models, and the absence of fully public run artifacts for independent reanalysis. The report does not provide regulatory milestones or peer-review timelines; it does not claim readiness for clinical deployment and recommends further validation steps documented in the paper.