Investigators conducted a silent deployment of PharmaAgent, an LLM-based agentic system, at Stanford Hospital for two months. The study team compared two operational configurations: a high-autonomy agentic workflow and a low-autonomy workflow; each configuration was evaluated both with and without local institutional guideline constraints. Deployment was observational and silent — the agents generated dosing recommendations that were not used to change clinical care. The authors reported that the treating pharmacist's actual orders served as the pragmatic reference standard for comparing agent outputs. The institutional review board at Stanford approved the study, and the authors stated that data are available on reasonable request.
Across 403 dosing events in 316 patients, investigators calculated mean absolute percentage error (MAPE) relative to the treating pharmacist's order. The high-autonomy configuration produced a MAPE of 21.8%, while the low-autonomy configuration produced a MAPE of 37.8%. Short, clear metrics framed the comparison: the higher-autonomy agent's numeric outputs were closer to pharmacist practice. The study team also measured the proportion of recommendations falling within 25% of the treating pharmacist's order. Recommendations were within that tolerance in 73.4% of events for the high-autonomy agent versus 50.9% for the low-autonomy agent (P < 0.001). For clinical context, a second pharmacist reviewer matched the treating pharmacist within 25% in 90.2% of events, providing a reference for human-to-human variability in dosing decisions.
The investigators flagged 196 agent recommendations as out-of-tolerance for pharmacist review. Pharmacists adjudicated these cases and found a safety concern in 65.8% of the reviewed recommendations. The authors reported that safety concerns were more frequent when the low-autonomy configuration generated the recommendation; the reported odds ratio for safety concerns with low autonomy versus high autonomy was 2.37. These figures illustrate that numerical proximity to a pharmacist's order does not fully capture safety risk; clinical adjudication revealed substantive safety signals in a majority of out-of-tolerance outputs.
Authors described differences in how the two agentic systems operated. The higher-autonomy agent included a self-debugging mechanism for intermediate reasoning steps, and the study team observed that this agent was more robust in practice than the version that relied on stricter engineering constraints. Short statement: self-debugging appeared beneficial. Longer explanation: the authors suggested that an agent allowed to iteratively inspect and correct its intermediate computations yielded outputs that better aligned with pharmacist decisions and produced fewer flagged safety concerns than the stricter, low-autonomy approach. The report did not provide additional technical details of the self-debugging algorithm or the exact engineering constraints applied; those specifics were not reported in the source document.
The report states that the IRB of Stanford University provided ethical approval for the work. The authors declared no competing interests. The study is a preprint and has not undergone peer review; the authors cautioned that findings should not be used to guide clinical practice. All data produced in the study are available upon reasonable request to the authors, according to the manuscript.
The authors were explicit about limitations: this was a silent, short-term deployment at a single center and the treating pharmacist's order was used as the reference standard rather than clinical outcomes. The study did not report clinical endpoints such as therapeutic drug monitoring results linked to patient outcomes, nor did it provide full technical details of model training, dosing algorithms, or the self-debugging methods. The authors concluded that, although the high-autonomy agent produced recommendations closer to pharmacist orders and was more robust in this dataset, neither the high- nor low-autonomy configuration is ready to dose patients without pharmacist oversight. That statement frames the immediate operational implication: clinical pharmacotherapy decisions require maintained human review until further validation and safety demonstration are completed.