pipette
ESEspañol

Automating postmarketing pharmacovigilance with large language model agents

R. Zhang, Y. Zhou, C. S. L. Chui, D. Yu, Z. Wu, R. Luo

PreprintBold claims, read criticallyReal-world use

In the authors' words

Postmarketing pharmacovigilance relies on spontaneous adverse event reporting and labor-intensive observational studies, which suffer from underreporting and delayed risk detection. Large language model (LLM) agents could automate real-world evidence generation, but generalist agents make ad hoc analytical decisions that compromise epidemiological validity and reproducibility. Here we present MedCheckpoint, a multi-agent LLM system for automated postmarketing pharmacovigilance. MedCheckpoint grounds each analytical stage in a curated, source-traceable medical knowledge base, locks core study-design parameters before execution, performs patient-level analyses with human-written deterministic statistical tools, and enforces checkpoint rules that abort infeasible analyses, with optional human-in-the-loop review. We evaluated MedCheckpoint using a 54-question benchmark on a dual-purpose simulated electronic health record (EHR) testbed of 2.7 million individuals, designed with reliable clinical signals to evaluate routine analytical accuracy alongside spurious signals to stress-test system rigor. Across the benchmark, MedCheckpoint outperformed two coding-agent baselines in report quality across five dimensions rated by three independent LLM judges (overall means 4.81-4.88 versus 4.25-4.55 and 4.02-4.29) and yielded higher run-to-run consistency (mean within-question standard deviation 0.116-0.186 versus up to 0.440). Crucially, while baselines failed on up to 2 of 21 runs when handling infeasible comparisons or unmeasurable outcomes, MedCheckpoint produced no misleading failures across all seven stress-test trap questions with known data limitations. MedCheckpoint enables reliable, auditable, and scalable pharmacovigilance analysis.

Main resultThe abstract does not state a limitation.

Appeared: Friday, November 13. medRxiv. Preprint, not yet peer-reviewed.

DOI: 10.64898/2026.11.10.26367711