2026 80th Anniversary
International Convention of PSK

D+12
October 21-23, 2026

Abstracts

P12-10

Beyond protocol generation: evaluating the causal validity of large language models for target trial emulation

  • Chaeyeon Lee1, Sojeong Lee1, Jiyoung Yang1, Woo Hyeon Yoo1, Gyeongmin Lim1, Dongwon Yoon2, Eun-Young Choi3, Bin Hong4,5, In-Sun Oh*1, Ju-Young Shin*6
  • 1Department of Biohealth Regulatory Science, Sungkyunkwan University
  • 2College of Pharmacy, Ewha Womans University
  • 3Centre for Pharmacoepidemiology, Karolinska Institutet, Sweden
  • 4School of Pharmacy, Sungkyunkwan University
  • 5Gerald Bronfman Department of Oncology, McGill University, Canada
  • 6College of Pharmacy, Seoul National University

Target trial emulation (TTE) strengthens causal inference in observational research by specifying the protocol of a hypothetical randomised trial. Large language models (LLMs) may assist protocol development, but whether they preserve the causal principles required for valid emulation remains uncertain. We conducted a blinded comparative evaluation of 18 TTE protocols generated by ChatGPT GPT-5.2, Gemini 3.1 Pro, and Claude Opus 4.6 across three clinical scenarios and two prompting conditions: methodology-only prompting and prompting supplemented with database-variable specifications. Three blinded reviewers assessed eight TTE-specific components, three methodological quality criteria, and six prespecified critical methodological errors. A conservative criterion (≥1 reviewer) served as the primary analysis, with the majority criterion (≥2 reviewers) examined in sensitivity analysis. Claude achieved the highest median total score (46.33), followed by ChatGPT (39.00) and Gemini (36.83; p=0.0041). Under the conservative criterion, 50.0% of protocols contained at least one critical methodological error. LLMs can generate structurally coherent TTE protocols, but their ability to operationalize key causal inference principles remains inconsistent. Aggregate completeness scores alone are insufficient to judge methodological acceptability. Explicit critical-error checks and expert oversight remain necessary when LLMs are used to support TTE protocol development.


Q&A

작성하기
  • There are no registered questions

Sponsored by