2026 CONVENTION
Beyond protocol generation: evaluating the causal validity of large language models for target trial emulation
Target trial emulation (TTE) strengthens causal inference in observational research by specifying the protocol of a hypothetical randomised trial. Large language models (LLMs) may assist protocol development, but whether they preserve the causal principles required for valid emulation remains uncertain. We conducted a blinded comparative evaluation of 18 TTE protocols generated by ChatGPT GPT-5.2, Gemini 3.1 Pro, and Claude Opus 4.6 across three clinical scenarios and two prompting conditions: methodology-only prompting and prompting supplemented with database-variable specifications. Three blinded reviewers assessed eight TTE-specific components, three methodological quality criteria, and six prespecified critical methodological errors. A conservative criterion (≥1 reviewer) served as the primary analysis, with the majority criterion (≥2 reviewers) examined in sensitivity analysis. Claude achieved the highest median total score (46.33), followed by ChatGPT (39.00) and Gemini (36.83; p=0.0041). Under the conservative criterion, 50.0% of protocols contained at least one critical methodological error. LLMs can generate structurally coherent TTE protocols, but their ability to operationalize key causal inference principles remains inconsistent. Aggregate completeness scores alone are insufficient to judge methodological acceptability. Explicit critical-error checks and expert oversight remain necessary when LLMs are used to support TTE protocol development.