Emulating Randomized Clinical Trials With Nonrandomized Real-World Evidence Studies: First Results From the RCT DUPLICATE Initiative

Abstract

Background: Regulators are evaluating the use of noninterventional real-world evidence (RWE) studies to assess the effectiveness of medical products. The RCT DUPLICATE initiative (Randomized, Controlled Trials Duplicated Using Prospective Longitudinal Insurance Claims: Applying Techniques of Epidemiology) uses a structured process to design RWE studies emulating randomized, controlled trials (RCTs) and compare results. We report findings of the first 10 trial emulations, evaluating cardiovascular outcomes of antidiabetic or antiplatelet medications.

Methods: We selected 3 active-controlled and 7 placebo-controlled RCTs for replication. Using patient-level claims data from US commercial and Medicare payers, we implemented inclusion and exclusion criteria, selected primary end points, and comparator populations to emulate those of each corresponding RCT. Within the trial-mimicking populations, we conducted propensity score matching to control for >120 preexposure confounders. All study measures were prospectively defined and protocols registered before hazard ratios and 95% CIs were computed. Success criteria for the primary analysis were prespecified for each replication.

Results: Despite attempts to emulate RCT design as closely as possible, differences between the RCT and corresponding RWE study populations remained. The regulatory conclusions were equivalent in 6 of 10. The RWE emulations achieved a hazard ratio estimate that was within the 95% CI from the corresponding RCT in 8 of 10 studies. In 9 of 10, either the regulatory or estimate agreement success criteria were fulfilled. The largest differences in effect estimates were found for RCTs where second-generation sulfonylureas were used as a proxy for placebo regarding cardiovascular effects. Nine of 10 replications had a standardized difference between effect estimates of <2, which suggests differences within expected random variation.

Conclusions: Agreement between RCT and RWE findings varies depending on which agreement metric is used. Interim findings indicate that selection of active comparator therapies with similar indications and use patterns enhances the validity of RWE. Even in the context of active comparators, concordance between RCT and RWE findings is not guaranteed, partially because trials are not emulated exactly. More trial emulations are needed to understand how often and in what contexts RWE findings match RCTs. Registration: URL: https://www.clinicaltrials.gov; Unique identifiers: NCT03936049, NCT04215523, NCT04215536, NCT03936010, NCT03936036, NCT03936062, NCT03936023, NCT03648424, NCT04237935, NCT04237922.

Keywords: bias; diabetes mellitus; dipeptidyl-peptidase IV inhibitors; randomized controlled trial; sodium-glucose transporter 2 inhibitors.

Conflict of interest statement

Conflict of interest disclosures:

Dr. Schneeweiss is principal investigator of the FDA Sentinel Innovation Center funded by the FDA, co-principal investigator of an investigator-initiated grant to the Brigham and Women’s Hospital from Boehringer Ingelheim unrelated to the topic of this study. He is a consultant to Aetion Inc., a software manufacturer of which he owns equity. His interests were declared, reviewed, and approved by the Brigham and Women’s Hospital and Partners HealthCare System in accordance with their institutional compliance policies. Dr. Patorno is co-investigator of an investigator-initiated grant to the Brigham and Women’s Hospital from Boehringer-Ingelheim, not directly related to the topic of the submitted work. Dr. Desai has served as principal investigator for research grants from Bayer, Vertex, and Novartis to the Brigham and Women's Hospital for unrelated projects. Dr. Glynn has received research support from investigator-initiated grants to the Brigham and Women’s Hospital for clinical trials funded by AstraZeneca, Kowa, Pfizer, and Novartis. Dr. Garry is an employee of Aetion, Inc., with stock options.

The views expressed in the article are the personal views of the authors and may not be understood, quoted or stated on behalf of or reflecting the views or policies of the Department of Health and Human Services or the U.S. Food and Drug Administration.

Figures

Figure 1.. Comparison of cumulative event curves.

Cumulative event Kaplan-Meier plots for primary endpoints in the RCTs and corresponding RWE emulations.

Figure 2.. Agreement between RCT findings and their pre-specified RWE emulations.

Open circles represent the estimated HR from RWE, and filled circles represent the estimated HR from the corresponding RCT. Under the null hypothesis of no bias in the RWE, we would expect approximately 5% of emulations to have a standardized difference > 2. RA = regulatory agreement reached; EA = estimate agreement reached; SD = standardized difference