Addressing identification bias in the design and analysis of cluster-randomized pragmatic trials: a case study

Abstract

Background: Pragmatic trials provide the opportunity to study the effectiveness of health interventions to improve care in real-world settings. However, use of open-cohort designs with patients becoming eligible after randomization and reliance on electronic health records (EHRs) to identify participants may lead to a form of selection bias referred to as identification bias. This bias can occur when individuals identified as a result of the treatment group assignment are included in analyses.

Methods: To demonstrate the importance of identification bias and how it can be addressed, we consider a motivating case study, the PRimary care Opioid Use Disorders treatment (PROUD) Trial. PROUD is an ongoing pragmatic, cluster-randomized implementation trial in six health systems to evaluate a program for increasing medication treatment of opioid use disorders (OUDs). A main study objective is to evaluate whether the PROUD intervention decreases acute care utilization among patients with OUD (effectiveness aim). Identification bias is a particular concern, because OUD is underdiagnosed in the EHR at baseline, and because the intervention is expected to increase OUD diagnosis among current patients and attract new patients with OUD to the intervention site. We propose a framework for addressing this source of bias in the statistical design and analysis.

Results: The statistical design sought to balance the competing goals of fully capturing intervention effects and mitigating identification bias, while maximizing power. For the primary analysis of the effectiveness aim, identification bias was avoided by defining the study sample using pre-randomization data (pre-trial modeling demonstrated that the optimal approach was to use individuals with a prior OUD diagnosis). To expand generalizability of study findings, secondary analyses were planned that also included patients newly diagnosed post-randomization, with analytic methods to account for identification bias.

Conclusion: As more studies seek to leverage existing data sources, such as EHRs, to make clinical trials more affordable and generalizable and to apply novel open-cohort study designs, the potential for identification bias is likely to become increasingly common. This case study highlights how this bias can be addressed in the statistical study design and analysis.

Trial registration: ClinicalTrials.gov, NCT03407638. Registered on 23 January 2018.

Keywords: Cluster-randomized trials; Electronic health records; Identification bias; Implementation trial; Open-cohort trial; Opioid use disorder; Recruitment bias.

Conflict of interest statement

The authors declare that they have no competing interests.

Figures

Fig. 1

Analytic samples available for inclusion in analyses of PROUD intervention effects before and after randomization. Boxes not drawn to scale. * Increase in documentation of an OUD diagnosis may be due to increased skill in diagnosing and treating OUD or increased patient disclosure due to reduced stigma. ** Includes patients who are attracted to the intervention site because they are seeking OUD treatment (e.g. due to limited access to treatment elsewhere or lower barriers to receiving care in the PROUD intervention site)

Fig. 2

Comparison of statistical power across different options for the analytic sample for the effectiveness analysis. The x-axis shows the intervention effect size, parameterized as the percentage decrease in the expected number of days of acute care utilization comparing patients with OUD (recognized or unrecognized) in the intervention versus usual care arm. All options for the analytic sample (described in Table 2) use pre-randomization data. Each panel represents a different true prevalence of OUD (1%, 2%, or 4%). Options 3a, 3b, and 3c correspond to different assumptions of the properties of an algorithm for defining “increased risk” of OUD (see Table 3). Higher sensitivity includes more patients with true OUD whereas higher specificity excludes more patients without true OUD. Power calculations were based on closed form sample size formula based on Poisson regression (details are in the Additional File)