Powering Bias and Clinically Important Treatment Effects in Randomized Trials of Critical Illness

Darryl Abrams, Sydney B Montesi, Sarah K L Moore, Daniel K Manson, Kaitlin M Klipper, Meredith A Case, Daniel Brodie, Jeremy R Beitler, Darryl Abrams, Sydney B Montesi, Sarah K L Moore, Daniel K Manson, Kaitlin M Klipper, Meredith A Case, Daniel Brodie, Jeremy R Beitler

Abstract

Objectives: Recurring issues in clinical trial design may bias results toward the null, yielding findings inconclusive for treatment effects. This study evaluated for powering bias among high-impact critical care trials and the associated risk of masking clinically important treatment effects.

Design, setting, and patients: Secondary analysis of multicenter randomized trials of critically ill adults in which mortality was the main endpoint. Trials were eligible for inclusion if published between 2008 and 2018 in leading journals. Analyses evaluated for accuracy of estimated control group mortality, adaptive sample size strategy, plausibility of predicted treatment effect, and results relative to the minimal clinically important difference. The main outcome was the mortality risk difference at the study-specific follow-up interval.

Interventions: None.

Measurements and main results: Of 101 included trials, 12 met statistical significance for their main endpoint, five for increased intervention-associated mortality. Most trials (77.3%) overestimated control group mortality in power calculations (observed minus predicted difference, -6.7% ± 9.8%; p < 0.01). Due to this misestimation of control group mortality, in 14 trials, the intervention would have had to prevent at least half of all deaths to achieve the hypothesized treatment effect. Seven trials prespecified adaptive sample size strategies that might have mitigated this issue. The observed risk difference for mortality fell within 5% of predicted in 20 trials, of which 16 did not reach statistical significance. Half of trials (47.0%) were powered for an absolute risk reduction greater than or equal to 10%, but this effect size was observed in only three trials with a statistically significant treatment benefit. Most trials (67.3%) could not exclude clinically important treatment benefit or harm.

Conclusions: The design of most high-impact critical care trials biased results toward the null by overestimating control group mortality and powering for unrealistic treatment effects. Clinically important treatment effects often cannot be excluded.

Figures

Figure 1.. Waterfall plot of control group…
Figure 1.. Waterfall plot of control group mortality misestimation.
Absolute difference in control group mortality, observed minus expected. Each bar represents an individual trial.
Figure 2.. Butterfly plot of predicted and…
Figure 2.. Butterfly plot of predicted and observed mortality risk difference.
Data are sorted by sample size within each study type. One trial (sample size 1,439 patients) contained insufficient information in its reported power and sample size determination to identify predicted risk difference.
Figure 3.. Impact of control group mortality…
Figure 3.. Impact of control group mortality misestimation, hypothesized risk ratio, and sample size on statistical power.
For a given sample size and risk ratio (also known as relative risk), power decreases with lower control group mortality. Overestimation of control group mortality in sample size calculations decreases power (1 – ß) and therefore increases probability of a “false negative” trial result (ß). The impact of lower control group mortality on diminishing power is exacerbated by smaller sample size.
Figure 4.. Trial results according to clinically…
Figure 4.. Trial results according to clinically important difference in mortality on absolute and relative scales.
Several trials that did not achieve statistical significance in conventional frequentist analysis nevertheless failed to exclude clinically important benefit or harm for the intervention studied. The prespecified threshold used for MCID was a 5% absolute risk difference (number needed to treat = 20) or 20% relative risk difference (risk ratio ≤ 0.8 or ≥ 1.2) for either benefit or harm with treatment. Thresholds are indicated by the shaded areas: blue for benefit and red for harm. Forest plots indicate the effect estimate and 95% confidence interval for absolute risk difference (left) and risk ratio (right) for each trial. The size of each square corresponds to the trial sample size relative to other included trials. The probability of a clinically important treatment effect (benefit or harm), given the trial results, was calculated for both the absolute risk difference and risk ratio using Bayesian statistics. MCID = minimal clinically important difference. * denotes statistical significance according to the trial’s main analysis.

References

    1. Significance of significant. N Engl J Med 1968; 278:1232–1233.
    1. Pocock SJ, Stone GW: The primary outcome fails - what next? N Engl J Med 2016; 375:861–870.
    1. Greenland S: Nonsignificance plus high power does not imply support for the null over the alternative. Ann Epidemiol 2012; 22:364–368.
    1. Calfee CS, Delucchi K, Parsons PE et al.: Subphenotypes in acute respiratory distress syndrome: latent class analysis of data from two randomised controlled trials. The Lancet Respiratory medicine 2014; 2:611–620.
    1. Seymour CW, Kennedy JN, Wang S et al.: Derivation, validation, and potential treatment implications of novel clinical phenotypes for sepsis. JAMA 2019; 321:2003–2017.
    1. Adrie C, Adib-Conquy M, Laurent I et al.: Successful cardiopulmonary resuscitation after cardiac arrest as a “sepsis-like” syndrome. Circulation 2002; 106:562–568.
    1. Reynolds HR, Hochman JS: Cardiogenic shock: current concepts and improving outcomes. Circulation 2008; 117:686–697.
    1. Garland A, Shaman Z, Baron J et al.: Physician-attributable differences in intensive care unit costs: a single-center study. Am J Respir Crit Care Med 2006; 174:1206–1210.
    1. Bellani G, Laffey JG, Pham T et al.: Epidemiology, patterns of care, and mortality for patients with acute respiratory distress syndrome in intensive care units in 50 countries. JAMA 2016; 315:788–800.
    1. Stevenson EK, Rubenstein AR, Radin GT et al.: Two decades of mortality trends among patients with severe sepsis: a comparative meta-analysis. Crit Care Med 2014; 42:625–631.
    1. du Bois RM, Weycker D, Albera C et al.: Six-minute-walk test in idiopathic pulmonary fibrosis: test validation and minimal clinically important difference. Am J Respir Crit Care Med 2011; 183:1231–1237.
    1. Harhay MO, Wagner J, Ratcliffe SJ et al.: Outcomes and statistical power in adult critical care randomized trials. Am J Respir Crit Care Med 2014; 189:1469–1478.
    1. Aberegg SK, Richards DR, O’Brien JM: Delta inflation: a bias in the design of randomized controlled trials in critical care medicine. Crit Care 2010; 14:R77.
    1. Kim WB, Worley B, Holmes J et al.: Minimal clinically important differences for measures of treatment efficacy in Stevens-Johnson syndrome and toxic epidermal necrolysis. J Am Acad Dermatol 2018; 79:1150–1152.
    1. Target Investigators, ANZICS Clinical Trials Group, Chapman M et al.: Energy-dense versus routine enteral nutrition in the critically ill. N Engl J Med 2018; 379:1823–1834.
    1. TARGET Investigators, Australian and New Zealand Intensive Care Society Clinical Trials Group: Statistical analysis plan for the augmented versus routine approach to giving energy trial (TARGET). Crit Care Resusc 2018; 20:15–21.
    1. Ridgeon EE, Bellomo R, Aberegg SK et al.: Effect sizes in ongoing randomized controlled critical care trials. Crit Care 2017; 21:132.
    1. Kaul S, Diamond GA: Trial and error. How to avoid commonly encountered limitations of published clinical trials. J Am Coll Cardiol 2010; 55:415–427.
    1. Brunkhorst FM, Engel C, Bloos F et al.: Intensive insulin therapy and pentastarch resuscitation in severe sepsis. N Engl J Med 2008; 358:125–139.
    1. Caironi P, Tognoni G, Masson S et al.: Albumin replacement in patients with severe sepsis or septic shock. N Engl J Med 2014; 370:1412–1421.
    1. Writing Group for the Alveolar Recruitment for Acute Respiratory Distress Syndrome Trial Investigators, Cavalcanti AB, Suzumura EA et al.: Effect of lung recruitment and titrated positive end-expiratory pressure (PEEP) vs low PEEP on mortality in patients with acute respiratory distress syndrome: a randomized clinical trial. JAMA 2017; 318:1335–1345.
    1. De Backer D, Biston P, Devriendt J et al.: Comparison of dopamine and norepinephrine in the treatment of shock. N Engl J Med 2010; 362:779–789.
    1. Holcomb JB, Tilley BC, Baraniuk S et al.: Transfusion of plasma, platelets, and red blood cells in a 1:1:1 vs a 1:1:2 ratio and mortality in patients with severe trauma: the PROPPR randomized clinical trial. JAMA 2015; 313:471–482.
    1. Ranieri VM, Thompson BT, Barie PS et al.: Drotrecogin alfa (activated) in adults with septic shock. N Engl J Med 2012; 366:2055–2064.
    1. Guerin C, Reignier J, Richard JC et al.: Prone positioning in severe acute respiratory distress syndrome. N Engl J Med 2013; 368:2159–2168.
    1. Combes A, Hajage D, Capellier G et al.: Extracorporeal membrane oxygenation for severe acute respiratory distress syndrome. N Engl J Med 2018; 378:1965–1975.
    1. Guntupalli K, Dean N, Morris PE et al.: A phase 2 randomized, double-blind, placebo-controlled study of the safety and efficacy of talactoferrin in patients with severe sepsis. Crit Care Med 2013; 41:706–716.
    1. Zhang Q, Li C, Shao F et al.: Efficacy and safety of combination therapy of shenfu injection and postresuscitation bundle in patients with return of spontaneous circulation after in-hospital cardiac arrest: a randomized, assessor-blinded, controlled trial. Crit Care Med 2017; 45:1587–1595.
    1. Karnad DR, Bhadade R, Verma PK et al.: Intravenous administration of ulinastatin (human urinary trypsin inhibitor) in severe sepsis: a multicenter randomized controlled study. Intensive Care Med 2014; 40:830–838.
    1. van den Berghe G, Wouters P, Weekers F et al.: Intensive insulin therapy in critically ill patients. N Engl J Med 2001; 345:1359–1367.
    1. NICE SUGAR Study Investigators, Finfer S, Chittock DR et al.: Intensive versus conventional glucose control in critically ill patients. N Engl J Med 2009; 360:1283–1297.
    1. Rivers E, Nguyen B, Havstad S et al.: Early goal-directed therapy in the treatment of severe sepsis and septic shock. N Engl J Med 2001; 345:1368–1377.
    1. Investigators ProCESS, Yealy DM Kellum JA et al.: A randomized trial of protocol-based care for early septic shock. N Engl J Med 2014; 370:1683–1693.
    1. Anthon CT, Granholm A, Perner A et al.: Overall bias and sample sizes were unchanged in ICU trials over time: a meta-epidemiological study. J Clin Epidemiol 2019; 113:189–199.
    1. Cochrane Handbook for Systematic Reviews of Interventions, version 5.1.0; .
    1. Hochman JS: Cardiogenic shock complicating acute myocardial infarction: expanding the paradigm. Circulation 2003; 107:2998–3002.
    1. Grissom CK, Hirshberg EL, Dickerson JB et al.: Fluid management with a simplified conservative protocol for the acute respiratory distress syndrome. Crit Care Med 2015; 43:288–295.
    1. Chalmers I, Matthews R: What are the implications of optimism bias in clinical research? Lancet 2006; 367:449–450.
    1. Mann H, Djulbegovic B: Choosing a control intervention for a randomised clinical trial. BMC Med Res Methodol 2003; 3:7.
    1. Chan AW, Hrobjartsson A, Haahr MT et al.: Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA 2004; 291:2457–2465.
    1. Bassler D, Briel M, Montori VM et al.: Stopping randomized trials early for benefit and estimation of treatment effects: systematic review and meta-regression analysis. JAMA 2010; 303:1180–1187.
    1. Ioannidis JP: Contradicted and initially stronger effects in highly cited clinical research. JAMA 2005; 294:218–228.
    1. de Grooth HJ, Parienti JJ, Postema J et al.: Positive outcomes, mortality rates, and publication bias in septic shock trials. Intensive Care Med 2018; 44:1584–1585.
    1. Spragg RG, Bernard GR, Checkley W et al.: Beyond mortality: future clinical research in acute lung injury. Am J Respir Crit Care Med 2010; 181:1121–1127.
    1. Ospina-Tascon GA, Buchele GL, Vincent JL: Multicenter, randomized, controlled trials evaluating mortality in intensive care: doomed to fail? Crit Care Med 2008; 36:1311–1322.
    1. Bekaert M, Timsit JF, Vansteelandt S et al.: Attributable mortality of ventilator-associated pneumonia: a reappraisal using causal analysis. Am J Respir Crit Care Med 2011; 184:1133–1139.
    1. Shankar-Hari M, Harrison DA, Rowan KM et al.: Estimating attributable fraction of mortality from sepsis to inform clinical trials. J Crit Care 2018; 45:33–39.
    1. Girbes ARJ, de Grooth HJ: Time to stop randomized and large pragmatic trials for intensive care medicine syndromes: the case of sepsis and acute respiratory distress syndrome. Journal of thoracic disease 2020; 12:S101–S109.
    1. Elsasser A, Regnstrom J, Vetter T et al.: Adaptive clinical trial designs for European marketing authorization: a survey of scientific advice letters from the European Medicines Agency. Trials 2014; 15:383.
    1. Bhatt DL, Mehta C: Adaptive designs for clinical trials. N Engl J Med 2016; 375:65–74.
    1. van der Graaf R, Roes KC, van Delden JJ: Adaptive trials in clinical research: scientific and ethical issues to consider. JAMA 2012; 307:2379–2380.
    1. Sterne JA, Davey Smith G: Sifting the evidence-what’s wrong with significance tests? BMJ 2001; 322:226–231.
    1. Gardner MJ, Altman DG: Confidence intervals rather than P values: estimation rather than hypothesis testing. Br Med J (Clin Res Ed) 1986; 292:746–750.
    1. Sackett DL: The principles behond the tactics of performing therapeutic trials In: Clinical Epiidemiology: How to Do Clinical Practice Research. Edited by Haynes RB, Sackett DL, Guyatt GH, Tugwell P. Philadelphia, PA: Lippincott Williams & Wilkins; 2006: 173–243.
    1. Diamond GA, Kaul S: Prior convictions: Bayesian approaches to the analysis and interpretation of clinical megatrials. J Am Coll Cardiol 2004; 43:1929–1939.
    1. Lewis RJ, Angus DC: Time for clinicians to embrace their inner Bayesian?: Reanalysis of results of a clinical trial of extracorporeal membrane oxygenation. JAMA 2018; 320:2208–2210.
    1. Ellis LM, Bernstein DS, Voest EE et al.: American Society of Clinical Oncology perspective: raising the bar for clinical trials by defining clinically meaningful outcomes. J Clin Oncol 2014; 32:1277–1280.
    1. Moher D, Dulberg CS, Wells GA: Statistical power, sample size, and their reporting in randomized controlled trials. JAMA 1994; 272:122–124.

Source: PubMed

Sponsorer og samarbejdspartnere

Medicinske tilstande

Narkotikainterventioner

3
Abonner