The statistical interpretation of pilot trials: should significance thresholds be reconsidered?

Ellen C Lee, Amy L Whitehead, Richard M Jacques, Steven A Julious, Ellen C Lee, Amy L Whitehead, Richard M Jacques, Steven A Julious

Abstract

Background: In an evaluation of a new health technology, a pilot trial may be undertaken prior to a trial that makes a definitive assessment of benefit. The objective of pilot studies is to provide sufficient evidence that a larger definitive trial can be undertaken and, at times, to provide a preliminary assessment of benefit.

Methods: We describe significance thresholds, confidence intervals and surrogate markers in the context of pilot studies and how Bayesian methods can be used in pilot trials. We use a worked example to illustrate the issues raised.

Results: We show how significance levels other than the traditional 5% should be considered to provide preliminary evidence for efficacy and how estimation and confidence intervals should be the focus to provide an estimated range of possible treatment effects. We also illustrate how Bayesian methods could also assist in the early assessment of a health technology.

Conclusions: We recommend that in pilot trials the focus should be on descriptive statistics and estimation, using confidence intervals, rather than formal hypothesis testing and that confidence intervals other than 95% confidence intervals, such as 85% or 75%, be used for the estimation. The confidence interval should then be interpreted with regards to the minimum clinically important difference. We also recommend that Bayesian methods be used to assist in the interpretation of pilot trials. Surrogate endpoints can also be used in pilot trials but they must reliably predict the overall effect on the clinical outcome.

Figures

Figure 1
Figure 1
Mean difference in SF-36 GH dimension scores between treatment and control with confidence intervals (based on n = 31 patients).
Figure 2
Figure 2
Prior, observed and posterior distributions for non-informative, pessimistic and optimistic priors.

References

    1. Wood J, Lambert M. Sample size calculations for trials in health services research. J Health Serv Res Policy. 1999;4(4):226–229.
    1. Julious SA, Patterson SD. Sample sizes for estimation in clinical research. Pharm Stat. 2004;3(3):213–215. doi: 10.1002/pst.125.
    1. Biomarkers Definitions Working Group. Biomarkers and surrogate endpoints: preferred definitions and conceptual framework. Clin Pharmacol Ther. 2001;69(3):89–95.
    1. Lancaster GA, Dodd S, Williamson PR. Design and analysis of pilot studies: recommendations for good practice. J Eval Clin Pract. 2004;10(2):307–312. doi: 10.1111/j..2002.384.doc.x.
    1. Thabane L, Ma J, Chu R, Cheng J, Ismaila A, Rios LP, Robson R, Thabane M, Giangregorio L, Goldsmith CH. A tutorial on pilot studies: the what, why and how. BMC Med Res Methodol. 2010;10:1. doi: 10.1186/1471-2288-10-1.
    1. Arain M, Campbell MJ, Cooper CL, Lancaster GA. What is a pilot or feasibility study? A review of current practice and editorial policy. BMC Med Res Methodol. 2010;10:67. doi: 10.1186/1471-2288-10-67.
    1. Kianifard F, Islam MZ. A guide to the design and analysis of small clinical studies. Pharm Stat. 2011;10(4):363–368. doi: 10.1002/pst.477.
    1. Stallard N. Optimal sample sizes for phase II clinical trials and pilot studies. Stat Med. 2012;31:1031–1042. doi: 10.1002/sim.4357.
    1. Schoenfeld D. Statistical considerations for pilot-studies. Int J Radiat Oncol Biol Phys. 1980;6(3):371–374. doi: 10.1016/0360-3016(80)90153-4.
    1. Papadakis S, Aitken D, Gocan S, Riley D, Laplante MA, Bhatnagar-Bost A, Cousineau D, Simpson D, Edjoc R, Pipe AL, Sharma M, Reid RD. A randomised controlled pilot study of standardised counselling and cost-free pharmacotherapy for smoking cessation among stroke and TIA patients. BMJ Open. 2011;1(2):e000366.
    1. Legault C, Jennings JM, Katula JA, Dagenbach D, Gaussoin SA, Sink KM, Rapp SR, Rejeski WJ, Shumaker SA, Espeland MA. Designing clinical trials for assessing the effects of cognitive training and physical activity interventions on cognitive outcomes: the Seniors Health and Activity Research Program Pilot (SHARP-P) study, a randomized controlled trial. BMC Geriatr. 2011;11:27. doi: 10.1186/1471-2318-11-27.
    1. Walters SJ. Consultants’ forum: should post hoc sample size calculations be done? Pharm Stat. 2009;8(2):163–169. doi: 10.1002/pst.334.
    1. Walters SJ, Morrell CJ, Dixon S. Measuring health-related quality of life in patients with venous leg ulcers. Qual Life Res. 1999;8(4):327–336. doi: 10.1023/A:1008992006845.
    1. Morrell CJ, Walters SJ, Dixon S, Collins KA, Brereton LML, Peters J, Brooker CGD. Cost effectiveness of community leg ulcer clinics: randomised controlled trial. Br Med J. 1998;316(7143):1487–1491. doi: 10.1136/bmj.316.7143.1487.
    1. Collins K, Morrell J, Peters J, Walters S, Brooker C, Brereton L. Problems associated with patient satisfaction surveys. Bri J Commun Health Nurs. 2007;2(3):156–163.
    1. Carpenter JR, Kenward MG. Multiple Imputation and its Application. Chichester: Wiley; 2013.
    1. De Gruttola VG, Clax P, DeMets DL, Downing GJ, Ellenberg SS, Friedman L, Gail MH, Prentice R, Wittes J, Zeger SL. Considerations in the evaluation of surrogate endpoints in clinical trials: Summary of a National Institutes of Health Workshop. Control Clin Trials. 2001;22(5):485–502. doi: 10.1016/S0197-2456(01)00153-2.
    1. Prentice RL. Surrogate endpoints in clinical-trials - definition and operational criteria. Stat Med. 1989;8(4):431–440. doi: 10.1002/sim.4780080407.
    1. International Conference on Harmonisation. ICH E9 statistical principals for clinical trials. 1998. .
    1. Fleming TR, DeMets DL. Surrogate end points in clinical trials: are we being misled? Ann Intern Med. 1996;125(7):605–613. doi: 10.7326/0003-4819-125-7-199610010-00011.
    1. Temple R. Are surrogate markers adequate to assess cardiovascular disease drugs? J Am Med Assoc. 1999;282(8):790–795. doi: 10.1001/jama.282.8.790.
    1. Julious SA, Machin D, Tan SB. An Introduction to Statistics in Early Phase Trials. Oxford: Wiley-Blackwell; 2010.
    1. Julious SA, Swank DJ. Moving statistics beyond the individual clinical trial: applying decision science to optimize a clinical development plan. Pharm Stat. 2005;4(1):37–46. doi: 10.1002/pst.149.
    1. Chuang-Stein C, Kirby S, French J, Kowalski K, Marshall S, Smith MK, Bycott P, Beltangady M. A quantitative approach for making go/no-go decisions in drug development. Drug Inform J. 2011;45(2):187–202.
    1. O’Hagan A, Stevens JW, Campbell MJ. Assurance in clinical trial design. Pharm Stat. 2005;4(3):187–201. doi: 10.1002/pst.175.
    1. Chuang-Stein C. Sample size and the probability of a successful trial. Pharm Stat. 2006;5(4):305–309. doi: 10.1002/pst.232.
    1. Parmar MKB, Ungerleider RS, Simon R. Assessing whether to perform a confirmatory randomized clinical trial. J Natl Canc Inst. 1996;88(22):1645–1651. doi: 10.1093/jnci/88.22.1645.
    1. Spiegelhalter DJ, Abrams KR, Myles JP. Bayesian Approaches to Clinical Trials and Health-Care Evaluation. Chichester: John Wiley & Sons; 2004.
    1. Lee PM. Bayesian Statistics: An Introduction. New York: Oxford University Press; Edward Arnold; 1989.

Source: PubMed

Szponzorok és közreműködők

Egészségi állapot

Kábítószer-beavatkozások

3
Iratkozz fel