Should treatment effects be estimated in pilot and feasibility studies?

Abstract

Background: Feasibility studies and external pilot studies are used increasingly to inform planning decisions related to a definitive randomized controlled trial. These studies can provide information on process measures, such as consent rates, treatment fidelity and compliance, and methods of outcome measurement. Additionally, they can provide initial parameter estimates for a sample size calculation, such as a standard deviation or the 'success' rate for a binary outcome in the control group. However, the issue of estimating treatment effects in pilot or feasibility studies is controversial.

Methodological discussion: Between-group estimates of treatment effect from pilot studies are sometimes used to calculate the sample size for a main trial, alongside estimated standard deviations. However, whilst estimating a standard deviation is an empirical matter, a targeted treatment effect should be established in terms of clinical judgement, as a minimum important difference (MID), not through analysis of pilot data. Secondly, between-group effects measured in pilot studies are sometimes used to indicate the magnitude of an effect that might be obtained in a main trial, and a decision on progression made with reference to the associated confidence interval. Such estimates will be imprecise in typically small pilot studies and therefore do not allow a robust decision on a main trial; both a decision to proceed and a decision not to proceed may be made too readily. Thirdly, a within-group change might be estimated from a pilot or a feasibility study in a desire to assess the potential efficacy of a novel intervention prior to testing it in a main trial, but again such estimates are liable to be imprecise and do not allow sound causal inferences.

Conclusion: Treatment effects calculated from pilot or feasibility studies should not be the basis of a sample size calculation for a main trial, as the MID to be detected should be based primarily on clinical judgement rather than statistics. Deciding on progression to a main trial based on these treatment effects is also misguided, as they will normally be imprecise, and may be biased if the pilot or feasibility study is unrepresentative of the main trial.

Keywords: Feasibility studies; Pilot studies; Randomized controlled trials; Treatment effects.

Conflict of interest statement

Competing interestsThe author declares that he has no competing interests

Figures

Fig. 1

Estimated treatment effects, with 95% confidence intervals, reported for four outcomes by Sheehan et al. [25]. A difference in percentage improvement of 40% (indicated by the dashed horizontal reference line) was taken to be the minimum important difference

Fig. 2

Simulated data for 20 pilot studies (each n = 34) estimating an unknown ‘true’ treatment effect of 10, with a standard deviation of 20, and with an assumed minimum important difference of 4 (dashed horizontal reference line). Error bars are 95% confidence intervals, and the observed underlying standard deviations range from 12.6 to 24.8

Fig. 3

Confidence intervals for the estimated treatment effect in study 18 (n = 34) in the simulated example, at varying confidence levels from 95 to 70%. The dashed horizontal reference line indicates the minimum important difference