Determining sample size for progression criteria for pragmatic pilot RCTs: the hypothesis test strikes back!

Abstract

Background: The current CONSORT guidelines for reporting pilot trials do not recommend hypothesis testing of clinical outcomes on the basis that a pilot trial is under-powered to detect such differences and this is the aim of the main trial. It states that primary evaluation should focus on descriptive analysis of feasibility/process outcomes (e.g. recruitment, adherence, treatment fidelity). Whilst the argument for not testing clinical outcomes is justifiable, the same does not necessarily apply to feasibility/process outcomes, where differences may be large and detectable with small samples. Moreover, there remains much ambiguity around sample size for pilot trials.

Methods: Many pilot trials adopt a 'traffic light' system for evaluating progression to the main trial determined by a set of criteria set up a priori. We construct a hypothesis testing approach for binary feasibility outcomes focused around this system that tests against being in the RED zone (unacceptable outcome) based on an expectation of being in the GREEN zone (acceptable outcome) and choose the sample size to give high power to reject being in the RED zone if the GREEN zone holds true. Pilot point estimates falling in the RED zone will be statistically non-significant and in the GREEN zone will be significant; the AMBER zone designates potentially acceptable outcome and statistical tests may be significant or non-significant.

Results: For example, in relation to treatment fidelity, if we assume the upper boundary of the RED zone is 50% and the lower boundary of the GREEN zone is 75% (designating unacceptable and acceptable treatment fidelity, respectively), the sample size required for analysis given 90% power and one-sided 5% alpha would be around n = 34 (intervention group alone). Observed treatment fidelity in the range of 0-17 participants (0-50%) will fall into the RED zone and be statistically non-significant, 18-25 (51-74%) fall into AMBER and may or may not be significant and 26-34 (75-100%) fall into GREEN and will be significant indicating acceptable fidelity.

Discussion: In general, several key process outcomes are assessed for progression to a main trial; a composite approach would require appraising the rules of progression across all these outcomes. This methodology provides a formal framework for hypothesis testing and sample size indication around process outcome evaluation for pilot RCTs.

Keywords: Outcome and process assessment; Pilots; Sample size, Statistics.

Conflict of interest statement

The authors declare that they have no competing interests.

Figures

Fig. 1

Illustration of power using the 1-tailed hypothesis testing against the traffic light signalling approach to pilot progression. E, observed point estimate; RUL, upper limit of RED zone; GLL, lower limit of GREEN zone; Ac, cut-off for statistical significance (at the 1-sided 5% level); α, type I error; β, type II error

Fig. 2

Probability of traffic light given true underlying probability of an event using the example from Table 5 (i). Two plots are presented: a relating to normal approximation approach and b relating to binomial exact approach. Based on n = 200, RUL = 40 and GLL = 70