Small studies, big decisions: the role of pilot/feasibility studies in incremental science and premature scale-up of behavioral interventions

Michael W Beets, Lauren von Klinggraeff, R Glenn Weaver, Bridget Armstrong, Sarah Burkart, Michael W Beets, Lauren von Klinggraeff, R Glenn Weaver, Bridget Armstrong, Sarah Burkart

Abstract

Background: Careful consideration and planning are required to establish "sufficient" evidence to ensure an investment in a larger, more well-powered behavioral intervention trial is worthwhile. In the behavioral sciences, this process typically occurs where smaller-scale studies inform larger-scale trials. Believing that one can do the same things and expect the same outcomes in a larger-scale trial that were done in a smaller-scale preliminary study (i.e., pilot/feasibility) is wishful thinking, yet common practice. Starting small makes sense, but small studies come with big decisions that can influence the usefulness of the evidence designed to inform decisions about moving forward with a larger-scale trial. The purpose of this commentary is to discuss what may constitute sufficient evidence for moving forward to a definitive trial. The discussion focuses on challenges often encountered when conducting pilot/feasibility studies, referred to as common (mis)steps, that can lead to inflated estimates of both feasibility and efficacy, and how the intentional design and execution of one or more, often small, pilot/feasibility studies can play a central role in developing an intervention that scales beyond a highly localized context.

Main body: Establishing sufficient evidence to support larger-scale, definitive trials, from smaller studies, is complicated. For any given behavioral intervention, the type and amount of evidence necessary to be deemed sufficient is inherently variable and can range anywhere from qualitative interviews of individuals representative of the target population to a small-scale randomized trial that mimics the anticipated larger-scale trial. Major challenges and common (mis)steps in the execution of pilot/feasibility studies discussed are those focused on selecting the right sample size, issues with scaling, adaptations and their influence on the preliminary feasibility and efficacy estimates observed, as well as the growing pains of progressing from small to large samples. Finally, funding and resource constraints for conducting informative pilot/feasibility study(ies) are discussed.

Conclusion: Sufficient evidence to scale will always remain in the eye of the beholder. An understanding of how to design informative small pilot/feasibility studies can assist in speeding up incremental science (where everything needs to be piloted) while slowing down premature scale-up (where any evidence is sufficient for scaling).

Keywords: Early-stage; Feasibility; Intervention; Pilot; Scaling; Translation.

Conflict of interest statement

The authors declare that they have no competing interests.

© 2021. The Author(s).

Figures

Fig. 1
Fig. 1
Theoretical voltage-drop in intervention impact from scaling with and without common (mis)steps

References

    1. Green LW, Glasgow RE. Evaluating the relevance, generalization, and applicability of research: issues in external validation and translation methodology. Eval Health Prof. 2006;29(1):126–153. doi: 10.1177/0163278705284445.
    1. Glasgow RE, Emmons KM. How can we increase translation of research into practice? Types of evidence needed. Annu Rev Public Health. 2007;28(1):413–433. doi: 10.1146/annurev.publhealth.28.021406.144145.
    1. Glasgow RE, Klesges LM, Dzewaltowski DA, Bull SS, Estabrooks P. The future of health behavior change research: what is needed to improve translation of research into health promotion practice? Ann Behav Med. 2004;27(1):3–12. doi: 10.1207/s15324796abm2701_2.
    1. Pyle DF. Nutrition interventions: problems associated with expanding pilot/demonstration projects into national-level programs. Prog Clin Biol Res. 1981;77:575–584.
    1. Milat AJ, King L, Bauman A, Redman S. Scaling up health promotion interventions: an emerging concept in implementation science. Health Promot J Austr. 2011;22(3):238. doi: 10.1071/HE11238.
    1. Milat AJ, King L, Bauman AE, Redman S. The concept of scalability: increasing the scale and potential adoption of health promotion interventions into policy and practice. Health Promot Int. 2013;28(3):285–298. doi: 10.1093/heapro/dar097.
    1. Milat AJ, Newson R, King L, Rissel C, Wolfenden L, Bauman A, Redman S, Giffin M. A guide to scaling up population health interventions. Public Health Res Pract. 2016;26(1):e2611604. doi: 10.17061/phrp2611604.
    1. Beets MW, Weaver RG, Ioannidis JPA, Geraci M, Brazendale K, Decker L, Okely AD, Lubans D, van Sluijs E, Jago R, Turner-McGrievy G, Thrasher J, Li X, Milat AJ. Identification and evaluation of risk of generalizability biases in pilot versus efficacy/effectiveness trials: a systematic review and meta-analysis. Int J Behav Nutr Phys Act. 2020;17(1):19. doi: 10.1186/s12966-020-0918-y.
    1. Indig D, Lee K, Grunseit A, Milat A, Bauman A. Pathways for scaling up public health interventions. BMC Public Health. 2017;18(1):68. doi: 10.1186/s12889-017-4572-5.
    1. Pearson N, Naylor PJ, Ashe MC, Fernandez M, Yoong SL, Wolfenden L. Guidance for conducting feasibility and pilot studies for implementation trials. Pilot Feasibility Stud. 2020;6(1):167. doi: 10.1186/s40814-020-00634-w.
    1. Stevens J, Taber DR, Murray DM, Ward DS. Advances and controversies in the design of obesity prevention trials. Obesity. 2007;15(9):2163–2170. doi: 10.1038/oby.2007.257.
    1. Thabane L, Ma J, Chu R, Cheng J, Ismaila A, Rios LP, et al. A tutorial on pilot studies: the what, why and how. Bmc Med Res Methodol. 2010;10(1). 10.1186/1471-2288-10-1.
    1. Czajkowski SM, Powell LH, Adler N, Naar-King S, Reynolds KD, Hunter CM, Laraia B, Olster DH, Perna FM, Peterson JC, Epel E, Boyington JE, Charlson ME. From ideas to efficacy: the ORBIT model for developing behavioral treatments for chronic diseases. Health Psychol. 2015;34(10):971–982. doi: 10.1037/hea0000161.
    1. Onken LS, Carroll KM, Shoham V, Cuthbert BN, Riddle M. Reenvisioning clinical science: unifying the discipline to improve the public health. Clin Psychol Sci. 2014;2(1):22–34. doi: 10.1177/2167702613497932.
    1. Billingham SA, Whitehead AL, Julious SA. An audit of sample sizes for pilot and feasibility trials being undertaken in the United Kingdom registered in the United Kingdom Clinical Research Network database. Bmc Med Res Methodol. 2013;13(1):104. doi: 10.1186/1471-2288-13-104.
    1. Stice E, Shaw H, Marti CN. A meta-analytic review of obesity prevention programs for children and adolescents: the skinny on interventions that work. Psychol Bull. 2006;132(5):667–691. doi: 10.1037/0033-2909.132.5.667.
    1. Bacchetti P, Deeks SG, McCune JM. Breaking free of sample size dogma to perform innovative translational research. Sci Transl Med. 2011;3(87):87ps24. doi: 10.1126/scitranslmed.3001628.
    1. Hedt BL, Pagano M. Health indicators: eliminating bias from convenience sampling estimators. Stat Med. 2011;30(5):560–568. doi: 10.1002/sim.3920.
    1. Nielsen M, Haun D, Kartner J, et al. The persistent sampling bias in developmental psychology: a call to action. J Exp Child Psychol. 2017;162:31–38. doi: 10.1016/j.jecp.2017.04.017.
    1. Cardona-Morrell M, Rychetnik L, Morrell SL, Espinel PT, Bauman A. Reduction of diabetes risk in routine clinical practice: are physical activity and nutrition interventions feasible and are the outcomes from reference trials replicable? A systematic review and meta-analysis. BMC Public Health. 2010;10(1):653. doi: 10.1186/1471-2458-10-653.
    1. Cajita MI, Kline CE, Burke LE, Bigini EG, Imes CC. Feasible but not yet efficacious: a scoping review of wearable activity monitors in interventions targeting physical activity, sedentary behavior, and sleep. Curr Epidemiol Rep. 2020;7(1):25–38. doi: 10.1007/s40471-020-00229-2.
    1. Kwasnicka D, Ten Hoor GA, Hekler E, et al. Proposing a new approach to funding behavioural interventions using iterative methods. Psychol Health. 2021;36(7):1–5. doi: 10.1080/08870446.2021.1945061.
    1. National Institute of Diabetes and Digestive and Kidney Diseases. Small R01s for clinical trials targeting diseases within the mission of NIDDK (R01 clinical trial required). Department of Health and Human Services; [cited 2021 01/13]; Available from: .
    1. National Institute of Diabetes and Digestive and Kidney Diseases. Notice of early expiration of NIDDK R21 funding opportunity announcements. [cited 2021 01/13]; Available from: .
    1. Chambers DA, Glasgow RE, Stange KC. The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change. Implement Sci. 2013;8(1):117. doi: 10.1186/1748-5908-8-117.
    1. Axford N, Berry V, Lloyd J, et al. Promoting learning from null or negative results in prevention science trials. Prev Sci. 2020. .

Source: PubMed

Szponzorok és közreműködők

Egészségi állapot

Kábítószer-beavatkozások

3
Iratkozz fel