Engagement and retention: measuring breadth and depth of participant use of an online intervention

Mick P Couper, Gwen L Alexander, Nanhua Zhang, Roderick J A Little, Noel Maddy, Michael A Nowak, Jennifer B McClure, Josephine J Calvi, Sharon J Rolnick, Melanie A Stopponi, Christine Cole Johnson, Mick P Couper, Gwen L Alexander, Nanhua Zhang, Roderick J A Little, Noel Maddy, Michael A Nowak, Jennifer B McClure, Josephine J Calvi, Sharon J Rolnick, Melanie A Stopponi, Christine Cole Johnson

Abstract

Background: The Internet provides us with tools (user metrics or paradata) to evaluate how users interact with online interventions. Analysis of these paradata can lead to design improvements.

Objective: The objective was to explore the qualities of online participant engagement in an online intervention. We analyzed the paradata in a randomized controlled trial of alternative versions of an online intervention designed to promote consumption of fruit and vegetables.

Methods: Volunteers were randomized to 1 of 3 study arms involving several online sessions. We created 2 indirect measures of breadth and depth to measure different dimensions and dynamics of program engagement based on factor analysis of paradata measures of Web pages visited and time spent online with the intervention materials. Multiple regression was used to assess influence of engagement on retention and change in dietary intake.

Results: Baseline surveys were completed by 2513 enrolled participants. Of these, 86.3% (n = 2168) completed the follow-up surveys at 3 months, 79.6% (n = 2027) at 6 months, and 79.4% (n = 1995) at 12 months. The 2 tailored intervention arms exhibited significantly more engagement than the untailored arm (P < .01). Breadth and depth measures of engagement were significantly associated with completion of follow-up surveys (odds ratios [OR] = 4.11 and 2.12, respectively, both P values < .001). The breadth measure of engagement was also significantly positively associated with a key study outcome, the mean increase in fruit and vegetable consumption (P < .001).

Conclusions: By exploring participants' exposures to online interventions, paradata are valuable in explaining the effects of tailoring in increasing participant engagement in the intervention. Controlling for intervention arm, greater engagement is also associated with retention of participants and positive change in a key outcome of the intervention, dietary change. This paper demonstrates the utility of paradata capture and analysis for evaluating online health interventions.

Trial registration: NCT00169312; https://ichgcp.net/clinical-trials-registry/NCT00169312 (Archived by WebCite at http://www.webcitation.org/5u8sSr0Ty).

Conflict of interest statement

None declared

References

    1. Griffiths F, Lindenmeyer A, Powell J, Lowe P, Thorogood M. Why are health care interventions delivered over the internet? A systematic review of the published literature. J Med Internet Res. 2006;8(2):e10. doi: 10.2196/jmir.8.2.e10. v8i2e10
    1. Strecher V. Internet methods for delivering behavioral and health-related interventions (eHealth) Annu Rev Clin Psychol. 2007;3:53–76. doi: 10.1146/annurev.clinpsy.3.022806.091428.
    1. Li D, Browne GJ, Wetherbe JC. Why do internet users stick with a specific web site? A relationship perspective. International Journal of Electronic Commerce. 2006;10(4):105–141.
    1. Lin JCC. Online stickiness: its antecedents and effect on purchasing intention. Behaviour & Information Technology. 2007;26(6):507–516.
    1. Glasgow RE, Nelson CC, Kearney KA, Reid R, Ritzwoller DP, Strecher VJ, Couper MP, Green B, Wildenhaus K. Reach, engagement, and retention in an Internet-based weight loss program in a multi-site randomized controlled trial. J Med Internet Res. 2007;9(2):e11. doi: 10.2196/jmir.9.2.e11. v9i2e11
    1. Wanner M, Martin-Diener E, Bauer G, Braun-Fahrländer C, Martin BW. Comparison of trial participants and open access users of a web-based physical activity intervention regarding adherence, attrition, and repeated participation. J Med Internet Res. 2010;12(1):e3. doi: 10.2196/jmir.1361. v12i1e3
    1. Murray E, Khadjesari Z, White IR, Kalaitzaki E, Godfrey C, McCambridge J, Thompson SG, Wallace P. Methodological challenges in online trials. J Med Internet Res. 2009;11(2):e9. doi: 10.2196/jmir.1052. v11i2e9
    1. Eysenbach G. The law of attrition. J Med Internet Res. 2005;7(1):e11. doi: 10.2196/jmir.7.1.e11. v7e11
    1. Wantland DJ, Portillo CJ, Holzemer WL, Slaughter R, McGhee EM. The effectiveness of Web-based vs. non-Web-based interventions: a meta-analysis of behavioral change outcomes. J Med Internet Res. 2004 Nov 10;6(4):e40. doi: 10.2196/jmir.6.4.e40. v6e40
    1. Crutzen R, de Nooijer J, Brouwer W, Oenema A, Brug J, de Vries NK. A conceptual framework for understanding and improving adolescents' exposure to Internet-delivered interventions. Health Promot Int. 2009 Sep;24(3):277–84. doi: 10.1093/heapro/dap018. dap018
    1. Danaher BG, Boles SM, Akers L, Gordon JS, Severson HH. Defining participant exposure measures in Web-based health behavior change programs. J Med Internet Res. 2006;8(3):e15. doi: 10.2196/jmir.8.3.e15. v8i3e15
    1. Neil AL, Batterham P, Christensen H, Bennett K, Griffiths KM. Predictors of adherence by adolescents to a cognitive behavior therapy website in school and community-based settings. J Med Internet Res. 2009;11(1):e6. doi: 10.2196/jmir.1050. v11i1e6
    1. Couper MP, Lyberg LE. The use of paradata in survey research. 55th Session of the International Statistical Institute; April 5-12, 2005; Sydney, Australia. 2005.
    1. Stern M. The use of client-side paradata in analyzing the effects of visual layout on changing responses in web surveys. Field Methods. 2008;20(4):377–398. doi: 10.1177/1525822X08320421.
    1. Haraldsen G, Kleven O, Stalnacke M. Paradata indications of problems in Web surveys. European Conference on Quality in Survey Statistics; April 24-26, 2006; Cardiff, Wales. 2006.
    1. Healey B. Drop downs and scroll mice: The effect of response option format and input mechanism employed on data quality in web surveys. Soc Sci Comput Rev. 2007;25(1):111–128. doi: 10.1177/0894439306293888.
    1. Heerwegh D. Explaining response latencies and changing answers using client-side paradata from a web survey. Soc Sci Comput Rev. 2003;21(3):360–373. doi: 10.1177/0894439303253985.
    1. Yan T, Tourangeau R. Fast times and easy questions: the effects of age, experience and question complexity on web survey response times. Appl Cogn Psychol. 2008;22(1):51–68.
    1. Strecher VJ, McClure J, Alexander G, Chakraborty B, Nair V, Konkel J, Greene S, Couper M, Carlier C, Wiese C, Little R, Pomerleau C, Pomerleau O. The role of engagement in a tailored web-based smoking cessation program: randomized controlled trial. J Med Internet Res. 2008;10(5):e36. doi: 10.2196/jmir.1002. v10i5e36
    1. Wagner EH, Greene SM, Hart G, Field TS, Fletcher S, Geiger AM, Herrinton LJ, Hornbrook MC, Johnson CC, Mouchawar J, Rolnick SJ, Stevens VJ, Taplin SH, Tolsma D, Vogt TM. Building a research consortium of large health systems: the Cancer Research Network. J Natl Cancer Inst Monogr. 2005;(35):3–11. doi: 10.1093/jncimonographs/lgi032.2005/35/3
    1. National Cancer Institute US National Institutes of Health. 2009. [2009-08-06]. Cancer Research Network
    1. Alexander GL, McClure JB, Calvi JH, Divine GW, Stopponi MA, Rolnick SJ, Heimendinger J, Tolsma DD, Resnicow K, Campbell MK, Strecher VJ, Johnson CC, MENU Choices Team A randomized clinical trial evaluating online interventions to improve fruit and vegetable consumption. Am J Public Health. 2010 Feb;100(2):319–26. doi: 10.2105/AJPH.2008.154468.AJPH.2008.154468
    1. Stopponi MA, Alexander GL, McClure JB, Carroll NM, Divine GW, Calvi JH, Rolnick SJ, Strecher VJ, Johnson CC, Ritzwoller DP. Recruitment to a randomized web-based nutritional intervention trial: characteristics of participants compared to non-participants. J Med Internet Res. 2009;11(3):e38. doi: 10.2196/jmir.1086. v11i3e38
    1. Alexander GL, Divine GW, Couper MP, McClure JB, Stopponi MA, Fortman KK, Tolsma DD, Strecher VJ, Johnson CC. Effect of incentives and mailing features on online health program enrollment. Am J Prev Med. 2008 May;34(5):382–8. doi: 10.1016/j.amepre.2008.01.028.S0749-3797(08)00153-0
    1. Schwarzer R, Schuz B, Ziegelmann JP, Lippke S, Luszczynska A, Scholz U. Adoption and maintenance of four health behaviors: theory-guided longitudinal studies on dental flossing, seat belt use, dietary behavior, and physical activity. Ann Behav Med. 2007 Apr;33(2):156–66. doi: 10.1080/08836610701308221.
    1. Miller W, Rolnick S. Motivational Interviewing: Preparing people to change addictive behavior. New York, NY: Guilford Press; 1991.
    1. Resnicow K, DiIorio C, Soet JE, Ernst D, Borrelli B, Hecht J. Motivational interviewing in health promotion: it sounds like something is changing. Health Psychol. 2002 Sep;21(5):444–51.
    1. Thompson FE, Subar AF, Smith AF, Midthune D, Radimer KL, Kahle LL, Kipnis V. Fruit and vegetable assessment: performance of 2 new short instruments and a food frequency questionnaire. J Am Diet Assoc. 2002 Dec;102(12):1764–72.
    1. Resnicow K, Odom E, Wang T, Dudley WN, Mitchell D, Vaughan R, Jackson A, Baranowski T. Validation of three food frequency questionnaires and 24-hour recalls with serum carotenoid levels in a sample of African-American adults. Am J Epidemiol. 2000 Dec 1;152(11):1072–80.
    1. Williams GC, Deci EL. Activating patients for smoking cessation through physician autonomy support. Med Care. 2001 Aug;39(8):813–23.
    1. Resnicow K, Davis RE, Zhang G, Konkel J, Strecher VJ, Shaikh AR, Tolsma D, Calvi J, Alexander G, Anderson JP, Wiese C. Tailoring a fruit and vegetable intervention on novel motivational constructs: results of a randomized study. Ann Behav Med. 2008 Apr;35(2):159–69. doi: 10.1007/s12160-008-9028-9.
    1. Liang K-Y, Zeger SL. Longitudinal data analysis using generalized linear models. Biometrika. 1986;73(1):13–22. doi: 10.1093/biomet/73.1.13.
    1. Nagelkerke NJD. A note on a general definition of the coefficient of determination. Biometrika. 1991;78(3):691–692. doi: 10.1093/biomet/78.3.691.

Source: PubMed

Sponsorer og samarbejdspartnere

Medicinske tilstande

Narkotikainterventioner

3
Abonner