A self-report measure of engagement with digital behavior change interventions (DBCIs): development and psychometric evaluation of the "DBCI Engagement Scale"

Olga Perski, Ann Blandford, Claire Garnett, David Crane, Robert West, Susan Michie, Olga Perski, Ann Blandford, Claire Garnett, David Crane, Robert West, Susan Michie

Abstract

Engagement with digital behavior change interventions (DBCIs) is a potentially important mediator of effectiveness; however, we lack validated measures of engagement. This study describes (a) the development of a self-report scale that captures the purported behavioral and experiential facets of engagement and (b) the evaluation of its validity in a real-world setting. A deductive approach to item generation was taken. The sample consisted of adults in the UK who drink excessively, downloaded the freely available Drink Less app with the intention to reduce alcohol consumption, and completed the scale immediately after their first login. Five types of validity (i.e., construct, criterion, predictive, incremental, divergent) were examined using exploratory factor analysis, correlational analyses, and through regressing the number of subsequent logins in the next 14 days onto total scale scores. Cronbach's α was calculated to assess internal reliability. A 10-item scale assessing amount and depth of use, interest, enjoyment, and attention was generated. Of 5,460 eligible users, only 203 (3.7%) users completed the scale. Seven items were retained, and the scale was found to be unifactorial and internally reliable (α = 0.77). Divergent and criterion validity were not established. Total scale scores were not significantly associated with the number of subsequent logins (B = 0.02; 95% CI = -0.01 to 0.05; p = .14). Behavioral and experiential indicators of engagement with DBCIs may constitute a single dimension, but low response rates to engagement surveys embedded in DBCIs may make their use impracticable in real-world settings.

Keywords: Alcohol reduction; Digital behavior change interventions; Engagement; Psychometric evaluation; Self-report scale; eHealth; mHealth.

© The Author(s) 2019. Published by Oxford University Press on behalf of the Society of Behavioral Medicine.

References

    1. Yardley L, Spring BJ, Riper H, et al. . Understanding and promoting effective engagement with digital behavior change interventions. Am J Prev Med. 2016;51(5):833–842.
    1. Perski O, Blandford A, West R, Michie S. Conceptualising engagement with digital behaviour change interventions: a systematic review using principles from critical interpretive synthesis. Transl Behav Med. 2017;7(2):254–267.
    1. Michie S, Yardley L, West R, Patrick K, Greaves F. Developing and evaluating digital interventions to promote behavior change in health and health care: recommendations resulting from an international workshop. J Med Internet Res. 2017;19(6):e232.
    1. Appboy. Spring 2016 Mobile Customer Retention Report: An Analysis of Retention by Day. 2016.
    1. Consumer Health Information Corporation. Motivating Patients to Use Smartphone Health Apps. 2015. Available at . Accessibilty verified August 10, 2015.
    1. Kelders SM, Kok RN, Ossebaard HC, Van Gemert-Pijnen JE. Persuasive system design does matter: a systematic review of adherence to web-based interventions. J Med Internet Res. 2012;14(6):e152.
    1. Alexander GL, McClure JB, Calvi JH, et al. ; MENU Choices Team . A randomized clinical trial evaluating online interventions to improve fruit and vegetable consumption. Am J Public Health. 2010;100(2):319–326.
    1. Cobb NK, Graham AL, Bock BC, Papandonatos G, Abrams DB. Initial evaluation of a real-world Internet smoking cessation system. Nicotine Tob Res. 2005;7(2):207–216.
    1. Donkin L, Christensen H, Naismith SL, Neal B, Hickie IB, Glozier N. A systematic review of the impact of adherence on the effectiveness of e-therapies. J Med Internet Res. 2011;13(3):e52.
    1. Hilvert-Bruce Z, Rossouw PJ, Wong N, Sunderland M, Andrews G. Adherence as a determinant of effectiveness of internet cognitive behavioural therapy for anxiety and depressive disorders. Behav Res Ther. 2012;50(7–8):463–468.
    1. Csikszentmihalyi M.Flow: The Psychology of Optimal Performance. New York, NY: Cambridge University Press; 1990.
    1. Brown E, Cairns P. A grounded investigation of game immersion. In: CHI ‘04 Extended Abstracts on Human Factors in Computing Systems. ACM. 2004:1297–1300.
    1. Michie S, Richardson M, Johnston M, et al. . The behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: building an international consensus for the reporting of behavior change interventions. Ann Behav Med. 2013;46(1):81–95.
    1. Bellg AJ, Borrelli B, Resnick B, et al. ; Treatment Fidelity Workgroup of the NIH Behavior Change Consortium . Enhancing treatment fidelity in health behavior change studies: best practices and recommendations from the NIH Behavior Change Consortium. Health Psychol. 2004;23(5):443–451.
    1. Kelders SM. Involvement as a working mechanism for persuasive technology. In: MacTavish T., Basapur S., ed. Persuasive Technology: 10th International Conference. Chicago, IL: Springer; 2015:3–14.
    1. O’Brien HL, Toms EG. The development and evaluation of a survey to measure user engagement. J Am Soc Inf Sci Technol. 2010;61(1):50–69.
    1. Lefebvre RC, Tada Y, Hilfiker SW, Baur C. The assessment of user engagement with ehealth content: the ehealth engagement scale. J Comput Mediat Commun. 2010;15:666–681.
    1. Jackson SA, Marsh HW. Development and validation of a scale to measure optimal experience: the flow state scale. J Sport Exerc Psychol. 1996;18:17–35.
    1. Jennett C, Cox AL, Cairns P, et al. . Measuring and defining the experience of immersion in games. Int J Hum-Comput Stud. 2008;66(9):641–661.
    1. Zaichkowsky JL. Measuring the involvement construct. J Cons Res. 1985;12(3):341–352.
    1. Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth. 2015;3(1):e27.
    1. Danaher BG, Boles SM, Akers L, Gordon JS, Severson HH. Defining participant exposure measures in Web-based health behavior change programs. J Med Internet Res. 2006;8(3):e15.
    1. Strecher VJ, McClure J, Alexander G, et al. . The role of engagement in a tailored web-based smoking cessation program: randomized controlled trial. J Med Internet Res. 2008;10(5):e36.
    1. McClure JB, Shortreed SM, Bogart A, et al. . The effect of program design on engagement with an internet-based smoking intervention: randomized factorial trial. J Med Internet Res. 2013;15(3):e69.
    1. Couper MP, Alexander GL, Zhang N, et al. . Engagement and retention: measuring breadth and depth of participant use of an online intervention. J Med Internet Res. 2010;12(4):e52.
    1. Hinkin TR. A brief tutorial on the development of measures for use in survey questionnaires. Organ Res Methods. 1998;1(1):104–121.
    1. Weng LJ. Impact of the number of response categories and anchor labels on coefficient alpha and test-retest reliability. Educ Psychol Meas. 2004;64(6):956–972.
    1. MacKenzie SB, Podsakoff PM, Fetter R. Organizational citizenship behavior and objective productivity as determinants of managerial evaluations of salespersons’ performance. Organ Behav Hum Decis Process. 1991;50:123–150.
    1. Hinkin TR, Schriesheim CA. Development and application of new scales to measure the French and Raven (1959) bases of social power. J Appl Psychol. 1989;74(4):561–567.
    1. Qualtrics. Qualtrics Survey Software. Provo, Utah. 2016. Available at . Accessibilty verified April, 2016.
    1. Crane D, Garnett C, Michie S, West R, Brown J. Publisher correction: a smartphone app to reduce excessive alcohol consumption: identifying the effectiveness of intervention components in a factorial randomised control trial. Sci Rep. 2018;8(1):6866.
    1. Babor TF, Higgins-Biddle JC, Saunders JB, Monteiro MG.. The Alcohol Use Disorders Identification Test: Guidelines for Use in Primary Care, 2nd ed. Geneva, Switzerland: World Health Organisation; 2001.
    1. Google Analytics. How a Web Session is Defined in Analytics.2017. Available at . Accessibility verified February 6, 2018.
    1. Fan W, Yan Z. Factors affecting response rates of the web survey: a systematic review. Comput Hum Behav. 2010; 26(2):132–139.
    1. IBM Corp. IBM SPSS Statistics for Windows, Version 21.0. Armonk, NY: IBM Corporation; 2012.
    1. Costello AB, Osborne JW. Best practices in exploratory factor analysis: four recommendations for getting the most from your analysis. Pract Assess Res Eval. 2005;10(7):1–9.
    1. Morrison L, Geraghty AWA, Lloyd S, et al. (n.d.). Comparing usage of a web- and smartphone app-delivered stress management intervention: an observational study. Internet Interv. 2018;12:74–82.
    1. Kaiser HF. An index of factorial simplicity. Psychometrika. 1974;39(1):31–36.
    1. Faul F, Erdfelder E, Lang AG, Buchner A. G*Power 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav Res Methods. 2007;39(2):175–191.
    1. Bricker JB, Mull KE, Kientz JA, et al. . Randomized, controlled pilot trial of a smartphone app for smoking cessation using acceptance and commitment therapy. Drug Alcohol Depend. 2014;143:87–94.
    1. Murray E, White IR, Varagunam M, Godfrey C, Khadjesari Z, McCambridge J. Attrition revisited: adherence and retention in a web-based alcohol trial. J Med Internet Res. 2013;15(8):e162.
    1. Willis GB.Cognitive Interviewing: A Tool for Improving Questionnaire Design. Thousand Oaks, CA: SAGE Publications; 2004.
    1. Milward J, Drummond C, Fincham-Campbell S, Deluca P. What makes online substance-use interventions engaging? A systematic review and narrative synthesis. Digit Health. 2018;4:1–25.
    1. Petty RE, Cacioppo J. The elaboration likelihood model of persuasion. Adv Exp Soc Psychol. 1986;19:123–205.
    1. Borrelli B. The assessment, monitoring, and enhancement of treatment fidelity in public health clinical trials. J Public Health Dent. 2011;71(suppl 1):S52–S63.
    1. Sieverink F, Kelders SM, van Gemert-Pijnen JE. Clarifying the concept of adherence to ehealth technology: systematic review on when usage becomes adherence. J Med Internet Res. 2017;19(12):e402.
    1. Petrides KV, Furnham A. Trait emotional intelligence: psychometric investigation with reference to established trait taxonomies. Eur J Pers. 2001;15(6):425–448.
    1. Michie S, West R, Campbell R, Brown J, Gainforth H.. ABC of Behaviour Change Theories. London, UK: Silverback Publishing; 2014.
    1. Hekler EB, Michie S, Pavel M, et al. . Advancing models and theories for digital behavior change interventions. Am J Prev Med. 2016;51(5):825–832.
    1. Stone AA, Shiffman S. Ecological momentary assessment (EMA) in behavorial medicine. Ann Behav Med. 1994;16(3):199–202.
    1. Riley WT, Rivera DE, Atienza AA, Nilsen W, Allison SM, Mermelstein R. Health behavior models in the age of mobile interventions: are our theories up to the task? Transl Behav Med. 2011;1(1):53–71.

Source: PubMed

Спонсоры и соавторы

Заболевания

Медикаментозные вмешательства

Подписаться