Psychometric Evaluation of the TWente Engagement with Ehealth Technologies Scale (TWEETS): Evaluation Study

Saskia Marion Kelders, Hanneke Kip, Japie Greeff, Saskia Marion Kelders, Hanneke Kip, Japie Greeff

Abstract

Background: Engagement emerges as a predictor for the effectiveness of digital health interventions. However, a shared understanding of engagement is missing. Therefore, a new scale has been developed that proposes a clear definition and creates a tool to measure it. The TWente Engagement with Ehealth Technologies Scale (TWEETS) is based on a systematic review and interviews with engaged health app users. It defines engagement as a combination of behavior, cognition, and affect.

Objective: This paper aims to evaluate the psychometric properties of the TWEETS. In addition, a comparison is made with the experiential part of the Digital Behavior Change Intervention Engagement Scale (DBCI-ES-Ex), a scale that showed some issues in previous psychometric analyses.

Methods: In this study, 288 participants were asked to use any step counter app on their smartphones for 2 weeks. They completed online questionnaires at 4 time points: T0=baseline, T1=after 1 day, T2=1 week, and T3=2 weeks. At T0, demographics and personality (conscientiousness and intellect/imagination) were assessed; at T1-T3, engagement, involvement, enjoyment, subjective usage, and perceived behavior change were included as measures that are theoretically related to our definition of engagement. Analyses focused on internal consistency, reliability, and the convergent, divergent, and predictive validity of both engagement scales. Convergent validity was assessed by correlating the engagement scales with involvement, enjoyment, and subjective usage; divergent validity was assessed by correlating the engagement scales with personality; and predictive validity was assessed by regression analyses using engagement to predict perceived behavior change at later time points.

Results: The Cronbach alpha values of the TWEETS were .86, .86, and .87 on T1, T2, and T3, respectively. Exploratory factor analyses indicated that a 1-factor structure best fits the data. The TWEETS is moderately to strongly correlated with involvement and enjoyment (theoretically related to cognitive and affective engagement, respectively; P<.001). Correlations between the TWEETS and frequency of use were nonsignificant or small, and differences between adherers and nonadherers on the TWEETS were significant (P<.001). Correlations between personality and the TWEETS were nonsignificant. The TWEETS at T1 was predictive of perceived behavior change at T3, with an explained variance of 16%. The psychometric properties of the TWEETS and the DBCI-ES-Ex seemed comparable in some aspects (eg, internal consistency), and in other aspects, the TWEETS seemed somewhat superior (divergent and predictive validity).

Conclusions: The TWEETS performs quite well as an engagement measure with high internal consistency, reasonable test-retest reliability and convergent validity, good divergent validity, and reasonable predictive validity. As the psychometric quality of a scale is a reflection of how closely a scale matches the conceptualization of a concept, this paper is also an attempt to conceptualize and define engagement as a unique concept, providing a first step toward an acceptable standard of defining and measuring engagement.

Keywords: adherence; adoption; attrition; digital health interventions; eHealth; engagement; questionnaire; scale validation.

Conflict of interest statement

Conflicts of Interest: None declared.

©Saskia Marion Kelders, Hanneke Kip, Japie Greeff. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 26.09.2020.

Figures

Figure 1
Figure 1
Engagement scores of adherers and nonadherers at the different time points. DBCI-ES-Ex: the experiential subscale of the Digital Behavior Change Intervention Engagement Scale; T: time point; TWEETS: TWente Engagement with Ehealth Technologies Scale.

References

    1. Christensen H, Griffiths KM, Farrer L. Adherence in Internet Interventions for Anxiety and Depression. J Med Internet Res. 2009 Apr 24;11(2):e13. doi: 10.2196/jmir.1194.
    1. Kelders SM, Kok RN, Ossebaard HC, Van Gemert-Pijnen JE. Persuasive System Design Does Matter: a Systematic Review of Adherence to Web-based Interventions. J Med Internet Res. 2012 Nov 14;14(6):e152. doi: 10.2196/jmir.2104.
    1. Donkin L, Christensen H, Naismith SL, Neal B, Hickie IB, Glozier N. A Systematic Review of the Impact of Adherence on the Effectiveness of e-Therapies. J Med Internet Res. 2011 Aug 05;13(3):e52. doi: 10.2196/jmir.1772.
    1. Donkin L, Hickie IB, Christensen H, Naismith SL, Neal B, Cockayne NL, Glozier N. Rethinking the Dose-Response Relationship Between Usage and Outcome in an Online Intervention for Depression: Randomized Controlled Trial. J Med Internet Res. 2013 Oct 17;15(10):e231. doi: 10.2196/jmir.2771.
    1. Sieverink F, Kelders SM, van Gemert-Pijnen JE. Clarifying the Concept of Adherence to eHealth Technology: Systematic Review on When Usage Becomes Adherence. J Med Internet Res. 2017 Dec 06;19(12):e402. doi: 10.2196/jmir.8578.
    1. Donkin L, Glozier N. Motivators and Motivations to Persist With Online Psychological Interventions: A Qualitative Study of Treatment Completers. J Med Internet Res. 2012 Jun 22;14(3):e91. doi: 10.2196/jmir.2100.
    1. Kelders S.M. M. International Conference on Persuasive Technology; 2015; Chicago. 2015.
    1. Kelders SM, van Zyl LE, Ludden GDS. The Concept and Components of Engagement in Different Domains Applied to eHealth: A Systematic Scoping Review. Front. Psychol. 2020 May 27;11:926. doi: 10.3389/fpsyg.2020.00926.
    1. Perski O, Blandford A, West R, Michie S. Conceptualising engagement with digital behaviour change interventions: a systematic review using principles from critical interpretive synthesis. Behav. Med. Pract. Policy Res. 2016 Dec 13;7(2):254–267. doi: 10.1007/s13142-016-0453-1.
    1. Yardley L, Spring BJ, Riper H, Morrison LG, Crane DH, Curtis K, Merchant GC, Naughton F, Blandford A. Understanding and Promoting Effective Engagement With Digital Behavior Change Interventions. American Journal of Preventive Medicine. 2016 Nov;51(5):833–842. doi: 10.1016/j.amepre.2016.06.015.
    1. Hibbard JH, Greene J. What The Evidence Shows About Patient Activation: Better Health Outcomes And Care Experiences; Fewer Data On Costs. Health Affairs. 2013 Feb;32(2):207–214. doi: 10.1377/hlthaff.2012.1061.
    1. Cacciamani Giovanni, Aron Monish, Gill Inderbir, Desai Mihir. Oncological outcome according to attainment of pentafecta after robot assisted radical cystectomy among bladder cancer patients using KORARC database (730 multicenter robot radical cystectomy database) BJU Int 2020 Jul 8. doi: 10.1111/ bju.15178. BJU Int. 2020 Sep 01;:15178. doi: 10.1111/bju.15222.
    1. Kelders S. Design for Engagement of Online Positive Psychology Interventions. In: van Zyl L, Rothmann S, editors. Positive Psychological Intervention Design and Protocols for Multi-Cultural Contexts. Cham: Springer; 2019. pp. 297–313.
    1. Short CE, DeSmet A, Woods C, Williams SL, Maher C, Middelweerd A, Müller AM, Wark PA, Vandelanotte C, Poppe L, Hingle MD, Crutzen R. Measuring Engagement in eHealth and mHealth Behavior Change Interventions: Viewpoint of Methodologies. J Med Internet Res. 2018 Nov 16;20(11):e292. doi: 10.2196/jmir.9397.
    1. Kelders S M, Kip H. Development and initial validation of a scale to measure engagement with eHealth technologies. Extended Abstracts of the CHI Conference on Human Factors in Computing Systems; 2019; Glasgow. 2019.
    1. Short CE, Rebar AL, Plotnikoff RC, Vandelanotte C. Designing engaging online behaviour change interventions: a proposed model of user engagement. European Health Psychologist. 2015;17(1):32–38.
    1. O'Brien HL, Toms EG. What is user engagement? A conceptual framework for defining user engagement with technology. J. Am. Soc. Inf. Sci. 2008 Apr;59(6):938–955. doi: 10.1002/asi.20801.
    1. Lefebvre R. C.., The assessment of user engagement with eHealth content: the eHealth engagement scale. Journal of Computer-Mediated Communication. 2010;4(15):666–681. doi: 10.1111/j.1083-6101.2009.01514.x.
    1. Perski O, Blandford A, Garnett C, Crane D, West R, Michie S. A self-report measure of engagement with digital behavior change interventions (DBCIs): development and psychometric evaluation of the “DBCI Engagement Scale”. Translational behavioral medicine. 2019 Mar 30;:267–277. doi: 10.1093/tbm/ibz039.
    1. Perski O, Lumsden J, Garnett C, Blandford A, West R, Michie S. Assessing the Psychometric Properties of the Digital Behavior Change Intervention Engagement Scale in Users of an App for Reducing Alcohol Consumption: Evaluation Study. J Med Internet Res. 2019 Nov 20;21(11):e16197. doi: 10.2196/16197.
    1. Graffigna G. Is a Transdisciplinary Theory of Engagement in Organized Settings Possible? A Concept Analysis of the Literature on Employee Engagement, Consumer Engagement and Patient Engagement. Front. Psychol. 2017 Jul 06;8:872. doi: 10.3389/fpsyg.2017.00872.
    1. Crutzen R, van ’t Riet J, Short CE. Enjoyment: A Conceptual Exploration and Overview of Experimental Evidence in the Context of Games for Health. Games for Health Journal. 2016 Feb;5(1):15–20. doi: 10.1089/g4h.2015.0059.
    1. Zaichkowsky JL. Measuring the Involvement Construct. J CONSUM RES. 1985 Dec;12(3):341. doi: 10.1086/208520.
    1. Kelders SM, Sommers-Spijkerman M, Goldberg J. Investigating the Direct Impact of a Gamified Versus Nongamified Well-Being Intervention: An Exploratory Experiment. J Med Internet Res. 2018 Jul 26;20(7):e247. doi: 10.2196/jmir.9923.
    1. Forbes L, Gutierrez D, Johnson SK. Investigating Adherence to an Online Introductory Mindfulness Program. Mindfulness. 2017 Jul 14;9(1):271–282. doi: 10.1007/s12671-017-0772-4.
    1. Donnellan MB, Oswald FL, Baird BM, Lucas RE. The Mini-IPIP Scales: Tiny-yet-effective measures of the Big Five Factors of Personality. Psychological Assessment. 2006;18(2):192–203. doi: 10.1037/1040-3590.18.2.192.
    1. Goldberg L. R., A broad-bandwidth, public domain, personality inventory measuring the lower-level facets of several five-factor models. Personality psychology in Europe. 1999;7(1):7–28.
    1. McAuley E, Duncan T, Tammen VV. Psychometric Properties of the Intrinsic Motivation Inventory in a Competitive Sport Setting: A Confirmatory Factor Analysis. Research Quarterly for Exercise and Sport. 1989 Mar;60(1):48–58. doi: 10.1080/02701367.1989.10607413.
    1. Bishara AJ, Hittner JB. Testing the significance of a correlation with nonnormal data: Comparison of Pearson, Spearman, transformation, and resampling approaches. Psychological Methods. 2012 Sep;17(3):399–417. doi: 10.1037/a0028087.
    1. Ghasemi A, Zahediasl S. Normality Tests for Statistical Analysis: A Guide for Non-Statisticians. Int J Endocrinol Metab. 2012 Dec 01;10(2):486–489. doi: 10.5812/ijem.3505.
    1. Schmidt AF, Finan C. Linear regression and the normality assumption. Journal of Clinical Epidemiology. 2018 Jun;98:146–151. doi: 10.1016/j.jclinepi.2017.12.006.
    1. Tavakol M, Dennick R. Making sense of Cronbach's alpha. Int. J. Medical Education. 2011 Jun 27;2:53–55. doi: 10.5116/ijme.4dfb.8dfd.
    1. Hinkin TR. A Brief Tutorial on the Development of Measures for Use in Survey Questionnaires. Organizational Research Methods. 2016 Jun 29;1(1):104–121. doi: 10.1177/109442819800100106.
    1. Osborne J W, Costello A B, Kellow J T. Best practices in exploratory factor analysis. Louisville, KY: CreateSpace Independent Publishing Platform. 2008:86–99. doi: 10.4135/9781412995627.d8.
    1. Crocker L, Algina J. Introduction to classical and modern test theory. Orlando, Florida: Holt, Rinehart and Winston; 1986.
    1. Sieverink F, Kelders S, Braakman-Jansen A, van Gemert-Pijnen J. Evaluating the implementation of a personal health record for chronic primary and secondary care: a mixed methods approach. BMC Med Inform Decis Mak. 2019 Nov 27;19(1):241. doi: 10.1186/s12911-019-0969-7.

Source: PubMed

Sponsorer og samarbeidspartnere

Medisinsk tilstand

Legemiddelintervensjoner

3
Abonnere