Reliability and Validity of Assessing User Satisfaction With Web-Based Health Interventions

Leif Boß, Dirk Lehr, Dorota Reis, Christiaan Vis, Heleen Riper, Matthias Berking, David Daniel Ebert, Leif Boß, Dirk Lehr, Dorota Reis, Christiaan Vis, Heleen Riper, Matthias Berking, David Daniel Ebert

Abstract

Background: The perspective of users should be taken into account in the evaluation of Web-based health interventions. Assessing the users' satisfaction with the intervention they receive could enhance the evidence for the intervention effects. Thus, there is a need for valid and reliable measures to assess satisfaction with Web-based health interventions.

Objective: The objective of this study was to analyze the reliability, factorial structure, and construct validity of the Client Satisfaction Questionnaire adapted to Internet-based interventions (CSQ-I).

Methods: The psychometric quality of the CSQ-I was analyzed in user samples from 2 separate randomized controlled trials evaluating Web-based health interventions, one from a depression prevention intervention (sample 1, N=174) and the other from a stress management intervention (sample 2, N=111). At first, the underlying measurement model of the CSQ-I was analyzed to determine the internal consistency. The factorial structure of the scale and the measurement invariance across groups were tested by multigroup confirmatory factor analyses. Additionally, the construct validity of the scale was examined by comparing satisfaction scores with the primary clinical outcome.

Results: Multigroup confirmatory analyses on the scale yielded a one-factorial structure with a good fit (root-mean-square error of approximation =.09, comparative fit index =.96, standardized root-mean-square residual =.05) that showed partial strong invariance across the 2 samples. The scale showed very good reliability, indicated by McDonald omegas of .95 in sample 1 and .93 in sample 2. Significant correlations with change in depressive symptoms (r=-.35, P<.001) and perceived stress (r=-.48, P<.001) demonstrated the construct validity of the scale.

Conclusions: The proven internal consistency, factorial structure, and construct validity of the CSQ-I indicate a good overall psychometric quality of the measure to assess the user's general satisfaction with Web-based interventions for depression and stress management. Multigroup analyses indicate its robustness across different samples. Thus, the CSQ-I seems to be a suitable measure to consider the user's perspective in the overall evaluation of Web-based health interventions.

Keywords: Internet; clinical effectiveness; evaluation; mental health; personal satisfaction.

Conflict of interest statement

Conflicts of Interest: None declared.

References

    1. Andersson G, Titov N. Advantages and limitations of Internet-based interventions for common mental disorders. World Psychiatry. 2014 Feb;13(1):4–11. doi: 10.1002/wps.20083. doi: 10.1002/wps.20083.
    1. Johansson R, Andersson G. Internet-based psychological treatments for depression. Expert Rev Neurother. 2012 Jul;12(7):861–869. doi: 10.1586/ern.12.63.
    1. Richards D, Richardson T. Computer-based psychological treatments for depression: a systematic review and meta-analysis. Clin Psychol Rev. 2012 Jun;32(4):329–342. doi: 10.1016/j.cpr.2012.02.004.S0272-7358(12)00027-X
    1. Mayo-Wilson E, Montgomery P. Media-delivered cognitive behavioural therapy and behavioural therapy (self-help) for anxiety disorders in adults. Cochrane Database Syst Rev. 2013;9:CD005330. doi: 10.1002/14651858.CD005330.pub4.
    1. Hedman E, Ljótsson B, Lindefors N. Cognitive behavior therapy via the Internet: a systematic review of applications, clinical efficacy and cost-effectiveness. Expert Rev Pharmacoecon Outcomes Res. 2012 Dec;12(6):745–764. doi: 10.1586/erp.12.67.
    1. Cheng SK, Dizon J. Computerised cognitive behavioural therapy for insomnia: a systematic review and meta-analysis. Psychother Psychosom. 2012;81(4):206–216. doi: 10.1159/000335379.000335379
    1. Riper H, Blankers M, Hadiwijaya H, Cunningham J, Clarke S, Wiers R, Ebert D, Cuijpers P. Effectiveness of guided and unguided low-intensity internet interventions for adult alcohol misuse: a meta-analysis. PLoS One. 2014;9(6):e99912. doi: 10.1371/journal.pone.0099912. PONE-D-14-12620
    1. Drozd F, Raeder S, Kraft P, Bjørkli CA. Multilevel growth curve analyses of treatment effects of a Web-based intervention for stress reduction: randomized controlled trial. J Med Internet Res. 2013;15(4):e84. doi: 10.2196/jmir.2570. v15i4e84
    1. Ebert DD, Heber E, Berking M, Riper H, Cuijpers P, Funk B, Lehr D. Self-guided internet-based and mobile-based stress management for employees: results of a randomised controlled trial. Occup Environ Med. 2016 May;73(5):315–323. doi: 10.1136/oemed-2015-103269.oemed-2015-103269
    1. Wolever RQ, Bobinet KJ, McCabe K, Mackenzie ER, Fekete E, Kusnick CA, Baime M. Effective and viable mind-body stress reduction in the workplace: a randomized controlled trial. J Occup Health Psychol. 2012 Apr;17(2):246–258. doi: 10.1037/a0027278.2012-04383-001
    1. Morledge TJ, Allexandre D, Fox E, Fu AZ, Higashi MK, Kruzikas DT, Pham SV, Reese PR. Feasibility of an online mindfulness program for stress management--a randomized, controlled trial. Ann Behav Med. 2013 Oct;46(2):137–148. doi: 10.1007/s12160-013-9490-x.
    1. Glasgow RE, Green LW, Klesges LM, Abrams DB, Fisher EB, Goldstein MG, Hayman LL, Ockene JK, Orleans CT. External validity: we need to do more. Ann Behav Med. 2006 Apr;31(2):105–108. doi: 10.1207/s15324796abm3102_1.
    1. Green LW, Glasgow RE. Evaluating the relevance, generalization, and applicability of research: issues in external validation and translation methodology. Eval Health Prof. 2006 Mar;29(1):126–153. doi: 10.1177/0163278705284445.29/1/126
    1. Manary MP, Boulding W, Staelin R, Glickman SW. The patient experience and health outcomes. N Engl J Med. 2013 Jan 17;368(3):201–203. doi: 10.1056/NEJMp1211775.
    1. Ware JE, Snyder MK, Wright WR, Davies AR. Defining and measuring patient satisfaction with medical care. Eval Program Plann. 1983;6(3-4):247–263.
    1. Ludden GD, van Rompay TJ, Kelders SM, van Gemert-Pijnen JE. How to Increase Reach and Adherence of Web-Based Interventions: A Design Research Viewpoint. J Med Internet Res. 2015;17(7):e172. doi: 10.2196/jmir.4201. v17i7e172
    1. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, Griffey R, Hensley M. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011 Mar;38(2):65–76. doi: 10.1007/s10488-010-0319-7.
    1. Glasgow RE. eHealth evaluation and dissemination research. Am J Prev Med. 2007 May;32(5 Suppl):S119–126. doi: 10.1016/j.amepre.2007.01.023.S0749-3797(07)00052-9
    1. Ankuta GY, Abeles N. Client satisfaction, clinical significance, and meaningful change in psychotherapy. Prof Psychol Res Pr. 1993;24(1):70–74. doi: 10.1037/0735-7028.24.1.70.
    1. Puschner B, Bauer S, Kraft S, Kordy H. Zufriedenheit von Patienten und Therapeuten mit ambulanter Psychotherapie [Patient and therapist satisfaction in outpatient psychotherapy] Psychother Psychosom Med Psychol. 2005 Dec;55(12):517–526. doi: 10.1055/s-2005-867050.
    1. Holcomb WR, Parker JC, Leong GB, Thiele J, Higdon J. Customer satisfaction and self-reported treatment outcomes among psychiatric inpatients. Psychiatr Serv. 1998 Jul;49(7):929–934. doi: 10.1176/ps.49.7.929.
    1. Gebhardt S, Wolak AM, Huber MT. Patient satisfaction and clinical parameters in psychiatric inpatients--the prevailing role of symptom severity and pharmacologic disturbances. Compr Psychiatry. 2013 Jan;54(1):53–60. doi: 10.1016/j.comppsych.2012.03.016.S0010-440X(12)00057-0
    1. Christensen H, Griffiths KM, Farrer L. Adherence in internet interventions for anxiety and depression. J Med Internet Res. 2009;11(2):e13. doi: 10.2196/jmir.1194. v11i2e13
    1. Barbosa CD, Balp M, Kulich K, Germain N, Rofail D. A literature review to explore the link between treatment satisfaction and adherence, compliance, and persistence. Patient Prefer Adherence. 2012;6:39–48. doi: 10.2147/PPA.S24752. doi: 10.2147/PPA.S24752.ppa-6-039
    1. van Ballegooijen W, Riper H, Cuijpers P, van Oppen P, Smit JH. Validation of online psychometric instruments for common mental health disorders: a systematic review. BMC Psychiatry. 2016;16:45. doi: 10.1186/s12888-016-0735-7. 10.1186/s12888-016-0735-7
    1. Marshall G, Hays R. RAND. The Patient Satisfaction Questionnaire Short Form (PSQ-18) .
    1. Greenfield T, Attkisson C. The UCSF client Satisfcation Scales: II. The Service Satisfaction Scale-30. In: Maruish ME, editor. The use of psychological testing for treatment planning and outcomes assessment. 3rd edition. Mashwah, NJ: Lawrance Erlbaum Associates; 2004. pp. 813–837.
    1. Pound P, Tilling K, Rudd AG, Wolfe CD. Does patient satisfaction reflect differences in care received after stroke? Stroke. 1999 Jan;30(1):49–55.
    1. Larsen DL, Attkisson CC, Hargreaves WA, Nguyen TD. Assessment of client/patient satisfaction: development of a general scale. Eval Program Plann. 1979;2(3):197–207.
    1. Attkisson CC, Zwick R. The client satisfaction questionnaire. Psychometric properties and correlations with service utilization and psychotherapy outcome. Eval Program Plann. 1982;5(3):233–237.
    1. Schmidt J, Lamprecht F, Wittmann WW. [Satisfaction with inpatient management. Development of a questionnaire and initial validity studies] Psychother Psychosom Med Psychol. 1989 Jul;39(7):248–255.
    1. Kriz D, Nübling R, Steffanowski A, Wittmann W, Schmidt J. Patientenzufriedenheit in der stationären Rehabilitation: Psychometrische Reanalyse des ZUF-8 auf der Basis multizentrischer Stichproben verschiedener Indikation [Patients’ satisfaction in inpatient rehabilitation. Psychometrical evaluation of the ZUF-8 based on a multicenter sample of different indications] Z Med psychol. 2008;17:67–79.
    1. Donker T, Bennett K, Bennett A, Mackinnon A, van SA, Cuijpers P, Christensen H, Griffiths KM. Internet-delivered interpersonal psychotherapy versus internet-delivered cognitive behavioral therapy for adults with depressive symptoms: randomized controlled noninferiority trial. J Med Internet Res. 2013;15(5):e82. doi: 10.2196/jmir.2307. v15i5e82
    1. Berger T, Hämmerli K, Gubser N, Andersson G, Caspar F. Internet-based treatment of depression: a randomized controlled trial comparing guided with unguided self-help. Cogn Behav Ther. 2011 Dec;40(4):251–266. doi: 10.1080/16506073.2011.616531.
    1. Palfai TP, Saitz R, Winter M, Brown TA, Kypri K, Goodness TM, O'Brien LM, Lu J. Web-based screening and brief intervention for student marijuana use in a university health center: pilot study to examine the implementation of eCHECKUP TO GO in different contexts. Addict Behav. 2014 Sep;39(9):1346–1352. doi: 10.1016/j.addbeh.2014.04.025. S0306-4603(14)00137-3
    1. Ebert DD, Berking M, Thiart H, Riper H, Laferton JA, Cuijpers P, Sieland B, Lehr D. Restoring depleted resources: Efficacy and mechanisms of change of an internet-based unguided recovery training for better sleep and psychological detachment from work. Health Psychol. 2015 Dec;34 Suppl:1240–1251. doi: 10.1037/hea0000277.2015-56045-005
    1. Jöreskog KG. Simultaneous factor analysis in several populations. Psychometrika. 1971 Dec;36(4):409–426. doi: 10.1007/BF02291366.
    1. Terwee CB, Bot SD, de Boer MR, van der Windt DA, Knol DL, Dekker J, Bouter LM, de Vet HC. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol. 2007 Jan;60(1):34–42. doi: 10.1016/j.jclinepi.2006.03.012.S0895-4356(06)00174-0
    1. Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, Bouter LM, de Vet HC. The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study. Qual Life Res. 2010 May;19(4):539–549. doi: 10.1007/s11136-010-9606-8.
    1. Buntrock C, Ebert DD, Lehr D, Cuijpers P, Riper H, Smit F, Berking M. Evaluating the efficacy and cost-effectiveness of web-based indicated prevention of major depression: design of a randomised controlled trial. BMC Psychiatry. 2014;14:25. doi: 10.1186/1471-244X-14-25. 1471-244X-14-25
    1. Buntrock C, Ebert D, Lehr D, Riper H, Smit F, Cuijpers P, Berking M. Effectiveness of a web-based cognitive behavioural intervention for subthreshold depression: pragmatic randomised controlled trial. Psychother Psychosom. 2015;84(6):348–358. doi: 10.1159/000438673.000438673
    1. Heber E, Lehr D, Ebert DD, Berking M, Riper H. Web-Based and Mobile Stress Management Intervention for Employees: A Randomized Controlled Trial. J Med Internet Res. 2016;18(1):e21. doi: 10.2196/jmir.5112. v18i1e21
    1. Heber E, Ebert DD, Lehr D, Nobis S, Berking M, Riper H. Efficacy and cost-effectiveness of a web-based and mobile stress-management intervention for employees: design of a randomized controlled trial. BMC Public Health. 2013;13:655. doi: 10.1186/1471-2458-13-655. 1471-2458-13-655
    1. Hautzinger M, Bailer M, Hofmeister D, Keller F. Allgemeine Depressionsskala (ADS) [Center for Epidemiological Studies Depression Scale (CES-D; Radloff, L.S., 1977) - German adaptation.] Psychiat Prax. 2012 Aug 31;39(06):302–304. doi: 10.1055/s-0032-1326702.
    1. Cohen S, Kamarck T, Mermelstein R. A global measure of perceived stress. J Health Soc Behav. 1983 Dec;24(4):385–396.
    1. Michalsen A, Jeitler M, Brunnhuber S, Lüdtke R, Büssing A, Musial F, Dobos G, Kessler C. Iyengar yoga for distressed women: a 3-armed randomized controlled trial. Evid Based Complement Alternat Med. 2012;2012:408727. doi: 10.1155/2012/408727. doi: 10.1155/2012/408727.
    1. Ladwig I, Rief W, Nestoriuc Y. Welche Risiken und Nebenwirkungen hat Psychotherapie? - Entwicklung des Inventars zur Erfassung Negativer Effekte von Psychotherapie (INEP) [What are the Risks and Side Effects to Psychotherapy? – Development of an Inventory for the Assessment of Negative Effects of Psychotherapy (INEP)] Verhaltenstherapie. 2014;24:252–263. doi: 10.1159/000367928.
    1. Rosseel Y. lavaan: An R Package for Structural Equation Modeling. J Stat Soft. 2012;48(2):1–36. doi: 10.18637/jss.v048.i02.
    1. Yuan KH, Bentler PM. Normal theory based test statistics in structural equation modelling. Br J Math Stat Psychol. 1998 Nov;51 ( Pt 2):289–309.
    1. Dunn TJ, Baguley T, Brunsden V. From alpha to omega: a practical solution to the pervasive problem of internal consistency estimation. Br J Psychol. 2014 Aug;105(3):399–412. doi: 10.1111/bjop.12046.
    1. Widaman K, Reise S. Exploring the measurement invariance of psychological instruments: Applications in the substance use domain. In: Bryant KJ, Windle M, West SG, editors. The science of prevention: Methodological advances from alcohol and substance abuse research. Washington, DC: American Psychological Association; 1997. pp. 281–324.
    1. Meredith W. Measurement invariance, factor analysis and factorial invariance. Psychometrika. 1993 Dec;58(4):525–543. doi: 10.1007/BF02294825.
    1. Chen FF. Sensitivity of Goodness of Fit Indexes to Lack of Measurement Invariance. Struct Equ Modeling. 2007 Jul 31;14(3):464–504. doi: 10.1080/10705510701301834.
    1. Jacobson NS, Truax P. Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. J Consult Clin Psychol. 1991 Feb;59(1):12–19.
    1. Fornell C, Larcker DF. Evaluating Structural Equation Models with Unobservable Variables and Measurement Error. J Mark Res. 1981 Feb;18(1):39. doi: 10.2307/3151312.
    1. McKinney V, Yoon K, Zahedi F. The measurement of web-customer satisfaction: An expectation and disconfirmation approach. Inf Syst Res. 2002;13(3):296–315.
    1. Liao C, Chen J, Yen DC. Theory of planning behavior (TPB) and customer satisfaction in the continued use of e-service: An integrated model. Comput Human Behav. 2007 Nov;23(6):2804–2822. doi: 10.1016/j.chb.2006.05.006.
    1. Hair JF, Black WC, Babin BJ, Anderson RE. Multivariate Data Analysis. 7th edition. New Jersey: Pearson Prentice Hall; 2010.
    1. Steenkamp J, Baumgartner H. Assessing Measurement Invariance in Cross‐National Consumer Research. J Consum Res. 1998 Jun;25(1):78–107. doi: 10.1086/209528.
    1. Thompson M, Green S. Evaluating between-group differences in latent variable means. In: Hancock GR, Mueller RO, editors. Structural equation modeling: A second course. Charlotte, NC: Information Age; 2006. pp. 119–169.
    1. Hannöver W, Dogs C, Kordy H. Patientenzufriedenheit - ein Maß für Behandlungserfolg? [Patient satisfaction - a measure for success of treatment?] Psychotherapeut. 2000;45(5):292–300.
    1. Ware JE, Hays RD. Methods for measuring patient satisfaction with specific medical encounters. Med Care. 1988 Apr;26(4):393–402.
    1. Crow R, Gage H, Hampson S, Hart J, Kimber A, Storey L, Thomas H. The measurement of satisfaction with healthcare: implications for practice from a systematic review of the literature. Health Technol Assess. 2002;6(32):1–244.
    1. Moret L, Nguyen J, Pillet N, Falissard B, Lombrail P, Gasquet I. Improvement of psychometric properties of a scale measuring inpatient satisfaction with care: a better response rate and a reduction of the ceiling effect. BMC Health Serv Res. 2007;7:197. doi: 10.1186/1472-6963-7-197. 1472-6963-7-197
    1. Lebow JL. Research assessing consumer satisfaction with mental health treatment: a review of findings. Eval Program Plann. 1983;6(3-4):211–236.

Source: PubMed

Sponsorer og samarbeidspartnere

Medisinsk tilstand

Legemiddelintervensjoner

3
Abonnere