A Questionnaire for Assessing User Satisfaction With Mobile Health Apps: Development Using Rasch Measurement Theory

Jeanette Melin, Stephanie Erika Bonn, Leslie Pendrill, Ylva Trolle Lagerros, Jeanette Melin, Stephanie Erika Bonn, Leslie Pendrill, Ylva Trolle Lagerros

Abstract

Background: Mobile health (mHealth) apps offer great opportunities to deliver large-scale, cost-efficient digital solutions for implementing lifestyle changes. Furthermore, many mHealth apps act as medical devices. Yet, there is little research on how to assess user satisfaction with an mHealth solution.

Objective: This study presents the development of the mHealth Satisfaction Questionnaire and evaluates its measurement properties.

Methods: Respondents who took part in the Health Integrator Study and were randomized to use the Health Integrator smartphone app for lifestyle changes (n=112), with and without additional telephone coaching, rated their satisfaction with the app using the new 14-item mHealth Satisfaction Questionnaire. The ratings were given on a 5-point Likert scale and measurement properties were evaluated using Rasch measurement theory (RMT).

Results: Optimal scoring was reached when response options 2, 3, and 4 were collapsed, giving three response categories. After omitting two items that did not fit into the scale, fit residuals were within, or close to, the recommended range of ±2.5. There was no differential item functioning between intervention group, age group, or sex. The Person Separation Index was 0.79, indicating that the scale's ability to discriminate correctly between person leniency was acceptable for group comparisons but not for individual evaluations. The scale did not meet the criterion of unidimensionality; 16.1% (18/112) of the respondents were outside the desired range of -1.96 to 1.96. In addition, several items showed local dependency and three underlying dimensions emerged: negative experiences, positive experiences, and lifestyle consequences of using the mHealth solution.

Conclusions: In times where mHealth apps and digital solutions are given more attention, the mHealth Satisfaction Questionnaire provides a new possibility to measure user satisfaction to ensure usability and improve development of new apps. Our study is one of only a few cases where RMT has been used to evaluate the usability of such an instrument. There is, though, a need for further development of the mHealth Satisfaction Questionnaire, including the addition of more items and consideration of further response options. The mHealth Satisfaction Questionnaire should also be evaluated in a larger sample and with other mHealth apps and in other contexts.

Trial registration: ClinicalTrials.gov NCT03579342; https://ichgcp.net/clinical-trials-registry/NCT03579342.

Keywords: cell phone; healthy lifestyle; methods; mobile applications; mobile phone; psychometrics; smartphone; telemedicine.

Conflict of interest statement

Conflicts of Interest: None declared.

©Jeanette Melin, Stephanie Erika Bonn, Leslie Pendrill, Ylva Trolle Lagerros. Originally published in JMIR mHealth and uHealth (http://mhealth.jmir.org), 26.05.2020.

Figures

Figure 1
Figure 1
Screenshot of the Web-based mHealth Satisfaction Questionnaire.
Figure 2
Figure 2
Item characteristic curve (ICC) showing a line with the expected response (predicted from the model) and the dots corresponding to the observed response. The illustration shows how the dots deviated from the ICC for the item It has been difficult to remember to use it.
Figure 3
Figure 3
Item characteristic curve (ICC) showing a line with the expected response (predicted from the model) and the dots corresponding to the observed response. The illustration shows how the dots deviated from the ICC for the item It interrupted me in my daily activities.
Figure 4
Figure 4
Person-item threshold histograms for the 12-item version. Upper histogram (pink bars) shows person measurements reflecting lower leniency with the Health Integrator app to the left, and higher leniency with the Health Integrator app to the right, that is, the most lenient persons are to the right. The lower histogram (blue bars) shows item threshold estimates reflecting lower quality demands to the left and higher quality demands to the right. This implies that it was easier to agree to statements at the lower end corresponding to negative experiences of using the mobile health (mHealth) app compared with the less easy items to agree with at the upper end, corresponding to lifestyle consequences of using the mHealth app.

References

    1. World Health Organization . mHealth: New Horizons for Health Through Mobile Technologies. Geneva: World Health Organization; 2011.
    1. Druce KL, Dixon WG, McBeth J. Maximizing engagement in mobile health studies: lessons learned and future directions. Rheum Dis Clin North Am. 2019 May;45(2):159–72. doi: 10.1016/j.rdc.2019.01.004.
    1. King AC, Hekler EB, Grieco LA, Winter SJ, Sheats JL, Buman MP, Banerjee B, Robinson TN, Cirimele J. Harnessing different motivational frames via mobile phones to promote daily physical activity and reduce sedentary behavior in aging adults. PLoS One. 2013;8(4):e62613. doi: 10.1371/journal.pone.0062613.
    1. Mummah SA, Mathur M, King AC, Gardner CD, Sutton S. Mobile technology for vegetable consumption: a randomized controlled pilot study in overweight adults. JMIR Mhealth Uhealth. 2016 May 18;4(2):e51. doi: 10.2196/mhealth.5146.
    1. Bonn SE, Löf M, Östenson CG, Lagerros YT. App-technology to improve lifestyle behaviors among working adults - the Health Integrator study, a randomized controlled trial. BMC Public Health. 2019 Mar 7;19(1):273. doi: 10.1186/s12889-019-6595-6.
    1. Pendrill LR. Quality Assured Measurement: Unification Across Social and Physical Sciences. Springer, Cham: Springer; 2019.
    1. Wright BD, Linacre JM. Observations are always ordinal; measurements, however, must be interval. Arch Phys Med Rehabil. 1989 Nov;70(12):857–60.
    1. Rasch G. Probabilistic Models for Some Intelligence and Attainment Tests. Chicago: MESA Press; 1993.
    1. Pendrill L. Man as a measurement instrument. NCSLI Meas. 2014;9(4):24–35. doi: 10.1080/19315775.2014.11721702.
    1. Cano SJ, Pendrill LR, Barbic SP, Fisher WP. Patient-Centred Outcome Metrology for Healthcare Decision-Making. Proceedings of the Joint IMEKO TC1-TC7-TC13 Symposium: Measurement Science Challenges in Natural and Social Sciences; IMEKO'17; July 31-August 3, 2017; Rio de Janeiro, Brazil. 2017.
    1. Andrich D, Sheridan B, Luo G. Interpreting RUMM2030. Perth: RUMM Laboratory Pty Ltd; 2009.
    1. Hobart J, Cano S. Improving the evaluation of therapeutic interventions in multiple sclerosis: the role of new psychometric methods. Health Technol Assess. 2009 Feb;13(12):iii, ix–x, 1. doi: 10.3310/hta13120. doi: 10.3310/hta13120.
    1. Tennant A, Conaghan PG. The Rasch measurement model in rheumatology: what is it and why use it? When should it be applied, and what should one look for in a Rasch paper? Arthritis Rheum. 2007 Dec 15;57(8):1358–62. doi: 10.1002/art.23108. doi: 10.1002/art.23108.
    1. Nunnally JC, Bernstein IH. Psychometric Theory. Third Edition. New York, New York: McGraw-Hill; 1994.
    1. Pendrill LR, Fisher WP. Counting and quantification: comparing psychometric and metrological perspectives on visual perceptions of number. Meas. 2015 Jul;71:46–55. doi: 10.1016/j.measurement.2015.04.010. doi: 10.1016/j.measurement.2015.04.010.
    1. Hagquist C, Andrich D. Recent advances in analysis of differential item functioning in health research using the Rasch model. Health Qual Life Outcomes. 2017 Sep 19;15(1):181. doi: 10.1186/s12955-017-0755-0.
    1. Christensen KB, Makransky G, Horton M. Critical values for Yen's : identification of local dependence in the Rasch model using residual correlations. Appl Psychol Meas. 2017 May;41(3):178–94. doi: 10.1177/0146621616677520.
    1. Marais I. Local dependence. In: Christensen KB, Mesbah M, Kreiner S, editors. Rasch Models in Health. London, UK: John Wiley & Sons, Inc; 2013.
    1. Marais I, Andrich D. Formalizing dimension and response violations of local independence in the unidimensional Rasch model. J Appl Meas. 2008;9(3):200–15.
    1. Smith RM. A comparison of methods for determining dimensionality in Rasch measurement. Struct Equ Modeling. 1996 Jan;3(1):25–40. doi: 10.1080/10705519609540027.
    1. Andrich D. A rating formulation for ordered response categories. Psychometrika. 1978 Dec;43(4):561–73. doi: 10.1007/bf02293814.
    1. Chen W, Lenderking W, Jin Y, Wyrwich KW, Gelhorn H, Revicki DA. Is Rasch model analysis applicable in small sample size pilot studies for assessing item characteristics? An example using PROMIS pain behavior item bank data. Qual Life Res. 2014 Mar;23(2):485–93. doi: 10.1007/s11136-013-0487-5.
    1. Morel T, Cano SJ. Measuring what matters to rare disease patients - reflections on the work by the IRDiRC taskforce on patient-centered outcome measures. Orphanet J Rare Dis. 2017 Nov 2;12(1):171. doi: 10.1186/s13023-017-0718-x.
    1. Cleanthous S, Barbic SP, Smith S, Regnault A. Psychometric performance of the PROMIS depression item bank: a comparison of the 28- and 51-item versions using Rasch measurement theory. J Patient Rep Outcomes. 2019 Jul 30;3(1):47. doi: 10.1186/s41687-019-0131-4.
    1. Spearman C. Correlation calculated from faulty data. Br J Psychol. 1910;3(3):271–95. doi: 10.1111/j.2044-8295.1910.tb00206.x.
    1. Nickell GS, Pinto JN. The computer attitude scale. Comput Hum Behav. 1986 Jan;2(4):301–6. doi: 10.1016/0747-5632(86)90010-5.
    1. International Organization for Standardization. [2019-08-01]. ISO 9241-11:1998 Ergonomic Requirements for Office Work With Visual Display Terminals (VDTs) — Part 11: Guidance on Usability .
    1. International Organization for Standardization. [2019-08-01]. IEC 62366-1:2015 Medical Devices — Part 1: Application of Usability Engineering to Medical Devices .
    1. ‎The Government of Sweden. 2011. [2019-08-01]. National eHealth - The Strategy for Accessible and Secure Information in Health and Social Care .
    1. Food and Drug Administration. 2016. [2019-08-01]. Applying Human Factors and Usability Engineering to Medical Devices .
    1. Cano S, Hobart J. The problem with health measurement. Patient Prefer Adherence. 2011 Jun;2011(5):279–90. doi: 10.2147/ppa.s14399.
    1. Leung Y, Png M, Conaghan P, Tennant A. A systematic literature review on the application of Rasch analysis in musculoskeletal disease -- a special interest group report of OMERACT 11. J Rheumatol. 2014 Jan;41(1):159–64. doi: 10.3899/jrheum.130814.
    1. Patel AS, Siegert RJ, Creamer D, Larkin G, Maher TM, Renzoni EA, Wells AU, Higginson IJ, Birring SS. The development and validation of the King's Sarcoidosis Questionnaire for the assessment of health status. Thorax. 2013 Jan;68(1):57–65. doi: 10.1136/thoraxjnl-2012-201962.
    1. Tezza R, Bornia AC, de Andrade DF. Measuring web usability using item response theory: principles, features and opportunities. Interact Comput. 2011 Mar;23(2):167–75. doi: 10.1016/j.intcom.2011.02.004.
    1. Rice SK, Pendrill L, Petersson N, Nordlinder J, Farbrot A. Rationale and design of a novel method to assess the usability of body-worn absorbent incontinence care products by caregivers. J Wound Ostomy Continence Nurs. 2018;45(5):456–64. doi: 10.1097/WON.0000000000000462.

Source: PubMed

스폰서 및 공동 작업자

건강 상태

약물 개입

3
구독하다