Assessing the Usability of Complex Psychosocial Interventions: The Intervention Usability Scale

Aaron R Lyon, Michael D Pullmann, Jedediah Jacobson, Katie Osterhage, Morhaf Al Achkar, Brenna N Renn, Sean A Munson, Patricia A Areán, Aaron R Lyon, Michael D Pullmann, Jedediah Jacobson, Katie Osterhage, Morhaf Al Achkar, Brenna N Renn, Sean A Munson, Patricia A Areán

Abstract

Background: Usability - the extent to which an intervention can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction - may be a key determinant of implementation success. However, few instruments have been developed to measure the design quality of complex health interventions (i.e., those with several interacting components). This study evaluated the structural validity of the Intervention Usability Scale (IUS), an adapted version of the well-established System Usability Scale (SUS) for digital technologies, to measure the usability of a leading complex psychosocial intervention, Motivational Interviewing (MI), for behavioral health service delivery in primary care. Prior SUS studies have found both one- and two-factor solutions, both of which were examined in the current study of the IUS.

Method: A survey administered to 136 medical professionals from 11 primary care sites collected demographic information and IUS ratings for MI, the evidence-based psychosocial intervention that primary care providers reported using most often for behavioral health service delivery. Factor analyses replicated procedures used in prior research on the SUS.

Results: Analyses indicated that a two-factor solution (with "usable" and "learnable" subscales) best fit the data, accounting for 54.1% of the variance. Inter-item reliabilities for the total score, usable subscale, and learnable subscale were α = .83, α = .84, and α = .67, respectively.

Conclusions: This study provides evidence for a two-factor IUS structure consistent with some prior research, as well as acceptable reliability. Implications for implementation research evaluating the usability of complex health interventions are discussed, including the potential for future comparisons across multiple interventions and provider types, as well as the use of the IUS to evaluate the relationship between usability and implementation outcomes such as feasibility.

Keywords: complex health interventions; human-centered design; mental health; primary care; psychosocial interventions; usability.

Conflict of interest statement

Declaration of Conflicting Interests The Authors declare that there are no conflicts of interest.

References

    1. Alexopoulos G. S., Raue P. J., Gunning F., Kiosses D. N., Kanellopoulos D., Pollari C., Banerjee S., . . . Arean P. A. (2016). Engage therapy: Behavioral activation and improvement of late-life major depression. American Journal of Geriatric Psychiatry, 24(4), 320–326. 10.1016/j.jagp.2015.11.006
    1. Areán P. A., Ayalon L., Jin C., McCulloch C. E., Linkins K., Chen H., . . . Estes C. (2008). Integrated specialty mental health care among older minorities improves access but not outcomes: Results of the PRISMe study. International Journal of Geriatric Psychiatry: A Journal of the Psychiatry of Late Life and Allied Sciences, 23(10), 1086–1092. 10.1002/gps.2100
    1. Bangor A., Kortum P. T., Miller J. T. (2008). An empirical evaluation of the System Usability Scale. International Journal of Human-Computer Interaction, 24(6), 574–594. 10.1080/10447310802205776
    1. Brooke J. (1996). SUS: A quick and dirty usability scale. In Jordan P. W., Thomas B., McClelland I. L., Weerdmeester B. (Eds.), Usability evaluation in industry (pp. 189–194). CRC Press. London.
    1. Burchett H. E. D., Blanchard L., Kneale D., Thomas J. (2018). Assessing the applicability of public health intervention evaluations from one setting to another: A methodological study of the usability and usefulness of assessment tools and frameworks. Health Research Policy and Systems, 16(1), 88. 10.1186/s12961-018-0364-3
    1. Campbell N. C., Murray E., Darbyshire J., Emery J., Farmer A., Griffiths F., . . . Kinmonth A. L. (2007). Designing and evaluating complex interventions to improve health care. British Medical Journal, 334(7591), 455–459. 10.1136/
    1. Centers for Medicare & Medicaid Services. (2018). Behavioral health integration services booklet.
    1. Craig P., Dieppe P., Macintyre S., Michie S., Nazareth I., Petticrew M. (2013). Developing and evaluating complex interventions: The new Medical Research Council guidance. International Journal of Nursing Studies, 50(5), 585–587. 10.1016/j.ijnurstu.2012.09.010
    1. Dumas J. S., Dumas J. S., Redish J. (1999). A practical guide to usability testing. Intellect Books.
    1. Greenhalgh T., Robert G., Macfarlane F., Bate P., Kyriakidou O. (2004). Diffusion of innovations in service organizations: Systematic review and recommendations. The Milbank Quarterly, 82(4), 581–629. 10.1111/j.0887-378X.2004.00325.x
    1. Harris P. A., Taylor R., Thielke R., Payne J., Gonzalez N., Conde J. G. (2009). Research electronic data capture (REDCap)-A metadata-driven methodology and workflow process for providing translational research informatics support. Journal of Biomedical Informatics, 42(2), 377–381. 10.1016/j.jbi.2008.08.010
    1. Harte R., Glynn L., Rodríguez-Molinero A., Baker P. M., Scharf T., Quinlan L. R., ÓLaighin G. (2017). A human-centered design methodology to enhance the usability, human factors, and user experience of connected health systems: A three-phase methodology. JMIR Human Factors, 4(1), e8. 10.2196/humanfactors.5443
    1. Hettema J., Steele J., Miller W. R. (2005). Motivational interviewing. Annual Review of Clinical Psychology, 1, 91–111. 10.1146/annurev.clinpsy.1.102803.143833
    1. Horsky J., Schiff G. D., Johnston D., Mercincavage L., Bell D., Middleton B. (2012). Interface design principles for usable decision support: A targeted review of best practices for clinical prescribing interventions. Journal of Biomedical Informatics, 45(6), 1202–1216. 10.1016/j.jbi.2012.09.002
    1. Institute of Medicine. (2015). Psychosocial interventions for mental and substance use disorders: A framework for establishing evidence-based standards. National Academies Press.
    1. International Organization for Standardization. (1998). Ergonomic requirements for office work with visual display terminals (VDTs)—Part 11: Guidance on usability.
    1. International Organization for Standardization. (2019). 9241–210: Ergonomics of human system interaction-Part 210: Human-centered design for interactive systems.
    1. Kortum P., Peres S. C. (2014). The relationship between system effectiveness and subjective usability scores using the System Usability Scale. International Journal of Human-Computer Interaction, 30(7), 575–584. 10.1080/10447318.2014.904177
    1. Kortum P., Sorber M. (2015). Measuring the usability of mobile applications for phones and tablets. International Journal of Human-Computer Interaction, 31(8), 518–529. 10.1080/10447318.2015.1064658
    1. Kortum P. T., Bangor A. (2013). Usability ratings for everyday products measured with the System Usability Scale. International Journal of Human-Computer Interaction, 29(2), 67–76. 10.1080/10447318.2012.681221
    1. Landauer T. (1997). Behavioral research methods in human-computer interaction. In Handbook of human-computer interaction (pp. 203–227). 10.1016/B978-044481862-1.50075-3
    1. Lewis C. C., Fischer S., Weiner B. J., Stanick C., Kim M., Martinez R. G. (2015). Outcomes for implementation science: An enhanced systematic review of instruments using evidence-based rating criteria. Implementation Science, 10(1), 155. 10.1186/s13012-015-0342-x
    1. Lewis C. C., Stanick C. F., Martinez R. G., Weiner B. J., Kim M., Barwick M., Comtois K. A. (2015). The society for implementation research collaboration instrument review project: A methodology to promote rigorous evaluation. Implementation Science, 10(1), 2. 10.1186/s13012-014-0193-x
    1. Lewis J. R., Brown J., Mayes D. K. (2015). Psychometric evaluation of the EMO and the SUS in the context of a large-sample unmoderated usability study. International Journal of Human-Computer Interaction, 31(8), 545–553. 10.1080/10447318.2015.1064665
    1. Lewis J. R., Sauro J. (2017). Revisiting the factor structure of the System Usability Scale. Journal of Usability Studies, 12, 183–192.
    1. Lewis J. R., Utesch B. S., Maher D. E. (2013). UMUX—LITE—When there’s no time for the SUS. In Conference on human factors in computing systems—Proceedings (pp. 2099–2102). 10.1145/2470654.2481287
    1. Lewis J. R., Utesch B. S., Maher D. E. (2015). Measuring perceived usability: The SUS, UMUX-LITE, and AltUsability. International Journal of Human-Computer Interaction, 31(8), 496–505. 10.1080/10447318.2015.1064654
    1. Lyon A. R. (2016). Intervention Usability Scale (IUS). University of Washington.
    1. Lyon A. R., Brewer S. K., Areán P. A. (2020). Leveraging human-centered design to implement modern psychological science: Return on an early investment. American Psychologist, 75, 1067–1079. 10.1037/amp0000652
    1. Lyon A. R., Bruns E. J. (2019). User—centered redesign of evidence—based psychosocial interventions to enhance implementation—hospitable soil or better Seeds? JAMA Psychiatry, 761(1), 3–4. 10.1001/jamapsychiatry.2018.3060
    1. Lyon A. R., Koerner K., Chung J. (2020). Usability evaluation for evidence-based psychosocial interventions (USE-EBPI): A methodology for assessing complex intervention implementability. Implementation Research and Practice, 1, 1–17.
    1. Lyon A. R., Munson S. A., Renn B. N., Atkins D. C., Pullmann M. D., Friedman E., Areán P. A. (2019). Use of human-centered design to improve implementation of evidence-based psychotherapies in low-resource communities: Protocol for Studies Applying a Framework to Assess Usability. JMIR Research Protocols, 8(10), e14990. 10.2196/14990
    1. Mclellan S., Muddimer A., Peres S. C. (2012). The effect of experience on System Usability Scale ratings. Journal of Usability Studies, 7(2), 56–67.
    1. Miller W. R., Rollnick S. (2012). Motivational interviewing: Helping people change. Guilford Press.
    1. Nielsen J. (1994). Usability engineering. Morgan Kaufmann.
    1. Norman D. A., Draper S. W. (1986). User centered system design; new perspectives on human-computer interaction. Lawrence Erlbaum Associates.
    1. Nunnally J. C. (1978). Psychometric theory. McGraw-Hill.
    1. Perez Jolles M., Lengnick-Hall R., Mittman B. S. (2019). Core functions and forms of complex health interventions: A patient-centered medical home Illustration. Journal of General Internal Medicine, 34(6), 1032–1038. 10.1007/s11606-018-4818-7
    1. Rubin J., Chisnell D. (2008). Handbook of usability testing: How to plan, design, and conduct effective tests. Wiley.
    1. Sauro J. (2011). Does prior experience affect perceptions of usability?
    1. Sauro J., Lewis J. R. (2009, April 4–9). Correlations among prototypical usability metrics: Evidence for the construct of usability. In Proceedings of the 27th international conference on human factors in computing systems (pp. 1609–1618). ACM Press. 10.1145/1518701.1518947
    1. Sauro J., Lewis J. R. (2011). When designing usability questionnaires, does it hurt to be positive? In Conference on human factors in computing systems—Proceedings (pp. 2215–2223). 10.1145/1978942.1979266
    1. VanBuskirk K. A., Wetherell J. L. (2014). Motivational interviewing with primary care populations: A systematic review and meta-analysis. Journal of Behavioral Medicine, 37(4), 768–780.
    1. Weiner B. J., Lewis C. C., Stanick C., Powell B. J., Dorsey C. N., Clary A. S., . . . Halko H. (2017). Psychometric assessment of three newly developed implementation outcome measures. Implementation Science, 12(1), 1–12. 10.1186/s13012-017-0635-3
    1. World Health Organization Regional Office for Europe. (2016). Integrated care models: An overview Working document.

Source: PubMed

Sponsor e collaboratori

Condizioni mediche

Interventi farmacologici

3
Sottoscrivi