A Remote Patient-Monitoring System for Intensive Care Medicine: Mixed Methods Human-Centered Design and Usability Evaluation

Akira-Sebastian Poncette, Lina Katharina Mosch, Lars Stablo, Claudia Spies, Monique Schieler, Steffen Weber-Carstens, Markus A Feufel, Felix Balzer, Akira-Sebastian Poncette, Lina Katharina Mosch, Lars Stablo, Claudia Spies, Monique Schieler, Steffen Weber-Carstens, Markus A Feufel, Felix Balzer

Abstract

Background: Continuous monitoring of vital signs is critical for ensuring patient safety in intensive care units (ICUs) and is becoming increasingly relevant in general wards. The effectiveness of health information technologies such as patient-monitoring systems is highly determined by usability, the lack of which can ultimately compromise patient safety. Usability problems can be identified and prevented by involving users (ie, clinicians).

Objective: In this study, we aim to apply a human-centered design approach to evaluate the usability of a remote patient-monitoring system user interface (UI) in the ICU context and conceptualize and evaluate design changes.

Methods: Following institutional review board approval (EA1/031/18), a formative evaluation of the monitoring UI was performed. Simulated use tests with think-aloud protocols were conducted with ICU staff (n=5), and the resulting qualitative data were analyzed using a deductive analytic approach. On the basis of the identified usability problems, we conceptualized informed design changes and applied them to develop an improved prototype of the monitoring UI. Comparing the UIs, we evaluated perceived usability using the System Usability Scale, performance efficiency with the normative path deviation, and effectiveness by measuring the task completion rate (n=5). Measures were tested for statistical significance using a 2-sample t test, Poisson regression with a generalized linear mixed-effects model, and the N-1 chi-square test. P<.05 were considered significant.

Results: We found 37 individual usability problems specific to monitoring UI, which could be assigned to six subcodes: usefulness of the system, response time, responsiveness, meaning of labels, function of UI elements, and navigation. Among user ideas and requirements for the UI were high usability, customizability, and the provision of audible alarm notifications. Changes in graphics and design were proposed to allow for better navigation, information retrieval, and spatial orientation. The UI was revised by creating a prototype with a more responsive design and changes regarding labeling and UI elements. Statistical analysis showed that perceived usability improved significantly (System Usability Scale design A: mean 68.5, SD 11.26, n=5; design B: mean 89, SD 4.87, n=5; P=.003), as did performance efficiency (normative path deviation design A: mean 8.8, SD 5.26, n=5; design B: mean 3.2, SD 3.03, n=5; P=.001), and effectiveness (design A: 18 trials, failed 7, 39% times, passed 11, 61% times; design B: 20 trials, failed 0 times, passed 20 times; P=.002).

Conclusions: Usability testing with think-aloud protocols led to a patient-monitoring UI with significantly improved usability, performance, and effectiveness. In the ICU work environment, difficult-to-use technology may result in detrimental outcomes for staff and patients. Technical devices should be designed to support efficient and effective work processes. Our results suggest that this can be achieved by applying basic human-centered design methods and principles.

Trial registration: ClinicalTrials.gov NCT03514173; https://ichgcp.net/clinical-trials-registry/NCT03514173.

Keywords: digital health; implementation science; intensive care medicine; intensive care unit; interview; mixed methods; mobile phone; patient monitoring; qualitative research; technological innovation; usability; user experience; user-centered design.

Conflict of interest statement

Conflicts of Interest: CS and FB report funding from Medtronic. FB also reports grants from German Federal Ministry of Education and Research, grants from German Federal Ministry of Health, grants from Berlin Institute of Health, personal fees from Elsevier Publishing, grants from Hans Böckler Foundation, other from Robert Koch Institute, grants from Einstein Foundation, and grants from Berlin University Alliance outside the submitted work.

©Akira-Sebastian Poncette, Lina Katharina Mosch, Lars Stablo, Claudia Spies, Monique Schieler, Steffen Weber-Carstens, Markus A Feufel, Felix Balzer. Originally published in JMIR Human Factors (https://humanfactors.jmir.org), 11.03.2022.

Figures

Figure 1
Figure 1
The research approach, beginning with usability testing and identification of major problems in design A, followed by prototyping of design B and its usability testing, concluding with a comparison between design A and design B.
Figure 2
Figure 2
Home screen of the implemented patient-monitoring platform (design A). etCO2: end-tidal carbon dioxide; PR: pulse rate; RR: respiratory rate; SPO2: peripheral capillary oxygen saturation [36-38].
Figure 3
Figure 3
Admit Patient screen of the implemented patient-monitoring platform (design A) [36-38].
Figure 4
Figure 4
Results of qualitative analysis of the think-aloud transcript. Three main codes were identified (inner ring) and subcoded (middle ring). The outer ring represents further information derived from the concrete items that were assigned to the subcodes (ie, specific user ideas or positive findings). UI: user interface.
Figure 5
Figure 5
Number of occurrences for each subcode of usability problems. Meaning of labels (n=53), usefulness (n=14), function of UI elements (n=8), navigation (n=8), responsiveness (n=3), response time (n=2). UI: user interface.
Figure 6
Figure 6
Redesign of the user interface of the prototype (design B) patient admission screen.
Figure 7
Figure 7
Redesign of the user interface of the prototype (design B) patient tile overview.
Figure 8
Figure 8
Scores of normative path deviation for design A and design B. The circle symbolizes outliers. Outliers are defined in the box plots as values that have 1.5 times the distance between Q1 and Q3 (Q1 is the lower line of the box, and Q3 is the upper line of the box).

References

    1. Moreno RP, Rhodes A, Donchin Y, European Society of Intensive Care Patient safety in intensive care medicine: the Declaration of Vienna. Intensive Care Med. 2009;35(10):1667–72. doi: 10.1007/s00134-009-1621-2.
    1. Hu P, Galvagno Jr SM, Sen A, Dutton R, Jordan S, Floccare D, Handley C, Shackelford S, Pasley J, Mackenzie C, ONPOINT Group Identification of dynamic prehospital changes with continuous vital signs acquisition. Air Med J. 2014;33(1):27–33. doi: 10.1016/j.amj.2013.09.003.S1067-991X(13)00242-3
    1. Balzer F, Habicher M, Sander M, Sterr J, Scholz S, Feldheiser A, Müller M, Perka C, Treskatsch S. Comparison of the non-invasive Nexfin® monitor with conventional methods for the measurement of arterial blood pressure in moderate risk orthopaedic surgery patients. J Int Med Res. 2016;44(4):832–43. doi: 10.1177/0300060516635383. 0300060516635383
    1. Shah PS. Wireless monitoring in the ICU on the horizon. Nat Med. 2020;26(3):316–7. doi: 10.1038/s41591-020-0798-3.10.1038/s41591-020-0798-3
    1. Michard F, Bellomo R, Taenzer A. The rise of ward monitoring: opportunities and challenges for critical care specialists. Intensive Care Med. 2019;45(5):671–3. doi: 10.1007/s00134-018-5384-5.10.1007/s00134-018-5384-5
    1. Dziadzko MA, Harrison AM, Tiong IC, Pickering BW, Moreno Franco P, Herasevich V. Testing modes of computerized sepsis alert notification delivery systems. BMC Med Inform Decis Mak. 2016;16(1):156. doi: 10.1186/s12911-016-0396-y. 10.1186/s12911-016-0396-y
    1. Meyer A, Zverinski D, Pfahringer B, Kempfert J, Kuehne T, Sündermann SH, Stamm C, Hofmann T, Falk V, Eickhoff C. Machine learning for real-time prediction of complications in critical care: a retrospective study. Lancet Respir Med. 2018;6(12):905–14. doi: 10.1016/S2213-2600(18)30300-X.S2213-2600(18)30300-X
    1. Bhatia M, Sood SK. Temporal informative analysis in smart-ICU monitoring: m-healthcare perspective. J Med Syst. 2016;40(8):190. doi: 10.1007/s10916-016-0547-9.10.1007/s10916-016-0547-9
    1. Yamada T, Vacas S, Gricourt Y, Cannesson M. Improving perioperative outcomes through minimally invasive and non-invasive hemodynamic monitoring techniques. Front Med (Lausanne) 2018;5:144. doi: 10.3389/fmed.2018.00144. doi: 10.3389/fmed.2018.00144.
    1. Poncette AS, Meske C, Mosch L, Balzer F. How to overcome barriers for the implementation of new information technologies in intensive care medicine. Proceedings of the 21st Human-Computer Interaction International Conference; HCII '19; July 26-31, 2019; Orlando, FL. 2019. pp. 534–46.
    1. Baig MM, GholamHosseini H, Moqeem AA, Mirza F, Lindén M. A systematic review of wearable patient monitoring systems - current challenges and opportunities for clinical adoption. J Med Syst. 2017;41(7):115. doi: 10.1007/s10916-017-0760-1.10.1007/s10916-017-0760-1
    1. Ross J, Stevenson F, Lau R, Murray E. Factors that influence the implementation of e-health: a systematic review of systematic reviews (an update) Implement Sci. 2016;11(1):146. doi: 10.1186/s13012-016-0510-7. 10.1186/s13012-016-0510-7
    1. Tscholl DW, Handschin L, Rössler J, Weiss M, Spahn DR, Nöthiger CB. It's not you, it's the design - common problems with patient monitoring reported by anesthesiologists: a mixed qualitative and quantitative study. BMC Anesthesiol. 2019;19(1):87. doi: 10.1186/s12871-019-0757-z. 10.1186/s12871-019-0757-z
    1. von Dincklage F, Suchodolski K, Lichtner G, Friesdorf W, Podtschaske B, Ragaller M. Investigation of the usability of computerized critical care information systems in Germany. J Intensive Care Med. 2019;34(3):227–37. doi: 10.1177/0885066617696848.
    1. Wade VA, Eliott JA, Hiller JE. Clinician acceptance is the key factor for sustainable telehealth services. Qual Health Res. 2014;24(5):682–94. doi: 10.1177/1049732314528809.1049732314528809
    1. Howe JL, Adams KT, Hettinger AZ, Ratwani RM. Electronic health record usability issues and potential contribution to patient harm. JAMA. 2018;319(12):1276–8. doi: 10.1001/jama.2018.1171. 2676098
    1. Fairbanks RJ, Caplan S. Poor interface design and lack of usability testing facilitate medical error. Jt Comm J Qual Saf. 2004;30(10):579–84. doi: 10.1016/s1549-3741(04)30068-7.S1549-3741(04)30068-7
    1. Middleton B, Bloomrosen M, Dente MA, Hashmat B, Koppel R, Overhage JM, Payne TH, Rosenbloom ST, Weaver C, Zhang J, American Medical Informatics Association Enhancing patient safety and quality of care by improving the usability of electronic health record systems: recommendations from AMIA. J Am Med Inform Assoc. 2013;20(e1):e2–8. doi: 10.1136/amiajnl-2012-001458. amiajnl-2012-001458
    1. Kushniruk AW, Triola MM, Borycki EM, Stein B, Kannry JL. Technology induced error and usability: the relationship between usability problems and prescription errors when using a handheld application. Int J Med Inform. 2005;74(7-8):519–26. doi: 10.1016/j.ijmedinf.2005.01.003.S1386-5056(05)00011-0
    1. Bitkina OV, Kim HK, Park J. Usability and user experience of medical devices: an overview of the current state, analysis methodologies, and future challenges. Int J Ind Ergon. 2020;76:102932. doi: 10.1016/j.ergon.2020.102932.
    1. Saeed N, Manzoor M, Khosravi P. An exploration of usability issues in telecare monitoring systems and possible solutions: a systematic literature review. Disabil Rehabil Assist Technol. 2020;15(3):271–81. doi: 10.1080/17483107.2019.1578998.
    1. Yen PY, Bakken S. Review of health information technology usability study methodologies. J Am Med Inform Assoc. 2012;19(3):413–22. doi: 10.1136/amiajnl-2010-000020. amiajnl-2010-000020
    1. Vincent CJ, Li Y, Blandford A. Integration of human factors and ergonomics during medical device design and development: it's all about communication. Appl Ergon. 2014;45(3):413–9. doi: 10.1016/j.apergo.2013.05.009.S0003-6870(13)00120-8
    1. Shah SG, Robinson I. Benefits of and barriers to involving users in medical device technology development and evaluation. Int J Technol Assess Health Care. 2007;23(1):131–7. doi: 10.1017/S0266462307051677.S0266462307051677
    1. Peischl B, Ferk M, Holzinger A. The fine art of user-centered software development. Software Qual J. 2015;23(3):509–36. doi: 10.1007/s11219-014-9239-1.
    1. Roberts JP, Fisher TR, Trowbridge MJ, Bent C. A design thinking framework for healthcare management and innovation. Healthc (Amst) 2016;4(1):11–4. doi: 10.1016/j.hjdsi.2015.12.002.S2213-0764(15)00113-X
    1. Johnson CM, Johnson TR, Zhang J. A user-centered framework for redesigning health care interfaces. J Biomed Inform. 2005;38(1):75–87. doi: 10.1016/j.jbi.2004.11.005. S1532-0464(04)00153-4
    1. Fidler R, Bond R, Finlay D, Guldenring D, Gallagher A, Pelter M, Drew B, Hu X. Human factors approach to evaluate the user interface of physiologic monitoring. J Electrocardiol. 2015;48(6):982–7. doi: 10.1016/j.jelectrocard.2015.08.032.S0022-0736(15)00289-7
    1. Wiggermann N, Rempel K, Zerhusen RM, Pelo T, Mann N. Human-centered design process for a hospital bed: promoting patient safety and ease of use. Ergon Des. 2019;27(2):4–12. doi: 10.1177/1064804618805570.
    1. Applying human factors and usability engineering to medical devices: guidance for industry and Food and Drug Administration staff. U.S. Food and Drug Administration. 2016. [2022-03-03]. .
    1. Alexander G, Staggers N. A systematic review of the designs of clinical technology: findings and recommendations for future research. ANS Adv Nurs Sci. 2009;32(3):252–79. doi: 10.1097/ANS.0b013e3181b0d737. 00012272-200907000-00007
    1. Bazzano AN, Martin J, Hicks E, Faughnan M, Murphy L. Human-centred design in global health: a scoping review of applications and contexts. PLoS One. 2017;12(11):e0186744. doi: 10.1371/journal.pone.0186744. PONE-D-16-37632
    1. Schmettow M, Schnittker R, Schraagen JM. An extended protocol for usability validation of medical devices: research design and reference model. J Biomed Inform. 2017;69:99–114. doi: 10.1016/j.jbi.2017.03.010. S1532-0464(17)30059-X
    1. Daniels J, Fels S, Kushniruk A, Lim J, Ansermino JM. A framework for evaluating usability of clinical monitoring technology. J Clin Monit Comput. 2007;21(5):323–30. doi: 10.1007/s10877-007-9091-y.
    1. Gravetter FJ, Forzanno LB. Research methods for the behavioral sciences. 4th edition. Belmont, CA: Wadsworth Publishing; 2012.
    1. Quick guide for clinician users: vital SyncTM virtual patient monitoring platform. Medtronic. 2017. [2022-03-03]. .
    1. Vital SyncTM virtual patient monitoring platform: user guide. Medtronic. 2017. [2022-03-03]. .
    1. Health informatics and monitoring: vital SyncTM virtual patient monitoring platform. Medtronic. [2019-01-15]. .
    1. Ergonomie der Mensch-System-Interaktion - Methoden zur Gewährleistung der Gebrauchstauglichkeit, die eine benutzer-orientierte Gestaltung unterstützen. Beuth Publishing DIN. 2002. [2022-03-03]. .
    1. Fonteyn ME, Kuipers B, Grobe SJ. A description of think aloud method and protocol analysis. Qual Health Res. 1993;3(4):430–41. doi: 10.1177/104973239300300403.
    1. Wiklund ME, Kendler J, Strochlic AY. Selecting tasks. In: Wilkund ME, Kendler J, Strochlic AY, editors. Usability testing of medical devices. 2nd edition. Boca Raton, FL: CRC Press; 2016. pp. 261–316.
    1. Bangor A, Kortum PT, Miller JT. An empirical evaluation of the system usability scale. Int J Hum Comput Interact. 2008;24(6):574–94. doi: 10.1080/10447310802205776.
    1. Lewis JR. The system usability scale: past, present, and future. Int J Hum-Comput Interact. 2018;34(7):577–90. doi: 10.1080/10447318.2018.1455307.
    1. Kushniruk AW, Patel VL. Cognitive and usability engineering methods for the evaluation of clinical information systems. J Biomed Inform. 2004;37(1):56–76. doi: 10.1016/j.jbi.2004.01.003. S1532046404000206
    1. Sauro J, Lewis J. Quantifying user research. In: Sauro J, Lewis J, editors. Quantifying the user experience: practical statistics of user research. Cambridge, MA: Morgan Kaufmann; 2016. pp. 9–18.
    1. Rubin J, Chisnell D, Spool J. Handbook of usability testing: how to plan, design, and conduct effective tests. 2nd edition. Indianapolis, IN: Wiley; 2008.
    1. Broekhuis M, van Velsen L, Hermens H. Assessing usability of eHealth technology: a comparison of usability benchmarking instruments. Int J Med Inform. 2019;128:24–31. doi: 10.1016/j.ijmedinf.2019.05.001.S1386-5056(19)30067-X
    1. Nielsen J, Budiu R. Success rate: the simplest usability metric. Nielsen Norman Group. 2001. [2019-11-06].
    1. Schnittker R, Schmettow M, Verhoeven F, Schraagen JM. Combining situated Cognitive Engineering with a novel testing method in a case study comparing two infusion pump interfaces. Appl Ergon. 2016;55:16–26. doi: 10.1016/j.apergo.2016.01.004.S0003-6870(16)30004-7
    1. Albert B, Tullis T. Measuring the user experience: collecting, analyzing, and presenting usability metrics. 2nd edition. Amsterdam, The Netherlands: Morgan Kaufmann; 2013.
    1. Shapiro SS, Wilk MB. An analysis of variance test for normality (complete samples) Biometrika. 1965;52(3/4):591–611. doi: 10.2307/2333709.
    1. Levene H. Robust tests for equality of variances. In: Olkin I, editor. Contributions to probability and statistics: essays in honor of Harold hotelling. Redwood City, CA: Stanford University Press; 1960. pp. 278–92.
    1. Poncette AS, Mosch L, Spies C, Schmieding M, Schiefenhövel F, Krampe H, Balzer F. Improvements in patient monitoring in the intensive care unit: survey study. J Med Internet Res. 2020;22(6):e19091. doi: 10.2196/19091. v22i6e19091
    1. Poncette AS, Spies C, Mosch L, Schieler M, Weber-Carstens S, Krampe H, Balzer F. Clinical requirements of future patient monitoring in the intensive care unit: qualitative study. JMIR Med Inform. 2019;7(2):e13064. doi: 10.2196/13064. v7i2e13064
    1. Turner P, Kushniruk A, Nohr C. Are we there yet? Human factors knowledge and health information technology - the challenges of implementation and impact. Yearb Med Inform. 2017;26(1):84–91. doi: 10.15265/IY-2017-014.
    1. Labeling: regulatory requirements for medical devices. U.S. Food & Drug Administration. 1997. [2022-03-03]. .
    1. Roman LC, Ancker JS, Johnson SB, Senathirajah Y. Navigation in the electronic health record: a review of the safety and usability literature. J Biomed Inform. 2017;67:69–79. doi: 10.1016/j.jbi.2017.01.005. S1532-0464(17)30005-9
    1. Moradian S, Krzyzanowska MK, Maguire R, Morita PP, Kukreti V, Avery J, Liu G, Cafazzo J, Howell D. Usability evaluation of a mobile phone-based system for remote monitoring and management of chemotherapy-related side effects in cancer patients: mixed-methods study. JMIR Cancer. 2018;4(2):e10932. doi: 10.2196/10932. v4i2e10932
    1. Andrade E, Quinlan L, Harte R, Byrne D, Fallon E, Kelly M, Casey S, Kirrane F, O'Connor P, O'Hora D, Scully M, Laffey J, Pladys P, Beuchée A, ÓLaighin G. Novel interface designs for patient monitoring applications in critical care medicine: human factors review. JMIR Hum Factors. 2020;7(3):e15052. doi: 10.2196/15052. v7i3e15052
    1. Flohr L, Beaudry S, Johnson KT, West N, Burns CM, Ansermino JM, Dumont GA, Wensley D, Skippen P, Gorges M. Clinician-driven design of VitalPAD - an intelligent monitoring and communication device to improve patient safety in the intensive care unit. IEEE J Transl Eng Health Med. 2018;6:3000114. doi: 10.1109/JTEHM.2018.2812162. 3000114
    1. Faiola A, Srinivas P, Hillier S. Improving patient safety: integrating data visualization and communication into ICU workflow to reduce cognitive load. Proc Int Symp Hum Factors Ergon Healthc. 2015;4(1):55–61. doi: 10.1177/2327857915041013.
    1. Faiola A, Srinivas P, Duke J. Supporting clinical cognition: a human-centered approach to a novel ICU information visualization dashboard. AMIA Annu Symp Proc. 2015;2015:560–9.
    1. Chuang CH, Tseng PC, Lin CY, Lin KH, Chen YY. Burnout in the intensive care unit professionals: a systematic review. Medicine (Baltimore) 2016;95(50):e5629. doi: 10.1097/MD.0000000000005629. doi: 10.1097/MD.0000000000005629.00005792-201612160-00037
    1. Richardsen AM, Martinussen M, Kaiser S. Stress, human errors and accidents. In: Burke RJ, Richardsen AM, editors. Increasing occupational health and safety workplaces: individual, work and organizational factors. Cheltenham, UK: Edward Elgar Publishing; 2019. pp. 48–67.
    1. Kumar A, Pore P, Gupta S, Wani AO. Level of stress and its determinants among intensive care unit staff. Indian J Occup Environ Med. 2016;20(3):129–32. doi: 10.4103/0019-5278.203137. IJOEM-20-129
    1. Graber ML, Kissam S, Payne VL, Meyer AN, Sorensen A, Lenfestey N, Tant E, Henriksen K, Labresh K, Singh H. Cognitive interventions to reduce diagnostic error: a narrative review. BMJ Qual Saf. 2012;21(7):535–57. doi: 10.1136/bmjqs-2011-000149.bmjqs-2011-000149
    1. Kahn JM. What we talk about when we talk about intensive care unit strain. Ann Am Thorac Soc. 2014;11(2):219–20. doi: 10.1513/AnnalsATS.201311-406ED.
    1. The Topol review: preparing the healthcare workforce to deliver the digital future: an independent report on behalf of the Secretary of State for Health and Social Care. National Health Service. 2019. [2022-03-03]. .
    1. Gonzalez Velasco JM. Complex and transdisciplinary strategies for promotion and prevention in digital health: towards the ecology of knowledge. In: Mantel-Teeuwisse A, Khatri B, Uzman N, Mellianti S, editors. FIP Digital health in pharmacy education: developing a digitally enabled pharmaceutical workforce. The Hague, The Netherlands: International Pharmaceutical Federation; 2021. pp. 106–8.
    1. Machleid F, Kaczmarczyk R, Johann D, Balčiūnas J, Atienza-Carbonell B, von Maltzahn F, Mosch L. Perceptions of digital health education among European medical students: mixed methods survey. J Med Internet Res. 2020;22(8):e19827. doi: 10.2196/19827. v22i8e19827
    1. Matern U. Design, usability and staff training - what is more important? In: Doffy VG, editor. Advances in human aspects of healthcare. Boca Raton, FL: CRC Press; 2012. pp. 426–8.
    1. Poncette AS, Glauert DL, Mosch L, Braune K, Balzer F, Back DA. Undergraduate medical competencies in digital health and curricular module development: mixed methods study. J Med Internet Res. 2020;22(10):e22161. doi: 10.2196/22161. v22i10e22161
    1. Holeman I, Kane D. Human-centered design for global health equity. Inf Technol Dev. 2019;26(3):477–505. doi: 10.1080/02681102.2019.1667289. 1667289
    1. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50. doi: 10.1186/1748-5908-4-50. 1748-5908-4-50
    1. Laboratories: human factors engineering: Susan Hallbeck. Mayo Clinic. [2020-05-09]. .

Source: PubMed

Sponsors and Collaborators

Medical Conditions

Drug Interventions

3
Subscribe