Usability Testing and Technology Acceptance of an mHealth App at the Point of Care During Simulated Pediatric In- and Out-of-Hospital Cardiopulmonary Resuscitations: Study Nested Within 2 Multicenter Randomized Controlled Trials

Abstract

Background: Mobile apps are increasingly being used in various domains of medicine. Few are evidence-based, and their benefits can only be achieved if end users intend to adopt and use them. To date, only a small fraction of mobile apps have published data on their field usability and end user acceptance results, especially in emergency medicine.

Objective: This study aims to determine the usability and acceptance of an evidence-based mobile app while safely preparing emergency drugs at the point of care during pediatric in- and out-of-hospital cardiopulmonary resuscitations by frontline caregivers.

Methods: In 2 multicenter randomized controlled parent trials conducted at 6 pediatric emergency departments from March 1 to December 31, 2017, and 14 emergency medical services from September 3, 2019, to January 21, 2020, the usability and technology acceptance of the PedAMINES (Pediatric Accurate Medication in Emergency Situations) app were evaluated among skilled pediatric emergency nurses and advanced paramedics when preparing continuous infusions of vasoactive drugs and direct intravenous emergency drugs at pediatric dosages during standardized, simulation-based, pediatric in- and out-of-hospital cardiac arrest scenarios, respectively. Usability was measured using the 10-item System Usability Scale. A 26-item technology acceptance self-administered survey (5-point Likert-type scales), adapted from the Unified Theory of Acceptance and Use of Technology model, was used to measure app acceptance and intention to use.

Results: All 100% (128/128) of nurses (crossover trial) and 49.3% (74/150) of paramedics (parallel trial) were assigned to the mobile app. Mean total scores on the System Usability Scale were excellent and reached 89.5 (SD 8.8; 95% CI 88.0-91.1) for nurses and 89.7 (SD 8.7; 95% CI 87.7-91.7) for paramedics. Acceptance of the technology was very good and rated on average >4.5/5 for 5 of the 8 independent constructs evaluated. Only the image construct scored between 3.2 and 3.5 by both participant populations.

Conclusions: The results provide evidence that dedicated mobile apps can be easy to use and highly accepted at the point of care during in- and out-of-hospital cardiopulmonary resuscitations by frontline emergency caregivers. These findings can contribute to the implementation and valorization of studies aimed at evaluating the usability and acceptance of mobile apps in the field by caregivers, even in critical situations.

Trial registration: ClinicalTrials.gov NCT03021122; https://ichgcp.net/clinical-trials-registry/NCT03021122. ClinicalTrials.gov NCT03921346; https://ichgcp.net/clinical-trials-registry/NCT03921346.

International registered report identifier (irrid): RR2-10.1186/s13063-019-3726-4.

Keywords: System Usability Scale; Unified Theory of Acceptance and Use of Technology; cardiopulmonary resuscitation; drugs; emergency medical services; medication errors; mobile apps; mobile health; mobile phone; out-of-hospital cardiac arrest; paramedics; pediatrics; smartphone.

Conflict of interest statement

Conflicts of Interest: Geneva University Hospitals are owners of the PedAMINES (Pediatric Accurate Medication in Emergency Situations) app. The app is currently commercially available on Google Play Store and Apple App Store for research and educational purposes. JNS, CL, AG, FE, and SM declare individual intellectual property rights on this app and, as employees of Geneva University Hospitals, indirect institutional rewarding through its commercialization (ie, without personal enrichment). The authors declare no other relationships or activities that could appear to have influenced the submitted work. All authors have completed the International Committee of Medical Journal Editors uniform disclosure form and declare no support from commercial entities for the submitted work and no financial relationships with any commercial entities that might have an interest in the submitted work in the previous 3 years.

©Johan N Siebert, Laëtitia Gosetto, Manon Sauvage, Laurie Bloudeau, Laurent Suppan, Frédérique Rodieux, Kevin Haddad, Florence Hugon, Alain Gervaix, Christian Lovis, Christophe Combescure, Sergio Manzano, Frederic Ehrler, PedAMINES Trial Group, PedAMINES Prehospital Group. Originally published in JMIR Human Factors (https://humanfactors.jmir.org), 01.03.2022.

Figures

Figure 1

Overall System Usability Scale (SUS) scores to assess the usability of the PedAMINES (Pediatric Accurate Medication in Emergency Situations) app. The SUS score is located on a normalized scale ranging from a minimum score of 0 to a maximum of 100 [27]. Adjective ratings provide an interpretation of the SUS score [53]. The SUS also provides letter grades, similar to those used in the traditional school grading system [54]. The acceptability ranges indicate whether the tool is acceptable or not. Red dots represent the mean SUS score in paramedics and blue dots in nurses. Capped blue and red lines represent the 5th and 95th percentiles. Crosses represent medians (paramedics: 92.5, 5th-95th percentiles: 74.125-100; nurses: 90, 5th-95th percentiles: 72.5-100).

Figure 2

Distribution of counts of System Usability Scale (SUS) total scores. Red dots denote paramedics; blue dots denote nurses.

Figure 3

Percent distribution of item responses by (A) paramedics (n=74) and (B) nurses (n=128) on the (inversed) System Usability Scale (SUS) items. The SUS comprises 10 items (numbered as SUS1 to SUS10).