Assessing validity evidence for a serious game dedicated to patient clinical deterioration and communication

Antonia Blanié, Michel-Ange Amorim, Arnaud Meffert, Corinne Perrot, Lydie Dondelli, Dan Benhamou, Antonia Blanié, Michel-Ange Amorim, Arnaud Meffert, Corinne Perrot, Lydie Dondelli, Dan Benhamou

Abstract

Background: A serious game (SG) is a useful tool for nurse training. The objectives of this study were to assess validity evidence of a new SG designed to improve nurses' ability to detect patient clinical deterioration.

Methods: The SG (LabForGames Warning) was developed through interaction between clinical and pedagogical experts and one developer. For the game study, consenting nurses were divided into three groups: nursing students (pre-graduate) (group S), recently graduated nurses (graduated < 2 years before the study) (group R) and expert nurses (graduated > 4 years before the study and working in an ICU) (group E). Each volunteer played three cases of the game (haemorrhage, brain trauma and obstructed intestinal tract). The validity evidence was assessed following Messick's framework: content, response process (questionnaire, observational analysis), internal structure, relations to other variables (by scoring each case and measuring playing time) and consequences (a posteriori analysis).

Results: The content validity was supported by the game design produced by clinical, pedagogical and interprofessional experts in accordance with the French nurse training curriculum, literature review and pilot testing. Seventy-one nurses participated in the study: S (n = 25), R (n = 25) and E (n = 21). The content validity in all three cases was highly valued by group E. The response process evidence was supported by good security control. There was no significant difference in the three groups' high rating of the game's realism, satisfaction and educational value. All participants stated that their knowledge of the different steps of the clinical reasoning process had improved. Regarding the internal structure, the factor analysis showed a common source of variance between the steps of the clinical reasoning process and communication or the situational awareness errors made predominantly by students. No statistical difference was observed between groups regarding scores and playing time. A posteriori analysis of the results of final examinations assessing study-related topics found no significant difference between group S participants and students who did not participate in the study.

Conclusion: While it appears that this SG cannot be used for summative assessment (score validity undemonstrated), it is positively valued as an educational tool.

Trial registration: ClinicalTrials.gov ID: NCT03092440.

Keywords: Patient deterioration; Serious game; Simulation; Validity evidence.

Conflict of interest statement

Competing interestsThe authors declare that they have no competing interests.

© The Author(s) 2020.

Figures

Fig. 1
Fig. 1
Screenshots of LabForGames Warning
Fig. 2
Fig. 2
Links between errors (communication and situational awareness) and the clinical reasoning process as demonstrated by principal component factor analysis

References

    1. Buist M, Bernard S, Nguyen TV, Moore G, Anderson J. Association between clinically abnormal observations and subsequent in-hospital mortality: a prospective study. Resuscitation. 2004;62(2):137–141.
    1. Hillman KM, Bristow PJ, Chey T, et al. Duration of life-threatening antecedents prior to intensive care admission. Intensive Care Med. 2002;28(11):1629–1634.
    1. Ghaferi AA, Birkmeyer JD, Dimick JB. Complications, failure to rescue, and mortality with major inpatient surgery in medicare patients. Ann Surg. 2009;250(6):1029–1034.
    1. De Meester K, Verspuy M, Monsieurs KG, Van Bogaert P. SBAR improves nurse-physician communication and reduces unexpected death: a pre and post intervention study. Resuscitation. 2013;84(9):1192–1196.
    1. .
    1. Nagpal K, Arora S, Vats A, et al. Failures in communication and information transfer across the surgical care pathway: interview study. BMJ Qual Saf. 2012;21(10):843–849.
    1. Mackintosh N, Sandall J. Overcoming gendered and professional hierarchies in order to facilitate escalation of care in emergency situations: the role of standardised communication protocols. Soc Sci Med. 2010;71(9):1683–1686.
    1. Liaw SY, Chan SW, Chen FG, Hooi SC, Siau C. Comparison of virtual patient simulation with mannequin-based simulation for improving clinical performances in assessing and managing clinical deterioration: randomized controlled trial. J Med Internet Res. 2014;16(9):e214.
    1. Haerling KA. Cost-utility analysis of virtual and mannequin-based simulation. Simul Healthc. 2018;13(1):33–40.
    1. Graafland M, Schraagen JM, Schijven MP. Systematic review of serious games for medical education and surgical skills training. Br J Surg. 2012;99(10):1322–1330.
    1. Gentry SV, Gauthier A, L’Estrade Ehrstrom B, et al. Serious gaming and gamification education in health professions: systematic review. J Med Internet Res. 2019;21(3):e12994.
    1. Gorbanev I, Agudelo-Londono S, Gonzalez RA, et al. A systematic review of serious games in medical education: quality of evidence and pedagogical strategy. Med Educ Online. 2018;23(1):1438718.
    1. Blanie A, Amorim MA, Benhamou D. Comparative value of a simulation by gaming and a traditional teaching method to improve clinical reasoning skills necessary to detect patient deterioration: a randomized study in nursing students. BMC Med Educ. 2020;20(1):53.
    1. Cook DA, Hatala R. Validation of educational assessments: a primer for simulation and beyond. Adv Simul (Lond). 2016;1:31.
    1. Graafland M, Dankbaar M, Mert A, et al. How to systematically assess serious games applied to health care. JMIR Serious Games. 2014;2(2):e11.
    1. Gao H, McDonnell A, Harrison DA, et al. Systematic review and evaluation of physiological track and trigger warning systems for identifying at-risk patients on the ward. Intensive Care Med. 2007;33(4):667–679.
    1. Legifrance.gouv. = JORFTEXT000037218115&categorieLien = id. Accessed 03-17, 2020.
    1. Cook DA, Brydges R, Zendejas B, Hamstra SJ, Hatala R. Technology-enhanced simulation to assess health professionals: a systematic review of validity evidence, research methods, and reporting quality. Acad Med. 2013;88(6):872–883.
    1. Cook DA, Hatala R, Brydges R, et al. Technology-enhanced simulation for health professions education: a systematic review and meta-analysis. JAMA. 2011;306(9):978–988.
    1. Gerard JM, Scalzo AJ, Borgman MA, et al. Validity evidence for a serious game to assess performance on critical pediatric emergency medicine scenarios. Simul Healthc. 2018;13(3):168–180.
    1. Borgersen NJ, Naur TMH, Sorensen SMD, et al. Gathering validity evidence for surgical simulation: a systematic review. Ann Surg. 2018;267(6):1063–1068.
    1. Cook DA, Zendejas B, Hamstra SJ, Hatala R, Brydges R. What counts as validity evidence? Examples and prevalence in a systematic review of simulation-based assessment. Adv Health Sci Educ Theory Pract. 2014;19(2):233–250.
    1. Levett-Jones T, Hoffman K, Dempsey J, et al. The ‘five rights’ of clinical reasoning: an educational model to enhance nursing students' ability to identify and manage clinically ‘at risk’ patients. Nurse Educ Today. 2010;30(6):515–520.
    1. Koivisto JM, Multisilta J, Niemi H, Katajisto J, Eriksson E. Learning by playing: a cross-sectional descriptive study of nursing students' experiences of learning clinical reasoning. Nurse Educ Today. 2016;45:22–28.
    1. Biostatgv. https:///. Accessed 03-17, 2020.
    1. Cheng A, Kessler D, Mackinnon R, et al. Reporting guidelines for health care simulation research: extensions to the CONSORT and STROBE statements. Adv Simul (Lond). 2016;1:25.
    1. Phillips AC, Lewis LK, McEvoy MP, et al. Development and validation of the guideline for reporting evidence-based practice educational interventions and teaching (GREET) BMC Med Educ. 2016;16(1):237.
    1. Graafland M, Vollebergh MF, Lagarde SM, van Haperen M, Bemelman WA, Schijven MP. A serious game can be a valid method to train clinical decision-making in surgery. World J Surg. 2014;38(12):3056–3062.
    1. Sugand K, Mawkin M, Gupte C. Training effect of using Touch Surgery for intramedullary femoral nailing. Injury. 2016;47(2):448–452.
    1. Kowalewski KF, Hendrie JD, Schmidt MW, et al. Validation of the mobile serious game application Touch Surgery for cognitive training and assessment of laparoscopic cholecystectomy. Surg Endosc. 2017;31(10):4058–4066.
    1. Graafland M, Bemelman WA, Schijven MP. Appraisal of face and content validity of a serious game improving situational awareness in surgical training. J Laparoendosc Adv Surg Tech A. 2015;25(1):43–49.
    1. Rouse DN. Employing Kirkpatrick’s evaluation framework to determine the effectiveness of health information management courses and programs. Perspect Health Inf Manag. 2011;8:1c.
    1. Cutrer WB, Sullivan WM, Fleming AE. Educational strategies for improving clinical reasoning. Curr Probl Pediatr Adolesc Health Care. 2013;43(9):248–257.
    1. Charlin B, Lubarsky S, Millette B, et al. Clinical reasoning processes: unravelling complexity through graphical representation. Med Educ. 2012;46(5):454–463.
    1. Liou SR, Liu HC, Tsai HM, et al. The development and psychometric testing of a theory-based instrument to evaluate nurses’ perception of clinical reasoning competence. J Adv Nurs. 2016;72(3):707–717.
    1. Liaw SY, Rashasegaran A, Wong LF, et al. Development and psychometric testing of a Clinical Reasoning Evaluation Simulation Tool (CREST) for assessing nursing students’ abilities to recognize and respond to clinical deterioration. Nurse Educ Today. 2018;62:74–79.
    1. Cook DA, Erwin PJ, Triola MM. Computerized virtual patients in health professions education: a systematic review and meta-analysis. Acad Med. 2010;85(10):1589–1602.
    1. Knight JF, Carley S, Tregunna B, et al. Serious gaming technology in major incident triage training: a pragmatic controlled trial. Resuscitation. 2010;81(9):1175–1179.
    1. Kleinert R, Heiermann N, Plum PS, et al. Web-based immersive virtual patient simulators: positive effect on clinical reasoning in medical education. J Med Internet Res. 2015;17(11):e263.
    1. Johnsen HM, Fossum M, Vivekananda-Schmidt P, Fruhling A, Slettebo A. Teaching clinical reasoning and decision-making skills to nursing students: design, development, and usability evaluation of a serious game. Int J Med Inform. 2016;94:39–48.
    1. Pelaccia Thierry, Tardif Jacques, Triby Emmanuel, Charlin Bernard. An analysis of clinical reasoning through a recent and comprehensive approach: the dual-process theory. Medical Education Online. 2011;16(1):5890.
    1. Fodor JA. The modularity of mind: MIT Press/Bradford Books; 1983.
    1. Wright MC, Taekman JM, Endsley MR. Objective measures of situation awareness in a simulated medical environment. Qual Saf Health Care. 2004;13(Suppl 1):i65–i71.
    1. Bogossian FE, Cooper SJ, Cant R, Porter J, Forbes H, Team FAR A trial of e-simulation of sudden patient deterioration (FIRST2ACT WEB) on student learning. Nurse Educ Today. 2015;35(10):e36–e42.
    1. Liaw SY, Wong LF, Lim EY, et al. Effectiveness of a web-based simulation in improving nurses’ workplace practice with deteriorating ward patients: a pre- and postintervention study. J Med Internet Res. 2016;18(2):e37.
    1. Liaw SY, Chng DYJ, Wong LF, et al. The impact of a Web-based educational program on the recognition and management of deteriorating patients. J Clin Nurs. 2017;26(23-24):4848–4856.
    1. Mohan D, Farris C, Fischhoff B, et al. Efficacy of educational video game versus traditional educational apps at improving physician decision making in trauma triage: randomized controlled trial. BMJ. 2017;359:j5416.
    1. Chaudery M, Clark J, Dafydd DA, et al. The face, content, and construct validity assessment of a focused assessment in sonography for trauma simulator. J Surg Educ. 2015;72(5):1032–1038.

Source: PubMed

Sponsors and Collaborators

Medical Conditions

Drug Interventions

3
Subscribe