A Systematic Review and Meta-analysis on the Impact of Proficiency-based Progression Simulation Training on Performance Outcomes

Elio Mazzone, Stefano Puliatti, Marco Amato, Brendan Bunting, Bernardo Rocco, Francesco Montorsi, Alexandre Mottrie, Anthony G Gallagher, Elio Mazzone, Stefano Puliatti, Marco Amato, Brendan Bunting, Bernardo Rocco, Francesco Montorsi, Alexandre Mottrie, Anthony G Gallagher

Abstract

Objective: To analyze all published prospective, randomized, and blinded clinical studies on the proficiency-based progression (PBP) training using objective performance metrics.

Background: The benefit of PBP methodology to learning clinical skills in comparison to conventional training is not settled.

Methods: Search of PubMed, Cochrane library's Central, EMBASE, MEDLINE, and Scopus databases, from inception to 1st March 2020. Two independent reviewers extracted the data. The Medical Education Research Study Quality Instrument (MERSQI) was used to assess the methodological quality of included studies. Results were pooled using biased corrected standardized mean difference and ratio-of-means. Summary effects were evaluated using a series of fixed and random effects models. The primary outcome was the number of procedural errors performed comparing PBP and non-PBP-based training pathways. Secondary outcomes were the number of procedural steps completed and the time to complete the task/procedure.

Results: From the initial pool of 468 studies, 12 randomized clinical studies with a total of 239 participants were included in the analysis. In comparison to the non-PBP training, ratio-of-means results showed that PBP training reduced the number of performance errors by 60% (P < 0.001) and procedural time by 15% (P = 0.003) and increased the number of steps performed by 47% (P < 0.001).

Conclusions and relevance: Our systematic review and meta-analysis confirms that PBP training in comparison to conventional or quality assured training improved trainees' performances, by decreasing procedural errors and procedural time, while increasing the number of correct steps taken when compared to standard simulation-based training.

Conflict of interest statement

The authors report no conflicts of interest.

Copyright © 2020 The Author(s). Published by Wolters Kluwer Health, Inc.

References

    1. Salas E, Bowers CA, Rhodenizer L. It is not how much you have but how you use it: toward a rational use of simulation to support aviation training. Int J Aviat Psychol 1998; 8:197–208.
    1. Webster CS. The nuclear power industry as an alternative analogy for safety in anaesthesia and a novel approach for the conceptualisation of safety goals. Anaesthesia 2005; 60:1115–1122.
    1. Gaba DM, DeAnda A. A comprehensive anesthesia simulation environment: re-creating the operating room for research and training. Anesthesiology 1988; 69:387–394.
    1. Satava RM. Virtual reality surgical simulator. Surg Endosc 1993; 7:203–205.
    1. Seymour NE, Gallagher AG, Roman SA, et al. Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg 2002; 236:454–458.
    1. Seymour NE, Gallagher AG, Roman SA, et al. Analysis of errors in laparoscopic surgical procedures. Surg Endosc 2004; 18:592–595.
    1. Ericsson KA. Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Acad Med 2004; 79: (10 SUPPL): 70–81.
    1. Ahlberg G, Enochsson L, Gallagher AG, et al. Proficiency-based virtual reality training significantly reduces the error rate for residents during their first 10 laparoscopic cholecystectomies. Am J Surg 2007; 193:797–804.
    1. Mascheroni J, Mont L, Stockburger M, et al. International expert consensus on a scientific approach to training novice cardiac resynchronization therapy implanters using performance quality metrics. Int J Cardiol 2019; 289:63–69.
    1. Angelo RL, Ryu RKN, Pedowitz RA, et al. Metric development for an arthroscopic bankart procedure: assessment of face and content validity. Arthroscopy 2015; 31:1430–1440.
    1. Pedowitz RA, Nicandri GT, Angelo RL, et al. Objective assessment of knot-tying proficiency with the fundamentals of arthroscopic surgery training program workstation and knot tester. Arthrosc - J Arthrosc Relat Surg 2015; 31:1872–1879.
    1. Ritter EM, McClusky DA 3rd, Gallagher AG, et al. Real-time objective assessment of knot quality with a portable tensiometer is superior to execution time for assessment of laparoscopic knot-tying performance. Surg Innov 2005; 12:233–237.
    1. Angelo RL, Pedowitz RA, Ryu RKN, et al. The bankart performance metrics combined with a shoulder model simulator create a precise and accurate training tool for measuring surgeon skill. Arthroscopy 2015; 31:1639–1654.
    1. Angelo RL, Ryu RKN, Pedowitz RA, et al. The bankart performance metrics combined with a cadaveric shoulder create a precise and accurate assessment tool for measuring surgeon skill. Arthrosc 2015; 31:1655–1670.
    1. Van Sickle KR, Ritter EM, Baghai M, et al. Prospective, randomized, double-blind trial of curriculum-based training for intracorporeal suturing and knot tying. J Am Coll Surg 2008; 207:560–568.
    1. Angelo RL, Ryu RKN, Pedowitz RA, et al. A proficiency-based progression training curriculum coupled with a model simulator results in the acquisition of a superior arthroscopic bankart skill Set. Arthroscopy 2015; 31:1854–1871.
    1. Cates CU, Lönn L, Gallagher AG. Prospective, randomised and blinded comparison of proficiency-based progression full-physics virtual reality simulator training versus invasive vascular experience for learning carotid artery angiography by very experienced operators. BMJ Simul Technol Enhanc Learn 2016; 2:1–5.
    1. Srinivasan KK, Gallagher A, O’Brien N, et al. Proficiency-based progression training: an “end to end” model for decreasing error applied to achievement of effective epidural analgesia during labour: a randomised control study. BMJ Open 2018; 8:e020099.
    1. Breen D, O’Brien S, McCarthy N, et al. Effect of a proficiency-based progression simulation programme on clinical communication for the deteriorating patient: a randomised controlled trial. BMJ Open 2019; 9:e025992.
    1. Sutherland LM, Middleton PF, Anthony A, et al. Surgical simulation: a systematic review. Ann Surg 2006; 243:291–300.
    1. Gurusamy K, Aggarwal R, Palanivelu L, et al. Systematic review of randomized controlled trials on the effectiveness of virtual reality training for laparoscopic surgery. Br J Surg 2008; 95:1088–1097.
    1. Zendejas B, Brydges R, Hamstra SJ, et al. State of the evidence on simulation-based training for laparoscopic surgery: a systematic review. Ann Surg 2013; 257:586–593.
    1. PRISMA statement. Available at: .
    1. Ahmed OM, O’Donnell BD, Gallagher AG, et al. Development of performance and error metrics for ultrasound-guided axillary brachial plexus block. Adv Med Educ Pract 2017; 8:257–263.
    1. Palter VN, Grantcharov TP. Development and validation of a comprehensive curriculum to teach an advanced minimally invasive procedure: a randomized controlled trial. Ann Surg 2012; 256:25–32.
    1. Peeters SHP, Akkermans J, Slaghekke F, et al. Simulator training in fetoscopic laser surgery for twin-twin transfusion syndrome: a pilot randomized controlled trial. Ultrasound Obstet Gynecol 2015; 46:319–326.
    1. Jensen UJ, Jensen J, Ahlberg G, et al. Virtual reality training in coronary angiography and its transfer effect to real-life catheterisation lab. EuroIntervention 2016; 11:1503–1510.
    1. Reed DA, Cook DA, Beckman TJ, et al. Association between funding and quality of published medical education research. JAMA 2007; 298:1002–1009.
    1. Kazdin AE. Behavior Modification in Applied Settings. 5th Ed.Belmont, CA: Thomson Brooks/Cole Publishing Co; 1994.
    1. Gallagher AG. Metric-based simulation training to proficiency in medical education:- what it is and how to do it. Ulster Med J 2012; 81:107–113.
    1. Gallagher AG, Ritter EM, Champion H, et al. Virtual reality simulation for the operating room: proficiency-based training as a paradigm shift in surgical skills training. Ann Surg 2005; 241:364–372.
    1. Springer Publishing Company, Incorporated, Gallagher AG, O'Sullivan GC. Fundamentals of Surgical Simulation: Principles and Practice. 2011.
    1. Crossley R, Liebig T, Holtmannspoetter M, et al. Validation studies of virtual reality simulation performance metrics for mechanical thrombectomy in ischemic stroke. J Neurointerv Surg 2019; 11:775–780.
    1. Cook DA, Hatala R, Brydges R, et al. Technology-enhanced simulation for health professions education: a systematic review and meta-analysis. JAMA 2011; 306:978–988.
    1. Friedrich JO, Adhikari NKJ, Beyene J. Ratio of means for analyzing continuous outcomes in meta-analysis performed as well as mean difference methods. J Clin Epidemiol 2011; 64:556–564.
    1. Friedrich JO, Adhikari NKJ, Beyene J. The ratio of means method as an alternative to mean differences for analyzing continuous outcome variables in meta-analysis: a simulation study. BMC Med Res Methodol 2008; 8:32.
    1. Higgins JPT, Thompson SG, Deeks JJ, et al. Measuring inconsistency in meta-analyses. BMJ 2003; 327:557–560.
    1. DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials 1986; 7:177–188.
    1. Shim SR, Kim SJ. Intervention meta-analysis: application and practice using R software. Epidemiol Health 2019; 41:e2019008.
    1. Egger M, Davey Smith G, Schneider M, et al. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997; 315:629–634.
    1. Vargas MV, Moawad G, Denny K, et al. Transferability of virtual reality, simulation-based, robotic suturing skills to a live porcine model in novice surgeons: a single-blind randomized controlled trial. J Minim Invasive Gynecol 2017; 24:420–425.
    1. Varley M, Choi R, Kuan K, et al. Prospective randomized assessment of acquisition and retention of SILS skills after simulation training. Surg Endosc 2015; 29:113–118.
    1. Yang C, Kalinitschenko U, Helmert JR, et al. Transferability of laparoscopic skills using the virtual reality simulator. Surg Endosc 2018; 32:4132–4137.
    1. Maertens H, Aggarwal R, Moreels N, et al. A Proficiency Based Stepwise Endovascular Curricular Training (PROSPECT) Program enhances operative performance in real life: a randomised controlled trial. Eur J Vasc Endovasc Surg 2017; 54:387–396.
    1. Maertens H, Vermassen F, Aggarwal R, et al. Endovascular training using a simulation based curriculum is less expensive than training in the hybrid angiosuite. Eur J Vasc Endovasc Surg 2018; 56:583–590.
    1. Martin JR, Anton N, Timsina L, et al. Performance variability during training on simulators is associated with skill transfer. Surgery 2019; 165:1065–1068.
    1. Snyder CW, Vandromme MJ, Tyra SL, et al. Proficiency-based laparoscopic and endoscopic training with virtual reality simulators: a comparison of proctored and independent approaches. J Surg Educ 2009; 66:201–207.
    1. Snyder CW, Vandromme MJ, Tyra SL, et al. Effects of virtual reality simulator training method and observational learning on surgical performance. World J Surg 2011; 35:245–252.
    1. Stefanidis D, Scerbo MW, Montero PN, et al. Simulator training to automaticity leads to improved skill transfer compared with traditional proficiency-based training: a randomized controlled trial. Ann Surg 2012; 255:30–37.
    1. Stoller J, Joseph J, Parodi N, et al. Are there detrimental effects from proficiency-based training in fundamentals of laparoscopic surgery among novices? An exploration of goal theory. J Surg Educ 2016; 73:215–221.
    1. Ahlborg L, Hedman L, Nisell H, et al. Simulator training and non-technical factors improve laparoscopic performance among OBGYN trainees. Acta Obstet Gynecol Scand 2013; 92:1194–1201.
    1. Bjerrum F, Sorensen JL, Konge L, et al. Randomized trial to examine procedure-to-procedure transfer in laparoscopic simulator training. Br J Surg 2016; 103:44–50.
    1. Bjerrum F, Maagaard M, Led Sorensen J, et al. Effect of instructor feedback on skills retention after laparoscopic simulator training: follow-up of a randomized trial. J Surg Educ 2015; 72:53–60.
    1. Brydges R, Carnahan H, Rose D, et al. Comparing self-guided learning and educator-guided learning formats for simulation-based clinical training. J Adv Nurs 2010; 66:1832–1844.
    1. Dawidek MT, Roach VA, Ott MC, et al. Changing the learning curve in novice laparoscopists: incorporating direct visualization into the simulation training program. J Surg Educ 2017; 74:30–36.
    1. De Win G, Van Bruwaene S, Kulkarni J, et al. An evidence-based laparoscopic simulation curriculum shortens the clinical learning curve and reduces surgical adverse events. Adv Med Educ Pract 2016; 7:357–370.
    1. Franklin BR, Placek SB, Wagner MD, et al. Cost comparison of fundamentals of laparoscopic surgery training completed with standard fundamentals of laparoscopic surgery equipment versus low-cost equipment. J Surg Educ 2017; 74:459–465.
    1. Gala R, Orejuela F, Gerten K, et al. Effect of validated skills simulation on operating room performance in obstetrics and gynecology residents: a randomized controlled trial. Obstet Gynecol 2013; 121:578–584.
    1. Gauger PG, Hauge LS, Andreatta PB, et al. Laparoscopic simulation training with proficiency targets improves practice and performance of novice surgeons. Am J Surg 2010; 199:72–80.
    1. Gershuni V, Woodhouse J, Brunt LM. Retention of suturing and knot-tying skills in senior medical students after proficiency-based training: Results of a prospective, randomized trial. Surgery 2013; 154:823–830.
    1. Grierson LEM, Lyons JL, Dubrowski A. Gaze-down endoscopic practise leads to better novice performance on gaze-up displays. Med Educ 2013; 47:166–172.
    1. Hashimoto DA, Petrusa E, Phitayakorn R, et al. A proficiency-based virtual reality endoscopy curriculum improves performance on the fundamentals of endoscopic surgery examination. Surg Endosc 2018; 32:1397–1404.
    1. Hseino H, Nugent E, Lee MJ, et al. Skills transfer after proficiency-based simulation training in superficial femoral artery angioplasty. Simul Healthc 2012; 7:274–281.
    1. Kiely DJ, Gotlieb WH, Lau S, et al. Virtual reality robotic surgery simulation curriculum to teach robotic suturing: a randomized controlled trial. J Robot Surg 2015; 9:179–186.
    1. Lemke M, Lia H, Gabinet-Equihua A, et al. Optimizing resource utilization during proficiency-based training of suturing skills in medical students: a randomized controlled trial of faculty-led, peer tutor-led, and holography-augmented methods of teaching. Surg Endosc 2020; 34:1678–1687.
    1. Puliatti S, Mazzone E, Dell’Oglio P. Training in robot-assisted surgery. Curr Opin Urol 2020; 30:65–72.
    1. Puliatti S, Mazzone E, Amato M, et al. Development and validation of the objective assessment of robotic suturing and knot tying skills for chicken anastomotic model. Surg Endosc 2020; doi:10.1007/s00464-020-07918-5.
    1. Tippins N, Sackett P, Oswald F. Principles for the validation and use of personnel selection procedures. Ind Organ Psychol 2018; 11 (S1):1–97.
    1. Mascheroni J, Mont L, Stockburger M, et al. A validation study of intraoperative performance metrics for training novice cardiac resynchronization therapy implanters. Int J Cardiol 2020; 307:48–54.
    1. Kojima K, Graves M, Taha W, et al. AO international consensus panel for metrics on a closed reduction and fixation of a 31A2 pertrochanteric fracture. Injury 2018; 49:2227–2233.
    1. McGaghie WC, Issenberg SB, Barsuk JH, et al. A critical review of simulation-based mastery learning with translational outcomes. Med Educ 2014; 48:375–385.
    1. Zendejas B, Cook DA, Bingener J, et al. Simulation-based mastery learning improves patient outcomes in laparoscopic inguinal hernia repair: a randomized controlled trial. Ann Surg 2011; 254:502–511.

Source: PubMed

Sponsorer och medarbetare

Medicinska tillstånd

Läkemedelsinterventioner

3
Prenumerera