Development and Assessment of a Simulation Model of Focused and Ongoing Professional Practice Evaluations (OPPE16)

Since 2007, the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) has required evaluation of practicing attending physicians by management to ensure that procedural skills and medical care are meeting the accepted standards of care as assessed by each individual institution. This assessment is known as the ongoing professional practice evaluation (OPPE). There are several methods by which this can be accomplished, including through chart review of clinician practice and by verbally assessing knowledge of the steps required to do a particular procedure. However, for infrequently performed or complicated procedures, these methods may not allow objective evaluation on a regular basis. Simulation using task trainers or manikin models offers an alternative method of objective evaluation in a standardized setting.

The goal of this study is to develop two simulated scenarios to assess physician skill in relatively uncommon procedures and compare the data obtained against the verbal assessment and chart review model previously used at our institution. Two raters will assess each practitioner's verbal and simulated procedures. They will also assess globally their confidence that the physician is capable of performing the procedure safely and correctly based on the verbal or simulated trial.

The investigators anticipate that using a simulated experience for assessment will increase the ability of raters to assess proficiency for the purposes of an OPPE, specifically by increasing the number of critical procedural components that can be objectively evaluated. The investigators hypothesize that there will not be a strong relationship between the scores obtained on the verbal assessment and the scores obtained on the task trainer assessment. The investigators anticipate that there will be a portion of study participants that do not meet a minimum passing standard and may require additional deliberate practice and further testing.

Study Overview

• Status: Completed
• Conditions: Emergencies; Competence
• Intervention / Treatment: Other: Verbal versus procedural assessment

Detailed Description

Physician training has developed over recent years to include simulation-based learning as a method of both formative and summative assessment. Simulation based medical education (SBME) with deliberate practice has been shown to be superior to traditional clinical education with respect to procedural skills in physicians in training (McGaghie,W.C. 2011), and a similar concept with Mastery Learning has shown to increase resident skill in lumbar puncture placement. In addition, it has been shown that Simulation Based Mastery Learning (SBML) in central line placement reduces hospital wide rates of central line associated bacterial infections.

When used in continuing medical education (CME), simulation has been shown to be feasible, and to increase clinician confidence in patient management. A recent study of nephrology fellows and attendings in practice showed that both groups scored similarly on a pre-test examination, and that all fellows who completed mastery learning subsequently passed the checklist assessment at the competency level and rated the training session highly.

In other work, simulation has been used for ongoing professional practice assessments (OPPE), for example in psychiatry using online simulated patients and assessments as a way of identifying a smaller number of attendings requiring focused assessment in person). Using electronically collected clinical data, anesthesia cases were reviewed for deviation from the standard of care on several key pieces of documentation common to many anesthesia procedures. This method of direct chart assessment for OPPE may be possible in anesthesia, a field where many cases have similarities in procedure, but would not work for a field such as Emergency Medicine, where the type and acuity of patient encounters varies greatly from day to day.

The ability of Simulation to present a standardized, specific procedural need offers an opportunity to assess Attending Emergency Physicians on a variety of critical procedures in a concise way. The investigators' research aims to present one possible approach to this assessment.

• Study Type: Observational
• Enrollment (Actual): 17

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: Child; Adult; Older Adult
• Accepts Healthy Volunteers: No
• Genders Eligible for Study: All

• Sampling Method: Non-Probability Sample

Study Population

All faculty practicing in an urban emergency department were eligible for enrollment

Description

Inclusion Criteria:

• Emergency Medicine physician

Exclusion Criteria:

• Not Emergency Medicine physician

Study Plan

How is the study designed?

Design Details

• Observational Models: Case-Only
• Time Perspectives: Other

Number of groups / cohorts

1

Cohorts and Interventions

Group / Cohort	Intervention / Treatment
2016 OPPE; All faculty participating in yearly ongoing professional Practice evaluation were eligible for enrollment	Other: Verbal versus procedural assessment; Faculty were assessed on both verbal and simulated performance of procedures.

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Performance on Procedural Checklist	A 29-item procedural checklist for Central Line and a 21-item procedural checklist for Lumbar puncture will be used to assess performance by clinicians on Central Venous Catheter insertion and Lumbar Puncture performance.	Immediate

Secondary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Acceptance of simulated training and assessment format	A survey tool will be used to query participants on their preferred way of both practicing and being assessed on procedural skill. It will also assess whether participants feel inclined to complete a future simulated procedural training session.	Immediate

Collaborators and Investigators

• Sponsor: Rush University Medical Center
• Investigators: Principal Investigator: Sara M Hock, MD, Rush University Medical Center

Publications and helpful links

General Publications

• OPPE and FPPE: Tools to help make privileging decisions 2013. Wise RA. (Accessed Jul 19, 2017, at http://www.jointcommission.org/jc_physician_blog/oppe_fppe_tools_privileging_decisions/).
• McGaghie WC, Issenberg SB, Cohen ER, Barsuk JH, Wayne DB. Does simulation-based medical education with deliberate practice yield better results than traditional clinical education? A meta-analytic comparative review of the evidence. Acad Med. 2011 Jun;86(6):706-11. doi: 10.1097/ACM.0b013e318217e119.
• Barsuk JH, Cohen ER, Caprio T, McGaghie WC, Simuni T, Wayne DB. Simulation-based education with mastery learning improves residents' lumbar puncture skills. Neurology. 2012 Jul 10;79(2):132-7. doi: 10.1212/WNL.0b013e31825dd39d. Epub 2012 Jun 6.
• Barsuk JH, McGaghie WC, Cohen ER, O'Leary KJ, Wayne DB. Simulation-based mastery learning reduces complications during central venous catheter insertion in a medical intensive care unit. Crit Care Med. 2009 Oct;37(10):2697-701.
• Barsuk JH, Cohen ER, Potts S, Demo H, Gupta S, Feinglass J, McGaghie WC, Wayne DB. Dissemination of a simulation-based mastery learning intervention reduces central line-associated bloodstream infections. BMJ Qual Saf. 2014 Sep;23(9):749-56. doi: 10.1136/bmjqs-2013-002665. Epub 2014 Mar 14.
• Kerr B, Hawkins TL, Herman R, Barnes S, Kaufmann S, Fraser K, Ma IW. Feasibility of scenario-based simulation training versus traditional workshops in continuing medical education: a randomized controlled trial. Med Educ Online. 2013 Jul 18;18:21312. doi: 10.3402/meo.v18i0.21312.
• McQuillan RF, Clark E, Zahirieh A, Cohen ER, Paparello JJ, Wayne DB, Barsuk JH. Performance of Temporary Hemodialysis Catheter Insertion by Nephrology Fellows and Attending Nephrologists. Clin J Am Soc Nephrol. 2015 Oct 7;10(10):1767-72. doi: 10.2215/CJN.01720215. Epub 2015 Sep 25.
• Barsuk JH, Cohen ER, Nguyen D, Mitra D, O'Hara K, Okuda Y, Feinglass J, Cameron KA, McGaghie WC, Wayne DB. Attending Physician Adherence to a 29-Component Central Venous Catheter Bundle Checklist During Simulated Procedures. Crit Care Med. 2016 Oct;44(10):1871-81. doi: 10.1097/CCM.0000000000001831.

Study record dates

Study Major Dates

• Study Start (Actual): October 1, 2015
• Primary Completion (Actual): May 1, 2016
• Study Completion (Actual): May 1, 2016

Study Registration Dates

• First Submitted: July 17, 2017
• First Submitted That Met QC Criteria: July 24, 2017
• First Posted (Actual): July 25, 2017

Study Record Updates

• Last Update Posted (Actual): July 28, 2017
• Last Update Submitted That Met QC Criteria: July 26, 2017
• Last Verified: July 1, 2017

More Information

Terms related to this study

• Keywords: Simulation; Task Trainer
• Additional Relevant MeSH Terms: Pathologic Processes; Disease Attributes; Emergencies
• Other Study ID Numbers: 15100805-IRB01-AM01

Plan for Individual participant data (IPD)

• Plan to Share Individual Participant Data (IPD)?: Undecided
• IPD Plan Description: If requested, data may be shared. Limited generalizability of data may limit its usefulness to other investigators

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No