A Checklist to Help Faculty Assess ACGME Milestones in a Video-Recorded OSCE

Abstract

Background: Faculty members need to assess resident performance using the Accreditation Council for Graduate Medical Education Milestones.

Objective: In this randomized study we used an objective structured clinical examination (OSCE) around the disclosure of an adverse event to determine whether use of a checklist improved the quality of milestone assessments by faculty.

Methods: In 2013, a total of 20 anesthesiology faculty members from 3 institutions were randomized to 2 groups to assess 5 videos of trainees demonstrating advancing levels of competency on the OSCE. One group used milestones alone, and the other used milestones plus a 13-item checklist with behavioral anchors based on ideal performance. We classified faculty ratings as either correct or incorrect with regard to the competency level demonstrated in each video, and then used logistic regression analysis to assess the effect of checklist use on the odds of correct classification.

Results: Thirteen of 20 faculty members rated assessing performance using milestones alone as difficult or very difficult. Checklist use was associated with significantly greater odds of correct classification at entry level (odds ratio [OR] = 9.2, 95% confidence interval [CI] 4.0-21.2) and at junior level (OR = 2.7, 95% CI 1.3-5.7) performance. For performance at other competency levels checklist use did not affect the odds of correct classification.

Conclusions: A majority of anesthesiology faculty members reported difficulty with assessing a videotaped OSCE of error disclosure using milestones as primary assessment tools. Use of the checklist assisted in correct assessments at the entry and junior levels.

Conflict of interest statement

Conflict of interest: The authors declare they have no competing interests. These results were presented at the Society for Education in Anesthesiology 29th Spring Meeting, Boston, Massachusetts, May 30–June 1, 2014, and at Gerald S. Gotterer Health Professions Education Research Day, Vanderbilt University Medical Center, Nashville, Tennessee, September 22, 2014.