Group assessments of resident physicians improve reliability and decrease halo error

Matthew R Thomas, Thomas J Beckman, Karen F Mauck, Stephen S Cha, Kris G Thomas, Matthew R Thomas, Thomas J Beckman, Karen F Mauck, Stephen S Cha, Kris G Thomas

Abstract

Background: Individual faculty assessments of resident competency are complicated by inconsistent application of standards, lack of reliability, and the "halo" effect.

Objective: We determined whether the addition of faculty group assessments of residents in an ambulatory clinic, compared with individual faculty-of-resident assessments alone, have better reliability and reduced halo effects.

Design: This prospective, longitudinal study was performed in the outpatient continuity clinics of a large internal medicine residency program.

Main measures: Faculty-on-resident and group faculty-on-resident assessment scores were used for comparison.

Key results: Overall mean scores were significantly higher for group than individual assessments (3.92 ± 0.51 vs. 3.83 ± 0.38, p = 0.0001). Overall inter-rater reliability increased when combining group and individual assessments compared to individual assessments alone (intraclass correlation coefficient, 95% CI = 0.828, 0.785-0.866 vs. 0.749, 0.686-0.804). Inter-item correlations were less for group (0.49) than individual (0.68) assessments.

Conclusions: This study demonstrates improved inter-rater reliability and reduced range restriction (halo effect) of resident assessment across multiple performance domains by adding the group assessment method to traditional individual faculty-on-resident assessment. This feasible model could help graduate medical education programs achieve more reliable and discriminating resident assessments.

Figures

Figure 1
Figure 1
Model of a 4-week calendar for a single firm and annual calendar of group assessments. The top part of the figure conceptually illustrates a 4-week calendar for a single firm. Faculty members (indicated by the letters A through J) precept clinic on a fixed day every week. For example, Faculty A always precepts on Mondays. In contrast, residents (indicated by numbers 1 through 24) attend clinic on a different day each week, since their clinic calendar is dependent on their hospital call schedule, which, in turn, is independent of the day of the week. For example, Resident 1 attends clinic on Monday during the first week, and then Friday during the second week. This leads to numerous faculty members gaining experience with a specific resident over time, but no single faculty member who has extensive experience with any specific resident. The lower part of the figure illustrates the annual calendar of group assessments, wherein the performance of residents from a single PGY year is discussed on a quarterly basis in a rotating fashion.
Figure 2
Figure 2
Faculty opinions regarding group assessments (n = 87). Faculty members were asked immediately following each group assessment whether the discussion improved their understanding of resident strengths, weaknesses, and learning plans; improved their understanding of the process of assessment; improved their confidence in their own assessment abilities; and improved the overall process of assessment. They were then asked to rate their overall satisfaction with the process.

Source: PubMed

다가오는 임상 시험

스폰서 및 공동 작업자

건강 상태

약물 개입

쿠키 설정

당사는 당사 웹 사이트에서 최상의 경험을 제공하기 위해 쿠키를 사용합니다. 자세한 내용은 개인 정보 보호 및 쿠키 정책 을주의 깊게 읽어보십시오. ...>>>귀하는 정책에 설명 된대로 귀하의 웹 사이트 또는 컴퓨터에서 쿠키를 삭제하여 언제든지 귀하의 기본 설정을 변경하거나 동의를 철회 할 수 있습니다. 동의하고 아래 "설정 관리"섹션에서 해당 상자를 선택하여 기본 설정을 선택하십시오. 이 웹 사이트의 일부를 사용하기 전에 서비스 약관 을주의 깊게 읽어 보시기 바랍니다.
«설정 관리