Group assessments of resident physicians improve reliability and decrease halo error

Matthew R Thomas, Thomas J Beckman, Karen F Mauck, Stephen S Cha, Kris G Thomas, Matthew R Thomas, Thomas J Beckman, Karen F Mauck, Stephen S Cha, Kris G Thomas

Abstract

Background: Individual faculty assessments of resident competency are complicated by inconsistent application of standards, lack of reliability, and the "halo" effect.

Objective: We determined whether the addition of faculty group assessments of residents in an ambulatory clinic, compared with individual faculty-of-resident assessments alone, have better reliability and reduced halo effects.

Design: This prospective, longitudinal study was performed in the outpatient continuity clinics of a large internal medicine residency program.

Main measures: Faculty-on-resident and group faculty-on-resident assessment scores were used for comparison.

Key results: Overall mean scores were significantly higher for group than individual assessments (3.92 ± 0.51 vs. 3.83 ± 0.38, p = 0.0001). Overall inter-rater reliability increased when combining group and individual assessments compared to individual assessments alone (intraclass correlation coefficient, 95% CI = 0.828, 0.785-0.866 vs. 0.749, 0.686-0.804). Inter-item correlations were less for group (0.49) than individual (0.68) assessments.

Conclusions: This study demonstrates improved inter-rater reliability and reduced range restriction (halo effect) of resident assessment across multiple performance domains by adding the group assessment method to traditional individual faculty-on-resident assessment. This feasible model could help graduate medical education programs achieve more reliable and discriminating resident assessments.

Figures

Figure 1
Figure 1
Model of a 4-week calendar for a single firm and annual calendar of group assessments. The top part of the figure conceptually illustrates a 4-week calendar for a single firm. Faculty members (indicated by the letters A through J) precept clinic on a fixed day every week. For example, Faculty A always precepts on Mondays. In contrast, residents (indicated by numbers 1 through 24) attend clinic on a different day each week, since their clinic calendar is dependent on their hospital call schedule, which, in turn, is independent of the day of the week. For example, Resident 1 attends clinic on Monday during the first week, and then Friday during the second week. This leads to numerous faculty members gaining experience with a specific resident over time, but no single faculty member who has extensive experience with any specific resident. The lower part of the figure illustrates the annual calendar of group assessments, wherein the performance of residents from a single PGY year is discussed on a quarterly basis in a rotating fashion.
Figure 2
Figure 2
Faculty opinions regarding group assessments (n = 87). Faculty members were asked immediately following each group assessment whether the discussion improved their understanding of resident strengths, weaknesses, and learning plans; improved their understanding of the process of assessment; improved their confidence in their own assessment abilities; and improved the overall process of assessment. They were then asked to rate their overall satisfaction with the process.

Source: PubMed

Prochains essais cliniques

Paramètres des cookies

Nous utilisons des cookies pour nous assurer de vous offrir la meilleure expérience sur notre site Web. Pour plus de détails, lisez attentivement notre Politique de confidentialité et de cookies. ...>>>Vous pouvez modifier vos préférences ou retirer votre consentement à tout moment en supprimant les cookies de votre site Web ou de votre ordinateur comme décrit dans la politique. Veuillez l'accepter et choisir vos préférences en cochant les cases correspondantes dans la section "Gérer les paramètres" ci-dessous. Veuillez lire attentivement nos Conditions d'utilisation avant de continuer à utiliser une partie de ce site Web.
«Gérer les paramètres