Group assessments of resident physicians improve reliability and decrease halo error

Matthew R Thomas, Thomas J Beckman, Karen F Mauck, Stephen S Cha, Kris G Thomas, Matthew R Thomas, Thomas J Beckman, Karen F Mauck, Stephen S Cha, Kris G Thomas

Abstract

Background: Individual faculty assessments of resident competency are complicated by inconsistent application of standards, lack of reliability, and the "halo" effect.

Objective: We determined whether the addition of faculty group assessments of residents in an ambulatory clinic, compared with individual faculty-of-resident assessments alone, have better reliability and reduced halo effects.

Design: This prospective, longitudinal study was performed in the outpatient continuity clinics of a large internal medicine residency program.

Main measures: Faculty-on-resident and group faculty-on-resident assessment scores were used for comparison.

Key results: Overall mean scores were significantly higher for group than individual assessments (3.92 ± 0.51 vs. 3.83 ± 0.38, p = 0.0001). Overall inter-rater reliability increased when combining group and individual assessments compared to individual assessments alone (intraclass correlation coefficient, 95% CI = 0.828, 0.785-0.866 vs. 0.749, 0.686-0.804). Inter-item correlations were less for group (0.49) than individual (0.68) assessments.

Conclusions: This study demonstrates improved inter-rater reliability and reduced range restriction (halo effect) of resident assessment across multiple performance domains by adding the group assessment method to traditional individual faculty-on-resident assessment. This feasible model could help graduate medical education programs achieve more reliable and discriminating resident assessments.

Figures

Figure 1
Figure 1
Model of a 4-week calendar for a single firm and annual calendar of group assessments. The top part of the figure conceptually illustrates a 4-week calendar for a single firm. Faculty members (indicated by the letters A through J) precept clinic on a fixed day every week. For example, Faculty A always precepts on Mondays. In contrast, residents (indicated by numbers 1 through 24) attend clinic on a different day each week, since their clinic calendar is dependent on their hospital call schedule, which, in turn, is independent of the day of the week. For example, Resident 1 attends clinic on Monday during the first week, and then Friday during the second week. This leads to numerous faculty members gaining experience with a specific resident over time, but no single faculty member who has extensive experience with any specific resident. The lower part of the figure illustrates the annual calendar of group assessments, wherein the performance of residents from a single PGY year is discussed on a quarterly basis in a rotating fashion.
Figure 2
Figure 2
Faculty opinions regarding group assessments (n = 87). Faculty members were asked immediately following each group assessment whether the discussion improved their understanding of resident strengths, weaknesses, and learning plans; improved their understanding of the process of assessment; improved their confidence in their own assessment abilities; and improved the overall process of assessment. They were then asked to rate their overall satisfaction with the process.

Source: PubMed

Prossimi studi clinici

Sponsor e collaboratori

Condizioni mediche

Interventi farmacologici

Impostazioni dei cookie

Utilizziamo i cookie per assicurarci di darti la migliore esperienza sul nostro sito web. Per maggiori dettagli leggi attentamente la nostra Informativa sulla privacy e sui cookie. ...>>>Puoi modificare le tue preferenze o revocare il tuo consenso in qualsiasi momento eliminando i cookie dal tuo sito web o computer come descritto nella policy. Accettalo e scegli le tue preferenze spuntando le caselle corrispondenti nella sezione "Gestisci impostazioni" di seguito. Leggi attentamente i nostri Termini di servizio prima di procedere con l'utilizzo di qualsiasi parte di questo sito web.
«Gestisci impostazioni