Group assessments of resident physicians improve reliability and decrease halo error

Matthew R Thomas, Thomas J Beckman, Karen F Mauck, Stephen S Cha, Kris G Thomas, Matthew R Thomas, Thomas J Beckman, Karen F Mauck, Stephen S Cha, Kris G Thomas

Abstract

Background: Individual faculty assessments of resident competency are complicated by inconsistent application of standards, lack of reliability, and the "halo" effect.

Objective: We determined whether the addition of faculty group assessments of residents in an ambulatory clinic, compared with individual faculty-of-resident assessments alone, have better reliability and reduced halo effects.

Design: This prospective, longitudinal study was performed in the outpatient continuity clinics of a large internal medicine residency program.

Main measures: Faculty-on-resident and group faculty-on-resident assessment scores were used for comparison.

Key results: Overall mean scores were significantly higher for group than individual assessments (3.92 ± 0.51 vs. 3.83 ± 0.38, p = 0.0001). Overall inter-rater reliability increased when combining group and individual assessments compared to individual assessments alone (intraclass correlation coefficient, 95% CI = 0.828, 0.785-0.866 vs. 0.749, 0.686-0.804). Inter-item correlations were less for group (0.49) than individual (0.68) assessments.

Conclusions: This study demonstrates improved inter-rater reliability and reduced range restriction (halo effect) of resident assessment across multiple performance domains by adding the group assessment method to traditional individual faculty-on-resident assessment. This feasible model could help graduate medical education programs achieve more reliable and discriminating resident assessments.

Figures

Figure 1
Figure 1
Model of a 4-week calendar for a single firm and annual calendar of group assessments. The top part of the figure conceptually illustrates a 4-week calendar for a single firm. Faculty members (indicated by the letters A through J) precept clinic on a fixed day every week. For example, Faculty A always precepts on Mondays. In contrast, residents (indicated by numbers 1 through 24) attend clinic on a different day each week, since their clinic calendar is dependent on their hospital call schedule, which, in turn, is independent of the day of the week. For example, Resident 1 attends clinic on Monday during the first week, and then Friday during the second week. This leads to numerous faculty members gaining experience with a specific resident over time, but no single faculty member who has extensive experience with any specific resident. The lower part of the figure illustrates the annual calendar of group assessments, wherein the performance of residents from a single PGY year is discussed on a quarterly basis in a rotating fashion.
Figure 2
Figure 2
Faculty opinions regarding group assessments (n = 87). Faculty members were asked immediately following each group assessment whether the discussion improved their understanding of resident strengths, weaknesses, and learning plans; improved their understanding of the process of assessment; improved their confidence in their own assessment abilities; and improved the overall process of assessment. They were then asked to rate their overall satisfaction with the process.

Source: PubMed

即将进行的临床试验

Cookie设置

我们使用cookie来确保我们在您的网站上给您最好的体验。有关更多详细信息，请仔细阅读我们的隐私和Cookie政策。...>>>您可以按照政策中的说明，随时通过从网站或计算机中删除Cookie来更改首选项或撤回您的同意。请接受它，然后在下面的“管理设置”部分中勾选相应的框，选择您的偏好。在继续使用本网站的任何部分之前，请仔细阅读我们的服务条款。
«管理设定