Workplace-based assessment: effects of rater expertise

M J B Govaerts, L W T Schuwirth, C P M Van der Vleuten, A M M Muijtjens, M J B Govaerts, L W T Schuwirth, C P M Van der Vleuten, A M M Muijtjens

Abstract

Traditional psychometric approaches towards assessment tend to focus exclusively on quantitative properties of assessment outcomes. This may limit more meaningful educational approaches towards workplace-based assessment (WBA). Cognition-based models of WBA argue that assessment outcomes are determined by cognitive processes by raters which are very similar to reasoning, judgment and decision making in professional domains such as medicine. The present study explores cognitive processes that underlie judgment and decision making by raters when observing performance in the clinical workplace. It specifically focuses on how differences in rating experience influence information processing by raters. Verbal protocol analysis was used to investigate how experienced and non-experienced raters select and use observational data to arrive at judgments and decisions about trainees' performance in the clinical workplace. Differences between experienced and non-experienced raters were assessed with respect to time spent on information analysis and representation of trainee performance; performance scores; and information processing--using qualitative-based quantitative analysis of verbal data. Results showed expert-novice differences in time needed for representation of trainee performance, depending on complexity of the rating task. Experts paid more attention to situation-specific cues in the assessment context and they generated (significantly) more interpretations and fewer literal descriptions of observed behaviors. There were no significant differences in rating scores. Overall, our findings seemed to be consistent with other findings on expertise research, supporting theories underlying cognition-based models of assessment in the clinical workplace. Implications for WBA are discussed.

Figures

Fig. 1
Fig. 1
1-Dimensional overall performance rating (R1)
Fig. 2
Fig. 2
6-Dimensional global rating scale clinical competencies (R2)

References

    1. Arts JARM, Gijselaers WH, Boshuizen HPA. Understanding managerial problem-solving, knowledge use and information processing: Investigating stages from school to the workplace. Contemporary Educational Psychology. 2006;31(4):387–410. doi: 10.1016/j.cedpsych.2006.05.005.
    1. Boshuizen, H. P. A. (1989). The development of medical expertise; a cognitive-psychological approach. Doctoral dissertation. Maastricht: Rijksuniversiteit Limburg.
    1. Cardy RL, Bernardin HJ, Abbott JG, Senderak MP, Taylor K. The effects of individual performance schemata and dimension familiarization on rating accuracy. Journal of Occupational Psychology. 1987;60:197–205.
    1. Chi M. Quantifying qualitative analyses of verbal data: A practical guide. The Journal of the Learning Sciences. 1997;6(3):271–315. doi: 10.1207/s15327809jls0603_1.
    1. Chi MTH. Two approaches to the study of experts’ characteristics. In: Ericsson KA, Charness N, Feltovich PJ, Hoffman RR, editors. The Cambridge handbook of expertise and expert performance. Cambridge: Cambridge University Press; 2006. pp. 21–30.
    1. Chi MTH, Feltovich PJ, Glaser R. Categorization and representation of physics problems by experts and novices. Cognitive Science: A Multidisciplinary Journal. 1981;5(2):121–152. doi: 10.1207/s15516709cog0502_2.
    1. Chi MTH, de Leeuw N, Chiu MH, LaVancher C. Eliciting self-explanations improves understanding. Cognitive Science. 1994;5:121–152. doi: 10.1207/s15516709cog0502_2.
    1. Coderre S, Mandin H, Harasym PH, Fick GH. Diagnostic reasoning strategies and diagnostic success. Medical Education. 2003;37:695–703. doi: 10.1046/j.1365-2923.2003.01577.x.
    1. Cunnington J, Southgate L. Relicensure, recertification and practice-based assessment. In: Norman GR, van der Vleuten CPM, Newble DI DI, editors. International handbook of research in medical education. Dordrecht: Kluwer Academic Publishers; 2002. pp. 883–912.
    1. DeNisi AS. Cognitive approach to performance appraisal: A program of research. New York: Routledge; 1996.
    1. Ericsson KA. Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Academic Medicine. 2004;79(10):S70–S81. doi: 10.1097/00001888-200410001-00022.
    1. Ericsson KA. The Influence of experience and deliberate practice on the development of superior expert performance. In: Ericsson KA, Charness N, Feltovich PJ, Hoffman RR, editors. The Cambridge handbook of expertise and expert performance. Cambridge: Cambridge University Press; 2006. pp. 683–704.
    1. Ericsson KA, Lehmann AC. Expert and exceptional performance: Evidence of maximal adaptation to task constraints. Annual Review of Psychology. 1996;47:273–305. doi: 10.1146/annurev.psych.47.1.273.
    1. Ericsson KA, Simon HA. Protocol analysis: Verbal reports as data. Cambridge, MA: MIT Press; 1993.
    1. Eva KW. What every teacher needs to know about clinical reasoning. Medical Education. 2004;39:98–106. doi: 10.1111/j.1365-2929.2004.01972.x.
    1. Feldman JM. Beyond attribution theory: Cognitive processes in performance appraisal. Journal of Applied Psychology. 1981;66(2):127–148. doi: 10.1037/0021-9010.66.2.127.
    1. Feltovich PJ, Johnson PE, Moller JH, Swanson DB. LCS: The role and development of medical knowledge in diagnostic expertise. In: Clancey WJ, Shortliffe EH, editors. Readings in medical artificial intelligence: The first decade. New York: Addison Wesley; 1984. pp. 275–319.
    1. Feltovich PJ, Prietula MJ, Ericsson KA. Studies of expertise from psychological perspectives. In: Ericsson KA, Charness N, Feltovich PJ, Hoffman RR, editors. The Cambridge handbook of expertise and expert performance. Cambridge: Cambridge University Press; 2006. pp. 41–68.
    1. Field A. Discovering statistics using SPSS. London: Sage Publications Ltd.; 2009.
    1. Gray JD. Global rating scales in residency education. Academic Medicine. 1996;71:S55–S61. doi: 10.1097/00001888-199601000-00043.
    1. Gruppen LD, Frohna AZ. Clinical Reasoning. In: Norman GR, van der Vleuten CPM, Newble DI DI, editors. International handbook of research in medical education. Dordrecht: Kluwer Academic Publishers; 2002. pp. 205–230.
    1. Hawe E. It’s pretty difficult to fail: the reluctance of lecturers to award a failing grade. Assessment and Evaluation in Higher Education. 2003;28(4):371–382. doi: 10.1080/0260293032000066209.
    1. Hobus PP, Schmidt HG, Boshuizen HP, Patel VL. Contextual factors in the activation of first diagnosis hypotheses: Expert-novice differences. Medical Education. 1987;21:471–476. doi: 10.1111/j.1365-2923.1987.tb01405.x.
    1. Kerrins JA, Cushing KS. Taking a second look: Expert and novice differences when observing the same classroom teaching segment a second time. Journal of Personnel Evaluation in Education. 2000;14(1):5–24. doi: 10.1023/A:1008152928725.
    1. Klein GA. A recognition primed decision (RPD) model of rapid decision making. In: Klein GA, Orasanu J, Calderwood R, Zsambok CE, editors. Decision-making in action: Models and methods. Norwood, NJ: Ablex; 1993. pp. 138–147.
    1. Kreiter CD, Ferguson KJ. Examining the generalizability of ratings across clerkships using a clinical evaluation form. Evaluation & The Health Professions. 2001;24:36–46. doi: 10.1177/01632780122034768.
    1. Levy PE, Williams JR. The social context of performance appraisal: A review and framework for the future. Journal of Management. 2004;30:881–905.
    1. Lievens F. Assessor training strategies and their effects on accuracy, interrater reliability, and discriminant validity. Journal of Applied Psychology. 2001;86(2):255–264. doi: 10.1037/0021-9010.86.2.255.
    1. Mero NP, Motowidlo SJ, Anna AL. Effects of accountability on rating behavior and rater accuracy. Journal of Applied Social Psychology. 2003;33(12):2493–2514. doi: 10.1111/j.1559-1816.2003.tb02777.x.
    1. Murphy KR, Balzer WK. Systematic distortions in memory-based behavior ratings and performance evaluation: Consequences for rating accuracy. Journal of Applied Psychology. 1986;71:39–44. doi: 10.1037/0021-9010.71.1.39.
    1. Murphy KR, Cleveland JN. Understanding performance appraisal: Social, organizational and goal-based perspectives. Thousand Oaks, CA: Sage Publications; 1995.
    1. Norcini JJ. Current perspectives in assessment: The assessment of performance at work. Medical Education. 2005;39:880–889. doi: 10.1111/j.1365-2929.2005.02182.x.
    1. Norman GR, Brooks LR, Allen SW. Recall by expert medical practitioners and novices as a record of processing attention. Journal of Experimental Psychology. Learning, Memory, and Cognition. 1989;15:1166–1174. doi: 10.1037/0278-7393.15.6.1166.
    1. Norman G, Eva K, Brooks L, Hamstra S. Expertise in medicine and surgery. In: Ericsson KA, Charness N, Feltovich PJ, Hoffman RR, editors. The Cambridge handbook of expertise and expert performance. Cambridge: Cambridge University Press; 2006. pp. 339–354.
    1. Ross KG, Shafer JL, Klein G. Professional judgments and “naturalistic decision making”. In: Ericsson KA, Charness N, Feltovich PJ, Hoffman RR, editors. The Cambridge handbook of expertise and expert performance. Cambridge: Cambridge University Press; 2006. pp. 403–420.
    1. Sabers DS, Cushing KS, Berliner DC. Differences among teachers in a task characterized by simultaneity, multidimensionality and immediacy. American Educational Research Journal. 1991;28:63–88.
    1. Schmidt HG, Boshuizen HPA. On the origin of intermediate effects in clinical case recall. Memory and Cognition. 1993;21:338–351. doi: 10.3758/BF03208266.
    1. Silber CG, Nasca TJ, Paskin DL, Eiger G, Robeson M, Veloski JJ. Do global rating forms enable program directors to assess the ACGME competencies? Academic Medicine. 2004;79:549–556. doi: 10.1097/00001888-200406000-00010.
    1. Van Barneveld C. The dependability of medical students’ performance ratings as documented on in-training evaluations. Academic Medicine. 2005;80(3):309–312. doi: 10.1097/00001888-200503000-00023.
    1. Van De Wiel MWJ, Boshuizen HPA, Schmidt HG. Knowledge restructuring in expertise development: Evidence from pathophysiological representations of clinical cases by students and physicians. European Journal of Cognitive Psychology. 2000;12(3):323–356. doi: 10.1080/09541440050114543.
    1. Van der Vleuten CPM, Schuwirth LWT. Assessing professional competence: From methods to programmes. Medical Education. 2005;39:309–317. doi: 10.1111/j.1365-2929.2005.02094.x.
    1. Voss JF, Tyler SW, Yengo LA. Individual differences in the solving of social science problems. In: Dillon R, Schmeck R, editors. Individual differences in cognition. New York: Academic Press; 1983. pp. 205–232.
    1. Walsh JP. Managerial and organizational cognition: Notes from a trip down to memory lane. Organizational Science. 1995;6(3):280–321. doi: 10.1287/orsc.6.3.280.
    1. Williams RG, Dunnington GL. Prognostic value of resident clinical performance ratings. Journal of the American College of Surgeons. 2004;199:620–627. doi: 10.1016/j.jamcollsurg.2004.05.273.
    1. Williams RG, Klamen DA, McCaghie WC. Cognitive, social and environmental sources of bias in clinical performance ratings. Teaching and Learning in Medicine. 2003;15(4):270–292. doi: 10.1207/S15328015TLM1504_11.
    1. Wimmers PF, Schmidt HG, Verkoeijen PPJL, Van de Wiel MWJ. Inducing expertise effects in clinical case recall. Medical Education. 2005;39(9):949–957. doi: 10.1111/j.1365-2929.2005.02250.x.

Source: PubMed

Sponsors en medewerkers

Medische omstandigheden

Geneesmiddelinterventies

3
Abonneren