Development and reliability of a measure of clinician competence in providing illness management and recovery

Abstract

Objective: Illness management and recovery (IMR) is an evidence-based, manualized illness self-management program for people with severe mental illness. This study sought to develop a measure of IMR clinician competence and test its reliability and validity.

Methods: Two groups of subject matter experts each independently created a clinician-level IMR competence scale based on the IMR Fidelity Scale and on two unpublished instruments used to evaluate provider competence. The two versions were merged, and investigators used the initial version to independently rate recordings of IMR sessions. Ratings were compared and discussed, discrepancies were resolved, and the scale was revised through 14 iterations. The resulting IMR Treatment Integrity Scale (IT-IS) includes 13 required items and three optional items rated only when the particular skill is attempted. Four independent raters then used the IT-IS to score tapes of 60 IMR sessions and 20 control group sessions.

Results: The IT-IS showed excellent interrater reliability (.92). A factor analysis supported a one-factor model that showed good internal consistency. The scale successfully differentiated between IMR and control groups. Reliability and validity of individual items varied widely.

Conclusions: The IT-IS is a promising measure of clinician competence in providing IMR. The scale could be used for research and quality assurance and as a supervisory feedback tool. Future research is needed to examine item-level changes, predictive validity of the IT-IS, discriminant validity compared with other more structured interventions, and the reliability and validity of the scale for nongroup IMR.