Developing a Practice-Based Learning and Improvement Quality Improvement (QI) Systems Impact Assessment Questionnaire

Developing a PBLI QI Systems Impact Assessment Questionnaire

The Accreditation Council for Graduate Medical Education (ACGME) acknowledged the changing needs of physicians in training when it endorsed practice-based learning and improvement (PBLI) -- a competency that is typically omitted from medical curriculum. The goal is to have residents competent to investigate and evaluate their own patient care practices, integrate scientific evidence and be able to improve their practices. Available assessment tools do not adequately address all of the components of PBLI and few assessment tools attempt to capture the residents' ability to develop and implement clinically-based Continuous Quality Improvement (CQI) projects that involve the practice setting. Curriculums without such foci miss the importance of system perspectives and opportunities for interprofessional team development. Our aim is to evaluate preliminary data on the curriculum we developed to address the gaps, to develop an assessment tool, and to provide methods for assessing the sustainability of system projects.

The key component of the curriculum is the integration of system quality improvement projects. PBLI curriculum was offered on alternate rotations. Preliminary data is available from 6 PBLI QI Systems Curriculum blocks (n=50) and 5 comparison blocks (n=42). Data includes closed- and open-ended questions designed to assess resident PBLI application skills, the notes and presentation slides for the residents' presentation.

Study Overview

• Status: Completed
• Conditions: Quality Improvement
• Intervention / Treatment: Other: PBLI Curriculum

Detailed Description

Physicians in training operate in complex healthcare delivery systems but many have not been equipped with the knowledge or skills to analyze clinical environments and continually improve patient care.[1] Instead, their training emphasizes the clinical management of individual patients. The Accreditation Council for Graduate Medical Education (ACGME) acknowledged the changing needs of physicians in training when it endorsed one of two novel core competencies that are typically omitted from formal medical curriculum: practice-based learning and improvement (PBLI).[2,3] The ACGME's PBLI competency involves six points. The overall objective is to have residents competent to investigate and evaluate their own patient care practices, evaluate and integrate scientific evidence into their clinics and be able to improve their practices. However, the ACGME was not prescriptive about how to successfully implement and evaluate PBLI, and an established curriculum and validated assessment tool did not exist.[4] Efforts to date to develop an assessment tool provide a foundation, but none adequately addresses all of the components of PBLI as described by the ACGME. [3, 5-10] In particular, few assessment tools attempt to capture the residents' ability to develop and implement clinically-based CQI projects that involve the practice setting and assess impact on the practice setting and/or organization. Curriculums without such foci miss the importance of system perspectives, opportunities for interprofessional team development, and meaningful impacts on patient care at a broader organizational level.

After several iterations, a PBLI QI curriculum that addressed the gaps identified in many other curriculums was instituted. [11] The key component was the integration of system quality improvement projects that could evolve over several resident outpatient rotation blocks. Data from 11 blocks of the new curriculum is available for analysis. Data includes closed- and open-ended questions designed to assess resident PBLI application skills and the notes and presentation slides for the residents' presentation at the internal medicine's morbidity and mortality conference (IM MMC) at the end of each block. Funding would make it possible to more quickly evaluate the existing preliminary data. Our overall aim is to evaluate the preliminary data on the new curriculum and develop an assessment tool that addresses the six ACGME points and provides methods for assessing the sustainability of system projects. The next step will be to apply for a grant to more fully evaluate the curriculum and assessment tool.

• Study Type: Observational
• Enrollment (Actual): 92

Contacts and Locations

Study Locations

• United States
  • Ohio
    • Cleveland, Ohio, United States, 44106
      • Louis Stokes VA Medical Center, Cleveland, OH

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: 25 years to 75 years (Adult, Older Adult)
• Accepts Healthy Volunteers: Yes
• Genders Eligible for Study: All

• Sampling Method: Non-Probability Sample

Study Population

Internal Medicine residents

Description

Inclusion Criteria:

• None, data being used has already been collected from a previous study which the inclusion criteria included all Internal Medicine and Medicine-Pediatrics residents completing a 4 week ambulatory block from 2005-2006 were required to participate in the PBLI curriculum to satisfy an ACGME's core competency.

Exclusion Criteria:

• None, the residents from 2004 that did not complete a 4 week ambulatory block and residents participating in ambulatory block 7 and 13 were excluded from the study previously Institutional Review Board (IRB) approved and exempted study. Blocks 7 and 13 are not structured to permit teaching.

Study Plan

How is the study designed?

Number of groups / cohorts

2

Cohorts and Interventions

Group / Cohort	Intervention / Treatment
PBLI Curriculum group; To evaluate preliminary data on a PBLI curriculum grounded on QI system projects.	Other: PBLI Curriculum; PBLI curriculum was offered on alternate rotations
Comparison group; Received a different curriculum.

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Beliefs About Ability to Implement a CQI Project	Residents' belief about their ability to implement a CQI project was measured using a single efficacy item (values ranged from 1, strongly disagree, to 5, strongly agree). The item is from the Systems Quality Improvement Training and Assessment Tool. Differences (post minus pre) in this belief item were looked at with positive and higher difference values reflecting more positive change/increase in belief.	1 month

Secondary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Change in Knowledge Scores About Quality Improvement	Residents' knowledge was assessed using the knowledge scale (e.g., describe change concept, how a cause-effect diagram is created, elements of the improvement model) from the SQI TAT and scores could range from 0 to 54 points. Difference scores were used based on total score of the scale with larger positive values indicating more increase in knowledge.	1 Month

Collaborators and Investigators

• Sponsor: US Department of Veterans Affairs
• Investigators: Principal Investigator: David C Aron, MD MS, Louis Stokes VA Medical Center, Cleveland, OH

Publications and helpful links

General Publications

• Stulberg JJ, Delaney CP, Neuhauser DV, Aron DC, Fu P, Koroukian SM. Adherence to surgical care improvement project measures and the association with postoperative infections. JAMA. 2010 Jun 23;303(24):2479-85. doi: 10.1001/jama.2010.841.
• Lawrence RH, Tomolo AM. Development and preliminary evaluation of a practice-based learning and improvement tool for assessing resident competence and guiding curriculum development. J Grad Med Educ. 2011 Mar;3(1):41-8. doi: 10.4300/JGME-D-10-00102.1.
• Tomolo AM, Lawrence RH, Watts B, Augustine S, Aron DC, Singh MK. Pilot study evaluating a practice-based learning and improvement curriculum focusing on the development of system-level quality improvement skills. J Grad Med Educ. 2011 Mar;3(1):49-58. doi: 10.4300/JGME-D-10-00104.1.

Study record dates

Study Major Dates

• Study Start: January 1, 2008
• Primary Completion (Actual): July 1, 2008
• Study Completion (Actual): December 1, 2011

Study Registration Dates

• First Submitted: September 16, 2008
• First Submitted That Met QC Criteria: September 17, 2008
• First Posted (Estimate): September 18, 2008

Study Record Updates

• Last Update Posted (Estimate): May 12, 2015
• Last Update Submitted That Met QC Criteria: April 27, 2015
• Last Verified: April 1, 2015

More Information

Terms related to this study

• Keywords: Quality of Health Care; Education; Medical Residents; Program Evaluation
• Other Study ID Numbers: SHP 08-194