Formative Evaluation in Implementation Research: Guideline Development

The Veterans Administration (VA) has embraced the use of implementation research as a critical way to improve care by facilitating the adoption of proven medical practices. Yet implementation science still remains rudimentary. Challenges exist in judging the success of an implementation project. Rather than relying solely on hard outcomes, an investigator must also assess how effectively an intervention is put into place at a site. This assessment comes via a formative evaluation (FE) plan. The FE plan can contain such diverse data elements as the number of fliers distributed about the intervention, how often an intervention advocate speaks to participants, the content derived from team meeting notes, and themes extracted from focus groups. The diversity of such data elements can appear foreign to investigators trained in traditional health services research. The FE plan can also lead to confusion about what to measure, when, and how to integrate the various measures into a unified whole. Complicating matters is the fact that FE is often qualitative--such as interviews, focus groups, and meeting notes which can make data difficult for investigators to interpret. Thus, there is a pressing need to provide clarity on how FE is to be used in implementation studies. This Short Term Project proposal offers a three-part approach toward specifying the best FE practices: first, a systematic review of the literature; second, qualitative interviews with VA implementation investigators; and third, a panel discussion of national experts, funded through a VA Implementation Collaborative Award. The panel discussion will outline FE guidelines based on the researchers expertise, the systematic review, and the qualitative interviews.

Study Overview

• Status: Completed
• Conditions: Chronic Disease

Detailed Description

Objectives: The Veterans Administration (VA) has embraced the use of implementation research as a critical way to improve care by facilitating the adoption of proven medical practices. Yet implementation science still remains rudimentary. Challenges exist in judging the success of an implementation project. Rather than relying solely on hard outcomes, an investigator must also assess how effectively an intervention is put into place at a site. This assessment comes via a formative evaluation (FE) plan. The FE plan can contain such diverse data elements as the number of fliers distributed about the intervention, how often an intervention advocate speaks to participants, the content derived from team meeting notes, and themes extracted from focus groups. The diversity of such data elements can appear foreign to investigators trained in traditional health services research. The FE plan can also lead to confusion about what to measure, when, and how to integrate the various measures into a unified whole. Complicating matters is the fact that FE is often qualitative-- such as interviews, focus groups, and meeting notes which can make data difficult for investigators to interpret. Thus, there is a pressing need to provide clarity on how FE is to be used in implementation studies. This Short Term Project proposal offers a three-part approach toward specifying the best FE practices: first, a systematic review of the literature; second, qualitative interviews with VA implementation investigators; and third, a panel discussion of national experts, funded through a VA Implementation Collaborative Award. The panel discussion will outline FE guidelines based on the researchers expertise, the systematic review, and the qualitative interviews. The specific aims for this proposal include the following: Specific Aim 1: Conduct a systematic review of VA funded implementation projects to classify the specific types of FE procedures employed. Specific Aim 2: Conduct in-depth telephone interviews with investigators of VA-funded implementation projects to establish which FE methods were used and which were considered most/least successful.

Specific Aim 3: Using the national experts gathered together under the umbrella of the Center for Health Equity Research and Promotions (CHERP) Implementation Research Collaborative, develop a set of guidelines outlining best FE practices in VA implementation research. Relevance to the VA, Anticipated Results, and Future Directions: This Short Term Project is designed to provide rapid results for developing the best approach toward the use of FE in VA implementation science. As such this proposal is responsive to the VA Health Services Research & Development (HSR&D) priority topics for both implementation and management and research methodology. The systematic review, the qualitative report, and the FE guidelines that will emerge from the project will have wide-spread applications within the VA implementation community, thereby supporting the HSR&D research mission. The study will also serve as needed pilot data for a future Merit Review award application designed to provide the theoretical and practical approaches toward FE use within the VA.

• Study Type: Observational
• Enrollment (Actual): 9

Contacts and Locations

Study Locations

• United States
  • Pennsylvania
    • Pittsburgh, Pennsylvania, United States, 15240
      • VA Pittsburgh Healthcare System University Drive Division, Pittsburgh, PA

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: 18 years and older (Adult, Older Adult)
• Accepts Healthy Volunteers: No
• Genders Eligible for Study: All

• Sampling Method: Non-Probability Sample

Study Population

VA health services investigators

Description

Inclusion Criteria:

• VA health services investigators

Exclusion Criteria:

• N/A

Study Plan

How is the study designed?

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Time Frame
Investigators attitudes toward formative evaluation	Current

Collaborators and Investigators

• Sponsor: VA Office of Research and Development
• Investigators: Principal Investigator: Susan L. Zickmund, PhD, VA Pittsburgh Healthcare System University Drive Division, Pittsburgh, PA

Study record dates

Study Major Dates

• Study Start (Actual): January 23, 2009
• Primary Completion (Actual): March 30, 2009
• Study Completion (Actual): March 31, 2009

Study Registration Dates

• First Submitted: May 20, 2008
• First Submitted That Met QC Criteria: May 22, 2008
• First Posted (Estimate): May 23, 2008

Study Record Updates

• Last Update Posted (Actual): June 6, 2017
• Last Update Submitted That Met QC Criteria: June 2, 2017
• Last Verified: June 1, 2017

More Information

Terms related to this study

• Keywords: intervention; evaluation; implementation; research methodology
• Additional Relevant MeSH Terms: Pathologic Processes; Disease Attributes; Chronic Disease
• Other Study ID Numbers: SHP 08-156