Participatory System Dynamics vs Usual Quality Improvement: Staff Use of Simulation as an Effective, Scalable and Affordable Way to Improve Timely Mental Health Care?

Participatory System Dynamics vs Usual Quality Improvement: Is Staff Use of Simulation an Effective, Scalable and Affordable Way to Improve Timely Veteran Access to High-quality Mental Health Care?

Evidence-based VA care is best for meeting Veterans' mental health needs, such as depression, PTSD and opioid use disorder, to prevent suicide or overdose. But some key evidence-based practices only reach 3-28% of patients. Participatory system dynamics (PSD) helps improve quality with existing resources, critical in mental health and all VA health care. PSD uses learning simulations to improve staff decisions, showing how goals for quality can best be achieved given local resources and constraints. This study aims to significantly increase the proportion of patients who start and complete evidence-based care, and determine the costs of using PSD for improvement. Empowering frontline staff with PSD simulation encourages safe 'virtual' prototyping of complex changes to scheduling, referrals and staffing, before translating changes to the 'real world.' This study determines if PSD increases Veteran access to the highest quality care, and if PSD better maximizes VA resources when compared against usual trial-and-error approaches to improving quality.

Study Overview

• Status: Completed
• Conditions: Depression; PTSD; Opioid Use Disorder
• Intervention / Treatment: Behavioral: Modeling to Learn (MTL); Behavioral: Usual quality improvement (QI)

Detailed Description

Background: Evidence-based practices (EBPs) are the most high value treatments to meet Veterans' addiction and mental health needs, reduce chronic impairment, and prevent suicide or overdose. Over 10 years, VA invested in dissemination of evidence-based psychotherapies and pharmacotherapies based on substantial evidence of effectiveness as compared to usual care. Quality metrics also track progress. Despite these investments, patients with prevalent needs, such as depression, PTSD and opioid use disorder often don't receive EBPs. Systems theory explains limited EBP reach as a system behavior emerging dynamically from local components (e.g., patient demand/health service supply). Participatory research and engagement principles guide participatory system dynamics (PSD), a mixed-methods approach used in business and engineering, shown to be effective for improving quality with existing resources.

Significance/Impact: This study is proposed in the high priority area of VA addiction and mental health care to improve Veteran access to VA's highest quality care. The PSD program, Modeling to Learn (MTL), improves frontline management of dynamic complexity through simulations of staffing, scheduling and service referrals common in healthcare, across generalist and specialty programs, patient populations, and provider disciplines/treatments.

Innovation: Recent synthesis of VA data in the enterprise-wide SQL Corporate Data Warehouse (CDW) makes it feasible to scale participatory simulation learning activities with VA frontline addiction and mental health staff. MTL is an advanced quality improvement (QI) infrastructure that helps VA take a major step toward becoming a learning health care system, by empowering local multidisciplinary staff to develop change strategies that fit to local capacities and constraints. Model parameters are from one VA source and generic across health services. If findings show that MTL is superior to usual VA quality improvement activities of data review with facilitators from VA program offices, this paradigm could prove useful across VA services. The PSD approach also advances implementation science. Systems theory explains how dynamic system behaviors (EBP reach) are defined by general scientific laws, yet arise from idiographic local conditions. Empowering staff with systems science simulation encourages the safe prototyping of ideas necessary for learning, increasing ongoing quality improvement capacities, and saving time and money as compared to trial-and-error approaches.

Specific Aims:

1. Effectiveness: Test for superiority of MTL over usual QI for increasing the proportion of patients (1a) initiating, and (1b) completing a course of evidence-based psychotherapy (EBPsy) and evidence-based pharmacotherapy (EBPharm).
2. Scalable: (2a) Evaluate usual QI and MTL fidelity. (2b) Test MTL fidelity for convergent validity with participatory measures. (2c) Test the participatory theory of change: Evaluate whether 12 month period EBP reach is mediated by team scores on participatory measures.
3. Affordable: (3a) Determine the budget impact of MTL. (3b). Calculate the average marginal costs per 1% increase in EBP reach.

Methodology: This study proposes a two-arm, 24-clinic (12 per arm) cluster randomized trial to test for superiority of MTL over usual QI for increasing EBP reach. Clinics will be from 24 regional health care systems (HCS) below the SAIL mental health median, and low on 3 of 8 SAIL measures associated with EBPs. Computer-assisted stratified block randomization will balance MTL and usual QI arms at baseline using Corporate Data Warehouse (CDW) data. Participants will be the multidisciplinary frontline teams of addiction and mental health providers.

Next Steps/Implementation: MTL was developed in partnership with the VA Office of Mental Health and Suicide Prevention (OMHSP) and if shown to be effective, scalable, and affordable for improving timely Veteran access to EBPs, MTL will be scaled nationally to more clinics by expanding MTL online resources, and training more VA staff to facilitate MTL activities instead of usual QI.

• Study Type: Interventional
• Enrollment (Actual): 720
• Phase: Not Applicable

Contacts and Locations

Study Locations

• United States
  • California
    • Palo Alto, California, United States, 94304-1207
      • VA Palo Alto Health Care System, Palo Alto, CA

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: 18 years and older (Adult, Older Adult)
• Accepts Healthy Volunteers: Yes

Description

Inclusion Criteria:

24 health care systems currently functioning below the median VA mental health recommendations for Strategic Analytics for Improvement & Learning (SAIL) and below the median for 3 of 8 SAIL evidence-based treatment approaches.

• VA divisions and community-based outpatient clinics (CBOCs) or 'clinics' from regional VA health systems
• Must be below the overall VA quality median (as assessed by the Strategic Analytics for Improvement and Learning or SAIL), which includes 3 of 8 SAIL measures associated with four evidence-based psychotherapies and three evidence-based pharmacotherapies for depression, PTSD, and opioid use disorder.

Exclusion Criteria:

Health care systems functioning above median VA mental health recommendations for Strategic Analytics for Improvement & Learning (SAIL) and below the median for 3 of 8 SAIL evidence-based treatment approaches. Only one health care system can be included per arm - MTL vs QI.

• clinics with less than 12 months of data in 2018
• clinics involved in Office of Veterans Access to Care (OVACS) quality improvement program at baseline
• clinics where the VA Cerner electronic health record (EHR) implementation rollout will occur during the project period (Veterans Integrated Services Networks (VISNs) 20, 21 ,22, and 7)
• clinics who serve less than 122 unique patients each month on average
• clinics without an onsite multidisciplinary team of mental health or addiction service providers (minimum required: 1 psychiatrist, 1 psychologist, 1 social worker onsite)

Study Plan

How is the study designed?

Design Details

• Primary Purpose: Health Services Research
• Allocation: Randomized
• Interventional Model: Parallel Assignment
• Masking: None (Open Label)

Number of Arms

2

Arms and Interventions

Participant Group / Arm	Intervention / Treatment
Experimental: Modeling to Learn (MTL); 12 clinics randomly assigned to MTL	Behavioral: Modeling to Learn (MTL); Modeling to Learn is a facilitated health care quality improvement or evidence-based practice implementation strategy that includes frontline addiction and mental health staff running simulations of clinic improvement strategies to find the best approaches for improving the reach of evidence-based psychotherapy and evidence-based pharmacotherapy.
Experimental: Usual quality improvement (QI); 12 clinics randomly assigned to usual QI	Behavioral: Usual quality improvement (QI); Usual quality improvement is a health care quality improvement or evidence-based practice implementation strategy that includes frontline addiction and mental health staff reviewing data to find the best approaches for improving the reach of evidence-based psychotherapy and evidence-based pharmacotherapy.

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Proportion of patients initiating and completing a course of evidence-based psychotherapy (EBPsy) or evidence-based pharmacotherapy (EBPharm)	Proportion evidence-based practice (EBP) reach is defined as the proportion of VA outpatient addiction and mental health patients who receive evidence-based psychotherapy and/or evidence-based pharmacotherapy for opioid use disorder, depression, or PTSD in routine outpatient VA care.	Pre-/Post- 12-month period average of EBP reach (24 months total observation)]
Number of completed EBPsy templates during sessions with a relevant CPT code	Proportion of 3 EBPsy treatments for depression - Cognitive Behavior Therapy (CBT-D), Acceptance and Commitment Therapy (ACT), Interpersonal Psychotherapy (IPT) 2 EBPsy for PTSD - Prolonged Exposure (PE), and Cognitive Processing Therapy (CPT)	Pre-/Post- 12-month period average of EBP reach (24 months total observation)]
Number of combination of prescriptions placed with the VA pharmacy and sessions with a relevant CPT code	Proportion of 2 EBPharm treatments for depression - 84 and 180 days therapeutic continuity at new antidepressant start and 2 EBPharm for Opioid Use Disorder (OUD) - methadone and buprenorphine	Pre-/Post- 12-month period average of EBP reach (24 months total observation)]

Secondary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Differences in team perceptions of MTL and QI assessed by the Acceptability of Intervention Measure (AIM)	Assesses differences in team perceptions of MTL and QI using the 4 item survey 'Acceptability of Intervention Measure (AIM)'. Scale: 1-5, in 1 point increments (1 = completely disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, 5 = completely agree)	at 6 months
Differences in team perceptions of MTL and QI assessed by the Intervention Appropriateness Measure (IAM)	Assesses differences in team perceptions of MTL and QI using the 4 item survey 'Intervention Appropriateness Measure (IAM)'. Scale: 1-5, in 1 point increments (1 = completely disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, 5 = completely agree)	at 6 months
Differences in team perceptions of MTL and QI assessed by the Feasibility of Intervention Measure (FIM)	Assesses differences in team perceptions of MTL and QI using the 4 item survey 'Feasibility of Intervention Measure (FIM)'. Scale: 1-5, in 1 point increments (1 = completely disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, 5 = completely agree)	at 6 months
Patient Aligned Care Team Burnout Measure (PACT)	Quality of work satisfaction and burnout in a 4-item descriptive measure from VA team-based primary care that tracks 1) years of experience with the team, 2) working on more than one team, 3) turnover/change in team staff, 4) team overwork, and the single-item 5) self-reported burnout (sensitivity 83.2% and specificity 87.4%). Higher turnover numbers and high reported levels of burn out are considered negative.	At baseline and 6 months
Participatory Measure: Context	Assesses differences in MTL and QI decision context Scale: 1-5 (1 = VA Central Office, 2 = VA Facility Leadership, 3 = Clinic Managers, 4 = Our team, 5 = Individual providers on our team)	At baseline and 6 months
Participatory Measure: Partnership Structural Values	Assess differences in MTL and QI partnership structural values in a 22-item measure with subscales a) partner focus, b) core values, c) participation, d) cooperation, e) respect, and f) trust Subscale a) partner focus Scale: 1-5, in 1 point increments (1 = not at all , 5 = to a great extent) Alpha = 0.82 Subscale b-f) core values, participation, cooperation, respect, trust) Scale: 1-5, in 1 point increments (1 = strongly disagree, 5 = strongly agree) Subscale b) Alpha = 0.89 Subscale c) Alpha = 0.78 Subscale d) Alpha = 0.83 Subscale e) Alpha = 0.83 Subscale f) Alpha = 0.86	At 6 months
Participatory Measure: Relationships	Assess differences in MTL and QI relationships in a 15-item measure with subscales a) participatory decision-making, b) leadership, and c) use of resources Subscale a) participatory decision-making Scale: 1-5, in 1 point increments (1 = never, 5 = always) Alpha = 0.83 Subscale b) leadership Scale: 1-5, in 1 point increments (1 = very ineffective, 5 = very effective) Alpha = 0.94 Subscale c) use of resources Scale: 1-5, in 1 point increments (1 = makes poor use, 5 = makes excellent use) Alpha = n/a	At 6 months
Participatory Measure: Synergy	Assess differences in MTL and QI synergy in a 5-item measure. Scale: 1-5, in 1 point increments (1 = not at all, 5 = to a great extent) Alpha = 0.90	At 6 months
Participatory Measure: Capacity-Building Index	Assess differences in MTL and QI capacity-building index in a 5-item measure Scale: 1-5, in 1 point increments (1 = Not at all, 2 = Very Little, 3 = Somewhat, 4 = To a Large Extent, 5 = To a Very Great Extent) Alpha = 0.90	At 6 months
Facilitator Quality: Engagement Principles	10-item engagement principles measure that assesses investigator readiness to conduct participatory implementation science research. Assesses team and co-facilitator self-ratings of co-facilitators' use of engagement principles, such as building trusting relationships, knowledge of local conditions, and support for existing local capacities Scale: 1-5, in 1 point increments (1 = strongly disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, 5 = strongly agree) Response options range from 1 (strongly disagree) to 5 (strongly agree). Items will be summed for analyses, and the investigators will evaluate for convergence/divergence across facilitator and team ratings	At 6 months
MTL Fidelity Checklist for 12-Session Plan	Track MTL arm fidelity to 12-Session plan resources, sessions, online outputs, and standardized weekly emails	Throughout 6 months
QI Fidelity Checklist for 12-Session Plan	Track QI arm fidelity to 12-Session plan resources, sessions, online outputs, and standardized weekly emails	Throughout 6 months
Quality Improvement Activity Tracking	Tracking form adapted from a current VA operations-focused, implementation facilitation trial by the VA Team-Based Behavioral Health QUERI Program with four strengths specific to our study of: 1) assessment of activity costs readily comparable to other another VA multisite trial, 2) measure from Behavioral Health Interdisciplinary Program (BHIP) Enhancement Project, team-focused MH care, like PSD, 3) emphasis on quantifying a) staff and b) facilitator time, rather than categorizing content, 4) prior use in VA.	Throughout 6 months
Demographic Measures	4 item measure assessing ethnic (Hispanic, Latino, Latina, or Latinx), racial (American Indian/Alaska Native, Asian, Native Hawaiian or Other Pacific Islander, Black or African American, White, More than One Race) and gender (Man, Woman, Non-binary) identity of respondent. All items include a "Prefer not to say" option. Categories for demographic measures determined based on NIH reporting standards.	At baseline and 6 months

Collaborators and Investigators

• Sponsor: VA Office of Research and Development
• Investigators: Principal Investigator: Lindsey E. Zimmerman, PhD, VA Palo Alto Health Care System, Palo Alto, CA

Publications and helpful links

Helpful Links

• Home page of Modelling To Learn
• Modelling to Learn simulation
• Modelling to Learn facilitator page
• Modelling to Learn data page
• Modeling to Learn power bi data UI test environment

Study record dates

Study Major Dates

• Study Start (Actual): July 22, 2021
• Primary Completion (Actual): January 31, 2026
• Study Completion (Actual): January 31, 2026

Study Registration Dates

• First Submitted: December 18, 2019
• First Submitted That Met QC Criteria: December 18, 2019
• First Posted (Actual): December 23, 2019

Study Record Updates

• Last Update Posted (Actual): May 11, 2026
• Last Update Submitted That Met QC Criteria: May 6, 2026
• Last Verified: May 1, 2026

More Information

Terms related to this study

• Keywords: addiction; implementation science; quality improvement; modeling to learn; usual quality improvement; evidence-based psychotherapy; evidence-based pharmacotherapy; mental health/behavioral health
• Additional Relevant MeSH Terms: Trauma and Stressor Related Disorders; Narcotic-Related Disorders; Mental Disorders; Behavioral Symptoms; Substance-Related Disorders; Chemically-Induced Disorders; Compulsive Behavior; Impulsive Behavior; Stress Disorders, Traumatic; Behavior; Personal Satisfaction; Opioid-Related Disorders; Depression; Stress Disorders, Post-Traumatic; Behavior, Addictive; Psychological Well-Being
• Other Study ID Numbers: IIR 17-294; 12760065 (Other Grant/Funding Number: VA Office of Research & Development)

Plan for Individual participant data (IPD)

• Plan to Share Individual Participant Data (IPD)?: NO

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No
• product manufactured in and exported from the U.S.: No