Local Participatory Systems Dynamics to Increase Reach of Evidence Based Addiction and Mental Health Care

Participatory System Dynamics vs Audit and Feedback: A Cluster Randomized Trial of Mechanisms of Implementation Change to Expand Reach of Evidence-based Addiction and Mental Health Care

The most common reasons Veterans seek VA addiction and mental health care is for help with opioid and alcohol misuse, depression and PTSD. Research evidence has established highly effective treatments that prevent relapse, overdose and suicide, but even with policy mandates, performance metrics, and electronic health records to fix the problem, these treatments may only reach 3-28% of patients. This study tests participatory business engineering methods (Participatory System Dynamics) that engage patients, providers and policy makers against the status quo approaches, such as data review, and will determine if participatory system dynamics works, why it works, and whether it can be applied in many health care settings to guarantee patient access to the highest quality care and better meet the addiction and mental health needs of Veterans and the U.S. population.

Study Overview

• Status: Enrolling by invitation
• Conditions: Depression; PTSD; Opioid Use Disorder; Alcohol Use Disorder
• Intervention / Treatment: Other: Participatory System Dynamics (PSD); Other: Audit and Feedback (AF)

Detailed Description

The broad aim is to empower all healthcare stakeholders to provide the highest quality care to all patients. The specific aims address the complexities and tradeoffs associated with implementing evidence-based practices (EBPs) in outpatient addiction and mental health systems. There is scientific consensus about the best evidence-based psychotherapies and pharmacotherapies (EBPs) to meet the needs of patients with opioid and alcohol use disorder, PTSD and depression. However, EBP coordination over time, within and across multidisciplinary teams of providers, is complex and constantly changing. Veterans Health Administration (VA) policy mandates, national training programs, and incentivized quality measures, have been insufficient for reaching more than 3 to 28% of patients with the highest quality treatments. In fact, limited EBP reach is common in health systems and the field of implementation science seeks to address it. One routine strategy is data auditing with provider feedback (audit-and-feedback; AF), however, the impact is highly variable. As an alternative, participatory system dynamics (PSD) has been used to explain causes of complex problems in business management for 60 years. Partnering with frontline staff using PSD to determine how EBP reach emerges from local resources and constraints, and is determined by system dynamics, such as delays and feedback. The dynamics of EBP reach were formally specified in differential equation models, and tested against VA data drawn from a national VA SQL database. Using existing enterprise data to tailor model parameters to each care team. PSD models were made accessible via a 'Modeling to Learn' interface and training, during which teams safely evaluated local change scenarios via simulation to find the highest yield options for meeting Veterans' needs. PSD learning simulations produce immediate, real-time feedback to the teams who coordinate care, improving day-to-day decisions and long-term improvement plans. The study design is a two-arm, 24-site (12 sites/arm) cluster randomized trial to test the effectiveness of PSD simulation as compared to more standard team AF data review. The hypothesis is that PSD will be superior to AF for improving EBP initiation and dose (Aim 1). The investigators will test the PSD theory of change that the effect of PSD on improved EBP reach is explained by improvement in team systems thinking (Aim 2). To confirm the potential for widespread usefulness of PSD, by also testing the generalizability of PSD causal dynamics across PSD and AF arms (Aim 3). This study has the potential to inform a new paradigm, by determining what works to improve health system quality defined as EBP reach, why it works, and under what conditions. If PSD is effective, study activities will address a national priority to improve Veterans' addiction and mental health care to prevent chronic symptoms, relapse, suicide and overdose. Findings from the proposed tests of effectiveness, causality, and generality, could also catalyze future applications to make a significant public health impact across the continuum of healthcare.

• Study Type: Interventional
• Enrollment (Anticipated): 720
• Phase: Not Applicable

Contacts and Locations

Study Locations

• United States
  • California
    • Palo Alto, California, United States, 94304
      • VA Palo Alto Health Care System

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: 18 years and older (ADULT, OLDER_ADULT)
• Accepts Healthy Volunteers: Yes
• Genders Eligible for Study: All

Description

Inclusion Criteria Clinics:

• VA divisions and community-based outpatient clinics (CBOCs) or 'clinics' from regional VA health systems
• Must be below the overall VA quality median (as assessed by the Strategic Analytics for Improvement and Learning or SAIL), which includes 3 of 8 SAIL measures associated with four evidence-based psychotherapies and three evidence-based pharmacotherapies for depression, PTSD, and opioid use disorder.

Exclusion Criteria Clinics:

• clinics with less than 12 months of data in 2018
• clinics already involved in Office of Veterans Access to Care (OVACS) quality improvement program at baseline.
• clinics where the VA Cerner electronic health record (EHR) implementation rollout will occur during the project period (Veterans Integrated Services Networks (VISNs) 20, 21 ,22, and 7)
• clinics who serve less than 122 unique patients each month on average
• clinics without an onsite multidisciplinary team of mental health or addiction service providers (minimum required: 1 psychiatrist, 1 psychologist, 1 social worker onsite)

Study Plan

How is the study designed?

Design Details

• Primary Purpose: HEALTH_SERVICES_RESEARCH
• Allocation: RANDOMIZED
• Interventional Model: PARALLEL
• Masking: NONE

Number of Arms

2

Arms and Interventions

Participant Group / Arm	Intervention / Treatment
EXPERIMENTAL: Participatory System Dynamics (PSD); 12 clinics assigned to PSD	Other: Participatory System Dynamics (PSD); Participatory system dynamics is a facilitated health care quality improvement or evidence-based practice implementation strategy that includes frontline addiction and mental health staff running simulations of clinic improvement strategies to find the best approaches for improving the reach of evidence-based psychotherapy and evidence-based pharmacotherapy.
EXPERIMENTAL: Audit and Feedback (AF); 12 clinics assigned to AF	Other: Audit and Feedback (AF); Audit and feedback is a health care quality improvement or evidence-based practice implementation strategy that includes frontline addiction and mental health staff reviewing clinical care team data to find the best approaches for improving the reach of evidence-based psychotherapy and evidence-based pharmacotherapy.

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Proportion of patients diagnosed with alcohol use disorder, depression, opioid use disorder, or PTSD who meet evidence-based psychotherapy and pharmacotherapy initiation and course measures divided by total number of patients with these diagnoses	Initiation of an evidence-based practice is indicated by an evidence-based psychotherapy template or evidence-based pharmacotherapy prescription after intake. Adequate course is based on receiving an adequate number of evidence-based psychotherapy sessions to be a "completer" (typically 8 sessions) or enough refills for a guideline-recommended adequate trial of each medication (varies by medication). Data is gathered based on electronic health record data from the VA Corporate Data Warehouse (CDW).	Pre-/Post- 12-month period average of evidence-based practice reach (24 months total observation)
Proportion of completed evidence-based practice templates during sessions with a relevant CPT code	We will study 5 evidence-based psychotherapies: 3 for depression (Cognitive Behavior Therapy (CBT-D), Acceptance and Commitment Therapy (ACT), and Interpersonal Psychotherapy (IPT)) and 2 for PTSD (Prolonged Exposure (PE) and Cognitive Processing Therapy (CPT)). Data is gathered based on electronic health record data from the Corporate Data Warehouse (CDW).	Pre-/Post- 12-month period average of evidence-based practice reach (24 months total observation)
Proportion of combination of prescriptions placed with the VA pharmacy and sessions with a relevant CPT code	We will study 8 evidence-based pharmacotherapies: 2 for depression (84 and 180 days therapeutic continuity at new antidepressant start), 2 for Opioid Use Disorder (OUD) (methadone and buprenorphine), and 4 for Alcohol Use Disorder (AUD) (Acamprosate, Disulfiram, Naltrexone, and Topiramate). Data is gathered based on electronic health record data from the Corporate Data Warehouse (CDW).	Pre-/Post- 12-month period average of evidence-based practice reach (24 months total observation)

Secondary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Demographic Measures	4 item measure assessing ethnic (Hispanic, Latino, Latina, or Latinx), racial (American Indian/Alaska Native, Asian, Native Hawaiian or Other Pacific Islander, Black or African American, White, More than One Race) and gender (Man, Woman, Non-binary) identity of respondent. All items include a "Prefer not to say" option. Categories for demographic measures determined based on NIH reporting standards.	At baseline and 6 months
Degree of acceptability of intervention assessed by the Acceptability of Intervention Measure (AIM) [followed by its scale information in the Description]	Assesses degree of differences in team perceptions of PSD and AF on a survey with 4 items. Scale: 1-5, in 1 point increments (1 = completely disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, 5 = completely agree)	At 6 months
Degree of appropriateness of intervention assessed by the Intervention Appropriateness Measure (IAM) [followed by its scale information in the Description]	Assesses degree of for differences in team perceptions of PSD and AF on a survey with 4 items. Scale: 1-5, in 1 point increments (1 = completely disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, 5 = completely agree)	At 6 months
Degree of feasibility of intervention assessed by the Feasibility of Intervention Measure (FIM) [followed by its scale information in the Description]	Assesses degree of differences in team perceptions of PSD and AF on a survey with 4 items. Scale: 1-5, in 1 point increments (1 = completely disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, 5 = completely agree)	At 6 months
Patient Aligned Care team Burnout Measure (PACT) [followed by its scale information in the Description]	Quality of work satisfaction and burnout in a 4-item descriptive survey with measures from VA team-based primary care that tracks 1) years of experience with the team, 2) working on more than one team, 3) turnover/change in team staff, 4) team overwork, and the single-item 5) self-reported burnout (sensitivity 83.2% and specificity 87.4%) (Question 1) Answered in # of years (Question 2-3) Yes or No (Question 4-5) Scale: 1-5, in 1 point increments (1 = Never, 2 = Rarely, 3 = Sometimes, 4 = Very Often, 5 = Always)	At baseline and 6 months
Learning Organization Survey (LOS-27) [followed by its scale information in the Description]	Psychological safety in the workplace using a 27-item survey developed out of the learning organization tradition and demonstrated to have good psychometric properties during VA validation which assesses 7 clinic context factors: a) supportive learning environment (including psychological safety), b) leadership that reinforces learning, c) experimentation, d) training, e) knowledge acquisition, f) time for reflection, and g) performance monitoring (Questions 1-23) Scale: 0-4, in 1 point increments (0 = Never, 4 = Always) (Questions 24-27) Scale: 0 to 7, in 1 point increments (0 = Highly Inaccurate, 7 = Highly Accurate)	At baseline and 6 months
Team Decision Making Questionnaire (TDMQ) [followed by its scale information in the Description]	Team dynamics in the workplace using a four factor scale survey validated to assess the impact of a team intervention on team decision-making, support learning and development of quality services Scale: 1-7, in 1 point increments incl N/A (N/A = Not Applicable, 1 = Not at all, 2 = To a very small extent, 3 = To a small extent, 4 = To a moderate extent, 5 = To a great extent, 6 = To a very great extent, 7 = To a vast extent)	At 6 months
Systems Thinking Scale (STS) [followed by its scale information in the Description]	Use of systems thinking in the work place and the ability to recognize, understand, and synthesize interactions and interdependencies, including how actions and components can reinforce or counteract each other. Scale: 1-5, in 1 point increments (1 = Never, 2 = Seldom, 3 = Some of the time, 4 = Often, 5 = Most of the time)	At baseline and 6 months
Systems Thinking Codebook and Session Observations [followed by its scale information in the Description]	Observation of systems thinking in language/explanations, and performance demonstrating system thinking skills (competence) measured on four constructs: Complex, Feedback, Behavior, Time Scale: Level 1-4 in 1 point increments (1 = Construct is demonstrated at most simple level, 4 = Construct is fully demonstrated at most complex level)	Over 6 months
Facilitator Fidelity to Intervention Guides and Theory of Change	Review fidelity with qualitative checks against AF/PSD facilitator scripts for session learning objectives, 'key idea' and 'definitions,' including tracking the proportion of AF/PSD session activities (in minutes) on these components.	Over 6 months

Collaborators and Investigators

• Sponsor: Palo Alto Veterans Institute for Research
• Investigators: Principal Investigator: Lindsey E Zimmerman, PhD, National Center for PTSD, Dissemination & Training Division

Publications and helpful links

General Publications

• Bradley R, Greene J, Russ E, Dutra L, Westen D. A multidimensional meta-analysis of psychotherapy for PTSD. Am J Psychiatry. 2005 Feb;162(2):214-27. doi: 10.1176/appi.ajp.162.2.214. Erratum In: Am J Psychiatry. 2005 Apr;162(4):832. Am J Psychiatry. 2006 Feb;163(2):330.
• Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Health. 2011 Jan;38(1):4-23. doi: 10.1007/s10488-010-0327-7.
• Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999 Sep;89(9):1322-7. doi: 10.2105/ajph.89.9.1322.
• Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009 Aug 7;4:50. doi: 10.1186/1748-5908-4-50.
• Elbogen EB, Wagner HR, Johnson SC, Kinneer P, Kang H, Vasterling JJ, Timko C, Beckham JC. Are Iraq and Afghanistan veterans using mental health services? New data from a national random-sample survey. Psychiatr Serv. 2013 Feb 1;64(2):134-41. doi: 10.1176/appi.ps.004792011.
• Hermes ED, Hoff R, Rosenheck RA. Sources of the increasing number of Vietnam era veterans with a diagnosis of PTSD using VHA services. Psychiatr Serv. 2014 Jun 1;65(6):830-2. doi: 10.1176/appi.ps.201300232.
• Mott JM, Grubbs KM, Sansgiry S, Fortney JC, Cully JA. Psychotherapy Utilization Among Rural and Urban Veterans From 2007 to 2010. J Rural Health. 2015 Summer;31(3):235-43. doi: 10.1111/jrh.12099. Epub 2014 Dec 3.
• Hankin CS, Spiro A 3rd, Miller DR, Kazis L. Mental disorders and mental health treatment among U.S. Department of Veterans Affairs outpatients: the Veterans Health Study. Am J Psychiatry. 1999 Dec;156(12):1924-30. doi: 10.1176/ajp.156.12.1924.
• Hoggatt KJ, Williams EC, Der-Martirosian C, Yano EM, Washington DL. National prevalence and correlates of alcohol misuse in women veterans. J Subst Abuse Treat. 2015 May;52:10-6. doi: 10.1016/j.jsat.2014.12.003. Epub 2014 Dec 19.
• Fulton JJ, Calhoun PS, Wagner HR, Schry AR, Hair LP, Feeling N, Elbogen E, Beckham JC. The prevalence of posttraumatic stress disorder in Operation Enduring Freedom/Operation Iraqi Freedom (OEF/OIF) Veterans: a meta-analysis. J Anxiety Disord. 2015 Apr;31:98-107. doi: 10.1016/j.janxdis.2015.02.003. Epub 2015 Feb 19.
• Karlin BE, Cross G. From the laboratory to the therapy room: National dissemination and implementation of evidence-based psychotherapies in the U.S. Department of Veterans Affairs Health Care System. Am Psychol. 2014 Jan;69(1):19-33. doi: 10.1037/a0033888. Epub 2013 Sep 2.
• Karlin BE, Brown GK, Trockel M, Cunning D, Zeiss AM, Taylor CB. National dissemination of cognitive behavioral therapy for depression in the Department of Veterans Affairs health care system: therapist and patient-level outcomes. J Consult Clin Psychol. 2012 Oct;80(5):707-18. doi: 10.1037/a0029328. Epub 2012 Jul 23.
• Ruzek JI, Karlin BE, & Zeiss AM. Implementation of Evidence-Based Psychological Treatments in the Veterans Health Administration. In: McHugh RK, Barlow DH, eds. Dissemination of evidence-based psychological treatments. New York, NY: Oxford University Press. , 2012.
• Eftekhari A, Ruzek JI, Crowley JJ, Rosen CS, Greenbaum MA, Karlin BE. Effectiveness of national implementation of prolonged exposure therapy in Veterans Affairs care. JAMA Psychiatry. 2013 Sep;70(9):949-55. doi: 10.1001/jamapsychiatry.2013.36.
• Watts BV, Shiner B, Zubkoff L, Carpenter-Song E, Ronconi JM, Coldwell CM. Implementation of evidence-based psychotherapies for posttraumatic stress disorder in VA specialty clinics. Psychiatr Serv. 2014 May 1;65(5):648-53. doi: 10.1176/appi.ps.201300176.
• Department of Defense, & Department of Veterans Affairs. The management of MDD Working Group. VA/DOD clinical practice guideline for management of major depressive disorder (MDD). , 2009.
• Department of Veterans Affairs, & Department of Defense. VA/DoD Clinical practice guideline for the management of post-traumatic stress. , 2010.
• Department of Veterans Affairs, & Department of Defense. VA/DoD Clinical practice guideline for the management of substance use disorders. , 2009.
• Lemke S, Boden MT, Kearney LK, Krahn DD, Neuman MJ, Schmidt EM, Trafton JA. Measurement-based management of mental health quality and access in VHA: SAIL mental health domain. Psychol Serv. 2017 Feb;14(1):1-12. doi: 10.1037/ser0000097.
• Harris AH, Humphreys K, Bowe T, Kivlahan DR, Finney JW. Measuring the quality of substance use disorder treatment: evaluating the validity of the Department of Veterans Affairs continuity of care performance measure. J Subst Abuse Treat. 2009 Apr;36(3):294-305. doi: 10.1016/j.jsat.2008.05.011. Epub 2008 Oct 5.
• Rosen CS, Matthieu MM, Wiltsey Stirman S, Cook JM, Landes S, Bernardy NC, Chard KM, Crowley J, Eftekhari A, Finley EP, Hamblen JL, Harik JM, Kehle-Forbes SM, Meis LA, Osei-Bonsu PE, Rodriguez AL, Ruggiero KJ, Ruzek JI, Smith BN, Trent L, Watts BV. A Review of Studies on the System-Wide Implementation of Evidence-Based Psychotherapies for Posttraumatic Stress Disorder in the Veterans Health Administration. Adm Policy Ment Health. 2016 Nov;43(6):957-977. doi: 10.1007/s10488-016-0755-0.
• Seal KH, Maguen S, Cohen B, Gima KS, Metzler TJ, Ren L, Bertenthal D, Marmar CR. VA mental health services utilization in Iraq and Afghanistan veterans in the first year of receiving new mental health diagnoses. J Trauma Stress. 2010 Feb;23(1):5-16. doi: 10.1002/jts.20493.
• Mott JM, Mondragon S, Hundt NE, Beason-Smith M, Grady RH, Teng EJ. Characteristics of U.S. veterans who begin and complete prolonged exposure and cognitive processing therapy for PTSD. J Trauma Stress. 2014 Jun;27(3):265-73. doi: 10.1002/jts.21927.
• Harpaz-Rotem I, Rosenheck RA. Serving those who served: retention of newly returning veterans from Iraq and Afghanistan in mental health treatment. Psychiatr Serv. 2011 Jan;62(1):22-7. doi: 10.1176/ps.62.1.pss6201_0022.
• Lin LA, Bohnert AS, Ilgen MA, Pfeiffer PN, Ganoczy D, Blow FC. Outpatient Provider Contact Prior to Unintentional Opioid Overdose Among VHA Service Users. Psychiatr Serv. 2015 Nov;66(11):1149-54. doi: 10.1176/appi.ps.201400194. Epub 2015 Jul 1.
• Harris AH, Bowe T, Del Re AC, Finlay AK, Oliva E, Myrick HL, Rubinsky AD. Extended release naltrexone for alcohol use disorders: quasi-experimental effects on mortality and subsequent detoxification episodes. Alcohol Clin Exp Res. 2015 Jan;39(1):79-83. doi: 10.1111/acer.12597.
• Degenhardt L, Bucello C, Mathers B, Briegleb C, Ali H, Hickman M, McLaren J. Mortality among regular or dependent users of heroin and other opioids: a systematic review and meta-analysis of cohort studies. Addiction. 2011 Jan;106(1):32-51. doi: 10.1111/j.1360-0443.2010.03140.x. Epub 2010 Nov 4.
• Kaplan MS, Huguet N, McFarland BH, Newsom JT. Suicide among male veterans: a prospective population-based study. J Epidemiol Community Health. 2007 Jul;61(7):619-24. doi: 10.1136/jech.2006.054346. Erratum In: J Epidemiol Community Health. 2007 Aug;61(8):751.
• Desai RA, Dausey DJ, Rosenheck RA. Mental health service delivery and suicide risk: the role of individual patient and facility factors. Am J Psychiatry. 2005 Feb;162(2):311-8. doi: 10.1176/appi.ajp.162.2.311.
• Gradus JL, Suvak MK, Wisco BE, Marx BP, Resick PA. Treatment of posttraumatic stress disorder reduces suicidal ideation. Depress Anxiety. 2013 Oct;30(10):1046-53. doi: 10.1002/da.22117. Epub 2013 May 1.
• Hysong SJ. Meta-analysis: audit and feedback features impact effectiveness on care quality. Med Care. 2009 Mar;47(3):356-63. doi: 10.1097/MLR.0b013e3181893f6b.
• Foy R, Eccles MP, Jamtvedt G, Young J, Grimshaw JM, Baker R. What do we know about how to do audit and feedback? Pitfalls in applying evidence from a systematic review. BMC Health Serv Res. 2005 Jul 13;5:50. doi: 10.1186/1472-6963-5-50.
• Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, O'Brien MA, Johansen M, Grimshaw J, Oxman AD. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2012 Jun 13;(6):CD000259. doi: 10.1002/14651858.CD000259.pub3.
• Flottorp SA, Jamtvedt G, Gibis B, et al. Using audit and feedback to health professionals to improve the quality and safety of health care. World Health Organ. 2010.
• Ivers NM, Sales A, Colquhoun H, Michie S, Foy R, Francis JJ, Grimshaw JM. No more 'business as usual' with audit and feedback interventions: towards an agenda for a reinvigorated intervention. Implement Sci. 2014 Jan 17;9:14. doi: 10.1186/1748-5908-9-14.
• Colquhoun HL, Carroll K, Eva KW, Grimshaw JM, Ivers N, Michie S, Sales A, Brehaut JC. Advancing the literature on designing audit and feedback interventions: identifying theory-informed hypotheses. Implement Sci. 2017 Sep 29;12(1):117. doi: 10.1186/s13012-017-0646-0.
• Colquhoun H, Michie S, Sales A, Ivers N, Grimshaw JM, Carroll K, Chalifoux M, Eva K, Brehaut J. Reporting and design elements of audit and feedback interventions: a secondary review. BMJ Qual Saf. 2017 Jan;26(1):54-60. doi: 10.1136/bmjqs-2015-005004. Epub 2016 Jan 25.
• Landis-Lewis Z, Brehaut JC, Hochheiser H, Douglas GP, Jacobson RS. Computer-supported feedback message tailoring: theory-informed adaptation of clinical audit and feedback for learning and behavior change. Implement Sci. 2015 Jan 21;10:12. doi: 10.1186/s13012-014-0203-z.
• Atkinson JA, Wells R, Page A, Dominello A, Haines M, Wilson A. Applications of system dynamics modelling to support health policy. Public Health Res Pract. 2015 Jul 9;25(3):e2531531. doi: 10.17061/phrp2531531.
• Ng KY, Van Dyne L. Individualism-Collectivism as a Boundary Condition for Effectiveness of Minority Influence in Decision Making. Organ Behav Hum Decis Process. 2001 Mar;84(2):198-225. doi: 10.1006/obhd.2000.2927.
• Sterman JD. Modeling managerial behavior: Misperceptions of feedback in a dynamic decision making experiment. Manag Sci. 1989;35:321-339.
• Cronin MA, Gonzalez C, & Sterman JD. Why don't well-educated adults understand accumulation? A challenge to researchers, educators, and citizens. Organ Behav Hum Decis Process. 2009;108:116-130.
• Sterman JD. Learning from evidence in a complex world. Am J Public Health. 2006 Mar;96(3):505-14. doi: 10.2105/AJPH.2005.066043. Epub 2006 Jan 31.
• Sterman JD. Learning in and about complex systems. Syst Dyn Rev. 1994;10:291-330.
• Andersen DF, Vennix JA, Richardson GP, et al. Group model building: Problem structing, policy simulation and decision support. J Oper Res Soc. 2007;:691-694.
• Vennix JAM. Group model building: facilitating team learning using system dynamics. Chichester ; New York: J. Wiley, 1996: 1-297.
• Rouwette EAJA, Vennix JAM, & Mullekom T van. Group model building effectiveness: A review of assessment studies. Syst Dyn Rev. 2002;18:5-45.
• Bendoly E. System dynamics understanding in projects: Information sharing, psychological safety, and performance effects. Prod Oper Manag. 2014;23:1352-1369.
• Sterman JD. Does formal system dynamics training improve people's understanding of accumulation? Syst Dyn Rev. 2010;26:316-334.
• Simon HA. Bounded rationality and organizational learning. Organ Sci. 1991;2:125-134.
• Rahmandad H, Repenning N, & Sterman J. Effects of feedback delay on learning. Syst Dyn Rev. 2009;25:309-338.
• Zimmerman L, Lounsbury DW, Rosen CS, Kimerling R, Trafton JA, Lindley SE. Participatory System Dynamics Modeling: Increasing Stakeholder Engagement and Precision to Improve Implementation Planning in Systems. Adm Policy Ment Health. 2016 Nov;43(6):834-849. doi: 10.1007/s10488-016-0754-1. Erratum In: Adm Policy Ment Health. 2021 Mar;48(2):377.
• Oliva R. Model calibration as a testing strategy for system dynamics models. Eur J Oper Res. 2003;151:552-568.
• Trafton JA, Greenberg G, Harris AH, Tavakoli S, Kearney L, McCarthy J, Blow F, Hoff R, Schohn M. VHA mental health information system: applying health information technology to monitor and facilitate implementation of VHA Uniform Mental Health Services Handbook requirements. Med Care. 2013 Mar;51(3 Suppl 1):S29-36. doi: 10.1097/MLR.0b013e31827da836.
• Barlas Y. Formal aspects of model validity and validation in system dynamics. Syst Dyn Rev. 1996;12:183-210.
• Hirsch G, Homer J, Trogdon J, Wile K, Orenstein D. Using simulation to compare 4 categories of intervention for reducing cardiovascular disease risks. Am J Public Health. 2014 Jul;104(7):1187-95. doi: 10.2105/AJPH.2013.301816. Epub 2014 May 15.
• Lich KH, Tian Y, Beadles CA, Williams LS, Bravata DM, Cheng EM, Bosworth HB, Homer JB, Matchar DB. Strategic planning to reduce the burden of stroke among veterans: using simulation modeling to inform decision making. Stroke. 2014 Jul;45(7):2078-84. doi: 10.1161/STROKEAHA.114.004694. Epub 2014 Jun 12.
• Forrester, J.W. The model versus a modeling process. Syst Dyn Rev. 1985;:133-134.
• Derickson R, Fishman J, Osatuke K, Teclaw R, Ramsel D. Psychological safety and error reporting within Veterans Health Administration hospitals. J Patient Saf. 2015 Mar;11(1):60-6. doi: 10.1097/PTS.0000000000000082.
• Edmondson A. Psychological safety and learning behavior in work teams. Adm Sci Q. 1999;44:350-383.
• Campbell MJ, & Walters SJ. How to Design, Analyise and Report Cluster Randomised Trials in Medicine and Health Related Research. : Wiley, 2014: 1-264.
• Forrester JW. System dynamics, systems thinking, and soft OR. Syst Dyn Rev. 1994;10:245-256.
• Forrester JW. Some basic concepts in system dynamics. Sloan Sch Manag Mass Inst Technol Camb MA. 2009.
• Hong G. Ratio of mediator probability weighting for estimating natural direct and indirect effects. In: Proceedings of the American Statistical Association, Biometrics Section. : American Statistical Association Alexandria, VA, 2010: 2401-2415.
• Hong G, Deutsch J, & Hill HD. Ratio-of-Mediator-Probability Weighting for Causal Mediation Analysis in the Presence of Treatment-by-Mediator Interaction. J Educ Behav Stat. 2015;40:307-340.
• Qin X, & Hong G. Causal mediation analysis in multi-site trials: An application of ratio-of-mediatorprobability weighting to the Head Start Impact Study. JSM Proc Soc Stat Sect. 2014;:912-926.
• Homer JB. Partial-model testing as a validation tool for system dynamics (1983). Syst Dyn Rev. 2012;28:281-294.
• Rahmandad H, & Sterman JD. Reporting guidelines for simulation-based research in social sciences. Syst Dyn Rev. 2012;28:396-411.
• Singer SJ, Moore SC, Meterko M, Williams S. Development of a short-form Learning Organization Survey: the LOS-27. Med Care Res Rev. 2012 Aug;69(4):432-59. doi: 10.1177/1077558712448135. Epub 2012 May 25.
• Batorowicz B, Shepherd TA. Measuring the quality of transdisciplinary teams. J Interprof Care. 2008 Dec;22(6):612-20. doi: 10.1080/13561820802303664.
• Dolan ED, Mohr D, Lempa M, Joos S, Fihn SD, Nelson KM, Helfrich CD. Using a single item to measure burnout in primary care staff: a psychometric evaluation. J Gen Intern Med. 2015 May;30(5):582-7. doi: 10.1007/s11606-014-3112-6. Epub 2014 Dec 2.
• Helfrich CD, Simonetti JA, Clinton WL, Wood GB, Taylor L, Schectman G, Stark R, Rubenstein LV, Fihn SD, Nelson KM. The Association of Team-Specific Workload and Staffing with Odds of Burnout Among VA Primary Care Team Members. J Gen Intern Med. 2017 Jul;32(7):760-766. doi: 10.1007/s11606-017-4011-4. Epub 2017 Feb 23.
• Nelson KM, Helfrich C, Sun H, Hebert PL, Liu CF, Dolan E, Taylor L, Wong E, Maynard C, Hernandez SE, Sanders W, Randall I, Curtis I, Schectman G, Stark R, Fihn SD. Implementation of the patient-centered medical home in the Veterans Health Administration: associations with patient satisfaction, quality of care, staff burnout, and hospital and emergency department use. JAMA Intern Med. 2014 Aug;174(8):1350-8. doi: 10.1001/jamainternmed.2014.2488.
• Helfrich CD, Dolan ED, Simonetti J, Reid RJ, Joos S, Wakefield BJ, Schectman G, Stark R, Fihn SD, Harvey HB, Nelson K. Elements of team-based care in a patient-centered medical home are associated with lower burnout among VA primary care employees. J Gen Intern Med. 2014 Jul;29 Suppl 2(Suppl 2):S659-66. doi: 10.1007/s11606-013-2702-z.
• Weiner BJ, Lewis CC, Stanick C, Powell BJ, Dorsey CN, Clary AS, Boynton MH, Halko H. Psychometric assessment of three newly developed implementation outcome measures. Implement Sci. 2017 Aug 29;12(1):108. doi: 10.1186/s13012-017-0635-3.
• Sweeney LB, & Sterman JD. Bathtub dynamics: Initial results of a systems thinking inventory. Syst Dyn Rev. 2000;16:249-286.
• Liberati EG, Ruggiero F, Galuppo L, Gorli M, Gonzalez-Lorenzo M, Maraldi M, Ruggieri P, Polo Friz H, Scaratti G, Kwag KH, Vespignani R, Moja L. What hinders the uptake of computerized decision support systems in hospitals? A qualitative study and framework for implementation. Implement Sci. 2017 Sep 15;12(1):113. doi: 10.1186/s13012-017-0644-2.
• Centers for Medicare & Medicaid Services Alliance to Modernize Healthcare (CAMH). Independent Assessment of the Health Care Delivery Systems and Management Processes of the Department of Veterans Affairs (Volume 1: Integrated Report).
• Department of Veterans Affairs. Uniform mental health services in VA medical centers and clinics. Washington DC: Veterans Health Administration, 2008.
• Powers MB, Halpern JM, Ferenschak MP, Gillihan SJ, Foa EB. A meta-analytic review of prolonged exposure for posttraumatic stress disorder. Clin Psychol Rev. 2010 Aug;30(6):635-41. doi: 10.1016/j.cpr.2010.04.007. Epub 2010 May 2.
• Powers MB, Zum Vorde Sive Vording MB, Emmelkamp PM. Acceptance and commitment therapy: a meta-analytic review. Psychother Psychosom. 2009;78(2):73-80. doi: 10.1159/000190790. Epub 2009 Jan 14.
• Walser RD, Karlin BE, Trockel M, Mazina B, Barr Taylor C. Training in and implementation of Acceptance and Commitment Therapy for depression in the Veterans Health Administration: therapist and patient outcomes. Behav Res Ther. 2013 Sep;51(9):555-63. doi: 10.1016/j.brat.2013.05.009. Epub 2013 Jun 3.
• Stewart MO, Raffa SD, Steele JL, Miller SA, Clougherty KF, Hinrichsen GA, Karlin BE. National dissemination of interpersonal psychotherapy for depression in veterans: therapist and patient-level outcomes. J Consult Clin Psychol. 2014 Dec;82(6):1201-6. doi: 10.1037/a0037410. Epub 2014 Jul 21.
• Cuijpers P, Geraedts AS, van Oppen P, Andersson G, Markowitz JC, van Straten A. Interpersonal psychotherapy for depression: a meta-analysis. Am J Psychiatry. 2011 Jun;168(6):581-92. doi: 10.1176/appi.ajp.2010.10101411. Epub 2011 Mar 1. Erratum In: Am J Psychiatry. 2011 Jun;168(6):652.
• Fournier JC, DeRubeis RJ, Hollon SD, Dimidjian S, Amsterdam JD, Shelton RC, Fawcett J. Antidepressant drug effects and depression severity: a patient-level meta-analysis. JAMA. 2010 Jan 6;303(1):47-53. doi: 10.1001/jama.2009.1943.
• Knudsen HK, Roman PM. The Diffusion of Acamprosate for the Treatment of Alcohol Use Disorder: Results From a National Longitudinal Study. J Subst Abuse Treat. 2016 Mar;62:62-7. doi: 10.1016/j.jsat.2015.10.005. Epub 2015 Dec 1.
• Knudsen HK, Roman PM. Service delivery and pharmacotherapy for alcohol use disorder in the era of health reform: Data from a national sample of treatment organizations. Subst Abus. 2016;37(1):230-7. doi: 10.1080/08897077.2015.1028699. Epub 2015 Apr 20.
• Pettinati HM, O'Brien CP, Rabinowitz AR, Wortman SP, Oslin DW, Kampman KM, Dackis CA. The status of naltrexone in the treatment of alcohol dependence: specific effects on heavy drinking. J Clin Psychopharmacol. 2006 Dec;26(6):610-25. doi: 10.1097/01.jcp.0000245566.52401.20.
• Anton RF, O'Malley SS, Ciraulo DA, Cisler RA, Couper D, Donovan DM, Gastfriend DR, Hosking JD, Johnson BA, LoCastro JS, Longabaugh R, Mason BJ, Mattson ME, Miller WR, Pettinati HM, Randall CL, Swift R, Weiss RD, Williams LD, Zweben A; COMBINE Study Research Group. Combined pharmacotherapies and behavioral interventions for alcohol dependence: the COMBINE study: a randomized controlled trial. JAMA. 2006 May 3;295(17):2003-17. doi: 10.1001/jama.295.17.2003.
• Barnett PG, Rodgers JH, Bloch DA. A meta-analysis comparing buprenorphine to methadone for treatment of opiate dependence. Addiction. 2001 May;96(5):683-90. doi: 10.1046/j.1360-0443.2001.9656834.x.
• West SL, O'Neal KK, Graham CW. A meta-analysis comparing the effectiveness of buprenorphine and methadone. J Subst Abuse. 2000;12(4):405-14. doi: 10.1016/s0899-3289(01)00054-2.
• Hysong SJ, Kell HJ, Petersen LA, Campbell BA, Trautner BW. Theory-based and evidence-based design of audit and feedback programmes: examples from two clinical intervention studies. BMJ Qual Saf. 2017 Apr;26(4):323-334. doi: 10.1136/bmjqs-2015-004796. Epub 2016 Jun 10.
• Hysong SJ, Best RG, Pugh JA. Audit and feedback and clinical practice guideline adherence: making feedback actionable. Implement Sci. 2006 Apr 28;1:9. doi: 10.1186/1748-5908-1-9.
• Hysong SJ, Teal CR, Khan MJ, Haidet P. Improving quality of care through improved audit and feedback. Implement Sci. 2012 May 18;7:45. doi: 10.1186/1748-5908-7-45.
• Shiner B, D'Avolio LW, Nguyen TM, Zayed MH, Young-Xu Y, Desai RA, Schnurr PP, Fiore LD, Watts BV. Measuring use of evidence based psychotherapy for posttraumatic stress disorder. Adm Policy Ment Health. 2013 Jul;40(4):311-8. doi: 10.1007/s10488-012-0421-0.
• Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, Griffey R, Hensley M. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011 Mar;38(2):65-76. doi: 10.1007/s10488-010-0319-7.
• Brownson RC, Colditz GA, & Proctor EK. Dissemination and Implementation Research in Health: Translating Science to Practice. Oxford, UK: Oxford, 2012.
• Tabak RG, Khoong EC, Chambers DA, Brownson RC. Bridging research and practice: models for dissemination and implementation research. Am J Prev Med. 2012 Sep;43(3):337-50. doi: 10.1016/j.amepre.2012.05.024.
• Tabak RG, Khoong EC, Chambers D, et al. Models in dissemination and implementation research: useful tools in public health services and systems research. Front Public Health Serv Syst Res. 2013;2:8.
• Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, Proctor EK, Kirchner JE. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015 Feb 12;10:21. doi: 10.1186/s13012-015-0209-1.
• Chambers DA, Glasgow RE, Stange KC. The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change. Implement Sci. 2013 Oct 2;8:117. doi: 10.1186/1748-5908-8-117.
• Breuer E, Lee L, De Silva M, Lund C. Using theory of change to design and evaluate public health interventions: a systematic review. Implement Sci. 2016 May 6;11:63. doi: 10.1186/s13012-016-0422-6.
• Schierhout G, Hains J, Si D, Kennedy C, Cox R, Kwedza R, O'Donoghue L, Fittock M, Brands J, Lonergan K, Dowden M, Bailie R. Evaluating the effectiveness of a multifaceted, multilevel continuous quality improvement program in primary health care: developing a realist theory of change. Implement Sci. 2013 Oct 8;8:119. doi: 10.1186/1748-5908-8-119.
• Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci. 2015 Apr 21;10:53. doi: 10.1186/s13012-015-0242-0.
• Sales A, Smith J, Curran G, Kochevar L. Models, strategies, and tools. Theory in implementing evidence-based findings into health care practice. J Gen Intern Med. 2006 Feb;21 Suppl 2(Suppl 2):S43-9. doi: 10.1111/j.1525-1497.2006.00362.x.
• Damschroder LJ, & Lowery JC. Efficient Synthesis: Using Qualitative Comparative Analysis (QCA) and the CFIR across Diverse Studies. 2015.
• Baumgartner M, & Thiem A. Often Trusted but Never (Properly) Tested: Evaluating Qualitative Comparative Analysis. Sociol Methods Res. 2017;:4912411770148.
• Hug S. Qualitative Comparative Analysis: How Inductive Use and Measurement Error Lead to Problematic Inference. Polit Anal. 2013;21:252-265.
• Krogslund C, Choi DD, & Poertner M. Fuzzy Sets on Shaky Ground: Parameter Sensitivity and Confirmation Bias in fsQCA. Polit Anal. 2015;23:21-41.
• Braumoeller B. QCAfalsePositive: Tests for Type I Error in Qualitative Comparative Analysis (QCA). 2015.
• Powell BJ, Proctor EK, Glass JE. A Systematic Review of Strategies for Implementing Empirically Supported Mental Health Interventions. Res Soc Work Pract. 2014 Mar 1;24(2):192-212. doi: 10.1177/1049731513505778.
• Williams NJ. Multilevel Mechanisms of Implementation Strategies in Mental Health: Integrating Theory, Research, and Practice. Adm Policy Ment Health. 2016 Sep;43(5):783-798. doi: 10.1007/s10488-015-0693-2.
• Senge P. The Fifth Discipline. : Currency/Doubleday, 1990.
• Hysong SJ, Knox MK, Haidet P. Examining clinical performance feedback in Patient-Aligned Care Teams. J Gen Intern Med. 2014 Jul;29 Suppl 2(Suppl 2):S667-74. doi: 10.1007/s11606-013-2707-7.
• Hysong SJ, Thomas CL, Spitzmuller C, Amspoker AB, Woodard L, Modi V, Naik AD. Linking clinician interaction and coordination to clinical performance in Patient-Aligned Care Teams. Implement Sci. 2016 Jan 15;11:7. doi: 10.1186/s13012-015-0368-0.
• Richardson GP. Concept models in group model building: G. P. Richardson: Concept Models in Group Model Building. Syst Dyn Rev. 2013;29:42-55.
• Richardson GP. Reflections on the foundations of system dynamics: Foundations of System Dynamics. Syst Dyn Rev. 2011;27:219-243.
• Garcia HA, McGeary CA, Finley EP, et al. Burnout among psychiatrists in the Veterans Health Administration. Burn Res. 2015;2:108-114.
• Garcia HA, Finley EP, Ketchum N, Jakupcak M, Dassori A, Reyes SC. A survey of perceived barriers and attitudes toward mental health care among OEF/OIF veterans at VA outpatient mental health clinics. Mil Med. 2014 Mar;179(3):273-8. doi: 10.7205/MILMED-D-13-00076.
• Garcia HA, Kelley LP, Rentz TO, et al. Pretreatment predictors of dropout from cognitive behavioral therapy for PTSD in Iraq and Afghanistan war veterans. Psychol Serv. 2011;8:1-11.
• Senge, P.M. The fifth discipline: The art and practice of the learning organization. : Broadway Business, 2006.
• Meadows DH. Thinking in systems: A primer. : Chelsea Green Publishing, 2012.
• Senge PM, ed. The dance of change: the challenges of sustaining momentum in learning organizations, 1st ed. New York: Currency/Doubleday, 1999: 1-596.
• Sterman JD. Business Dynamics: Systems Thinking and Modeling for a Complex World. : McGraw-Hill Education, 2000: 1-1008.
• Rouwette EA, & Vennix JA. System dynamics and organizational interventions. Syst Res Behav Sci. 2006;23:451-466.
• Akhnif E, Macq J, Idrissi Fakhreddine MO, Meessen B. Scoping literature review on the Learning Organisation concept as applied to the health system. Health Res Policy Syst. 2017 Mar 1;15(1):16. doi: 10.1186/s12961-017-0176-x.
• Morecroft J, & Sherman J. Modeling for learning organizations. Portland OR: Productivity Press, 1994.
• Argyris C, & Schön DA. Organizational learning: A theory of action perspective. Reis. 1997;:345-348.
• Forrester JW. Industrial Dynamics. Cambridge, MA: MIT Press, 1961.
• Simon HA. Models of Man: Social and Rational-Mathematical Essays on Rational Human Behavior in a Social Setting. Oxford, UK: Wiley, 1957.
• Huz S, Andersen DF, Richardson GP, et al. A framework for evaluating systems thinking interventions: an experimental approach to mental health system change. Syst Dyn Rev. 1997;13:149-169.
• Evans JS. Dual-processing accounts of reasoning, judgment, and social cognition. Annu Rev Psychol. 2008;59:255-78. doi: 10.1146/annurev.psych.59.103006.093629.
• Evans JS. In two minds: dual-process accounts of reasoning. Trends Cogn Sci. 2003 Oct;7(10):454-9. doi: 10.1016/j.tics.2003.08.012.
• Evans JS, Stanovich KE. Dual-Process Theories of Higher Cognition: Advancing the Debate. Perspect Psychol Sci. 2013 May;8(3):223-41. doi: 10.1177/1745691612460685.
• Moore SM, Dolansky MA, Singh M, et al. The Systems Thinking Scale. 2010.
• Hovmand PS. Community Based System Dynamics. New York, NY: Springer New York, 2014.
• Minkler M, & Wallerstein N. Community-Based Participatory Research for Health: From Process to Outcomes. : John Wiley & Sons, 2011: 1-758.
• Case AD, Byrd R, Claggett E, DeVeaux S, Perkins R, Huang C, Sernyak MJ, Steiner JL, Cole R, LaPaglia DM, Bailey M, Buchanan C, Johnson A, Kaufman JS. Stakeholders' perspectives on community-based participatory research to enhance mental health services. Am J Community Psychol. 2014 Dec;54(3-4):397-408. doi: 10.1007/s10464-014-9677-8.
• Israel BA, Coombe CM, Cheezum RR, Schulz AJ, McGranaghan RJ, Lichtenstein R, Reyes AG, Clement J, Burris A. Community-based participatory research: a capacity-building approach for policy advocacy aimed at eliminating health disparities. Am J Public Health. 2010 Nov;100(11):2094-102. doi: 10.2105/AJPH.2009.170506. Epub 2010 Sep 23.
• Wallerstein NB, Duran B. Using community-based participatory research to address health disparities. Health Promot Pract. 2006 Jul;7(3):312-23. doi: 10.1177/1524839906289376. Epub 2006 Jun 7.
• Aarons GA. Measuring provider attitudes toward evidence-based practice: consideration of organizational context and individual differences. Child Adolesc Psychiatr Clin N Am. 2005 Apr;14(2):255-71, viii. doi: 10.1016/j.chc.2004.04.008.
• Aarons GA, Ehrhart MG, Farahnak LR, Sklar M. Aligning leadership across systems and organizations to develop a strategic climate for evidence-based practice implementation. Annu Rev Public Health. 2014;35:255-74. doi: 10.1146/annurev-publhealth-032013-182447.
• Aarons GA, Green AE, Trott E, Willging CE, Torres EM, Ehrhart MG, Roesch SC. The Roles of System and Organizational Leadership in System-Wide Evidence-Based Intervention Sustainment: A Mixed-Method Study. Adm Policy Ment Health. 2016 Nov;43(6):991-1008. doi: 10.1007/s10488-016-0751-4.
• Brimhall KC, Fenwick K, Farahnak LR, Hurlburt MS, Roesch SC, Aarons GA. Leadership, Organizational Climate, and Perceived Burden of Evidence-Based Practice in Mental Health Services. Adm Policy Ment Health. 2016 Sep;43(5):629-639. doi: 10.1007/s10488-015-0670-9.
• Senge P, & Forrester JW. Tests for building confidence in system dynamics models. 1980;14:209-228.
• Rahmandad H, & Sterman JD. Reporting guidelines for simulation-based research in social sciences: Reporting Guidelines for Simulation-Based Research. Syst Dyn Rev. 2012;28:396-411.
• Hovmand PS, Andersen DF, Rouwette E, et al. Group model-building "scripts" as a collaborative planning tool: Scripts as a collaborative planning tool. Syst Res Behav Sci. 2012;29:179-193.
• Andersen D, & Richardson G. Scripts for group model building. Syst Dyn Rev. 1997;13:107-129.
• Hovmand, Peter, S., et al. Scriptapedia 4.0.6. 2013.
• McCarthy JT, Hocum CL, Albright RC, Rogers J, Gallaher EJ, Steensma DP, Gudgell SF, Bergstralh EJ, Dillon JC, Hickson LJ, Williams AW, Dingli D. Biomedical system dynamics to improve anemia control with darbepoetin alfa in long-term hemodialysis patients. Mayo Clin Proc. 2014 Jan;89(1):87-94. doi: 10.1016/j.mayocp.2013.10.022.
• Gallaher E, Steensma DP, Chrisope TR, et al. Individualized Medicine and Biophysical System Dynamics: An Example from Clinical Practice in End Stage Renal Disease. 2011.
• Homer J. Levels of evidence in system dynamics modeling. Syst Dyn Rev. 2014;30:75-80.
• Oliva R, & Sterman JD. Death spirals and virtuous cycles. In: Handbook of Service Science. : Springer, 2010: 321-358.
• Snow G. Package "blockrand." 2015.
• Holowka DW, Marx BP, Gates MA, Litman HJ, Ranganathan G, Rosen RC, Keane TM. PTSD diagnostic validity in Veterans Affairs electronic records of Iraq and Afghanistan veterans. J Consult Clin Psychol. 2014 Aug;82(4):569-579. doi: 10.1037/a0036347. Epub 2014 Apr 14. Erratum In: J Consult Clin Psychol. 2014 Aug;82(4):579.
• Emmons KM, Weiner B, Fernandez ME, Tu SP. Systems antecedents for dissemination and implementation: a review and analysis of measures. Health Educ Behav. 2012 Feb;39(1):87-105. doi: 10.1177/1090198111409748. Epub 2011 Jul 1.
• Rabin BA, Lewis CC, Norton WE, Neta G, Chambers D, Tobin JN, Brownson RC, Glasgow RE. Measurement resources for dissemination and implementation research in health. Implement Sci. 2016 Mar 22;11:42. doi: 10.1186/s13012-016-0401-y.
• Lewis CC, Fischer S, Weiner BJ, Stanick C, Kim M, Martinez RG. Outcomes for implementation science: an enhanced systematic review of instruments using evidence-based rating criteria. Implement Sci. 2015 Nov 4;10:155. doi: 10.1186/s13012-015-0342-x.
• Glasgow RE. What does it mean to be pragmatic? Pragmatic methods, measures, and models to facilitate research translation. Health Educ Behav. 2013 Jun;40(3):257-65. doi: 10.1177/1090198113486805.
• Glasgow RE, Riley WT. Pragmatic measures: what they are and why we need them. Am J Prev Med. 2013 Aug;45(2):237-43. doi: 10.1016/j.amepre.2013.03.010.
• Singh MK, Ogrinc G, Cox KR, Dolansky M, Brandt J, Morrison LJ, Harwood B, Petroski G, West A, Headrick LA. The Quality Improvement Knowledge Application Tool Revised (QIKAT-R). Acad Med. 2014 Oct;89(10):1386-91. doi: 10.1097/ACM.0000000000000456.
• Rosseel Y, Oberski D, Byrnes J, et al. Package "lavaan" Version 0.5-23.1097. 2017.
• Shavelson RJ, & Webb NM. Generalizability theory: A primer. Newbury Park, CA: Sage., 1991.
• Millsap RE, Kwok OM. Evaluating the impact of partial factorial invariance on selection in two populations. Psychol Methods. 2004 Mar;9(1):93-115. doi: 10.1037/1082-989X.9.1.93.
• Wolak M. Package "ICC" Version 2.3.0. 2015.
• Bates D, Maechler M, Bolker B, et al. Package "lme4" Version 1.1-13. 2017.
• Rotondi MA. Package "CRTSize" Version 1.0. 2015.
• Rotondi MA, & Donner A. Sample Size Estimation in Cluster Randomized Educational Trials: An Empirical Bayes Approach. J Educ Behav Stat. 2009;34:229-237.
• Ventana Systems Inc. Vensim@ Version 6.3. 2014.
• Hyndman, R.J., & Khandakar, Y. Automatic time series forecasting: The forecast package for R. J Stat Softw. 2008;26.
• Hyndman R, O'Hara-Wild Mi, Bergmeir C, et al. Package "forecast" Version 8.0. 2017.
• Mott JM, Hundt NE, Sansgiry S, Mignogna J, Cully JA. Changes in psychotherapy utilization among veterans with depression, anxiety, and PTSD. Psychiatr Serv. 2014 Jan 1;65(1):106-12. doi: 10.1176/appi.ps.201300056.
• Oliva EM, Trafton JA, Harris AH, Gordon AJ. Trends in opioid agonist therapy in the Veterans Health Administration: is supply keeping up with demand? Am J Drug Alcohol Abuse. 2013 Mar;39(2):103-7. doi: 10.3109/00952990.2012.741167.
• Diaz M, Neuhauser D. Pasteur and parachutes: when statistical process control is better than a randomized controlled trial. Qual Saf Health Care. 2005 Apr;14(2):140-3. doi: 10.1136/qshc.2005.013763. No abstract available.
• Duclos A, Voirin N. The p-control chart: a tool for care improvement. Int J Qual Health Care. 2010 Oct;22(5):402-7. doi: 10.1093/intqhc/mzq037. Epub 2010 Jul 30.
• Ogrinc GS, & Headrick L. Fundamentals of health care improvement: A guide to improving your patients' care. : Joint Commission Resources, 2008.
• Ogrinc G, Mooney SE, Estrada C, Foster T, Goldmann D, Hall LW, Huizinga MM, Liu SK, Mills P, Neily J, Nelson W, Pronovost PJ, Provost L, Rubenstein LV, Speroff T, Splaine M, Thomson R, Tomolo AM, Watts B. The SQUIRE (Standards for QUality Improvement Reporting Excellence) guidelines for quality improvement reporting: explanation and elaboration. Qual Saf Health Care. 2008 Oct;17 Suppl 1(Suppl_1):i13-32. doi: 10.1136/qshc.2008.029058.
• Buntin MB, Burke MF, Hoaglin MC, Blumenthal D. The benefits of health information technology: a review of the recent literature shows predominantly positive results. Health Aff (Millwood). 2011 Mar;30(3):464-71. doi: 10.1377/hlthaff.2011.0178.
• Raffa SD, Maciejewski ML, Zimmerman LE, Damschroder LJ, Estabrooks PA, Ackermann RT, Tsai AG, Histon T, Goldstein MG. A System-Level Approach to Overweight and Obesity in the Veterans Health Administration. J Gen Intern Med. 2017 Apr;32(Suppl 1):79-82. doi: 10.1007/s11606-016-3948-z.
• Kilbourne AM, Elwy AR, Sales AE, Atkins D. Accelerating Research Impact in a Learning Health Care System: VA's Quality Enhancement Research Initiative in the Choice Act Era. Med Care. 2017 Jul;55 Suppl 7 Suppl 1(7 Suppl 1):S4-S12. doi: 10.1097/MLR.0000000000000683. Erratum In: Med Care. 2019 Nov;57(11):920.
• Mabry PL, Olster DH, Morgan GD, Abrams DB. Interdisciplinarity and systems science to improve population health: a view from the NIH Office of Behavioral and Social Sciences Research. Am J Prev Med. 2008 Aug;35(2 Suppl):S211-24. doi: 10.1016/j.amepre.2008.05.018. Erratum In: Am J Prev Med. 2008 Dec;35(6):611.
• Mabry PL, Milstein B, Abraido-Lanza AF, Livingood WC, Allegrante JP. Opening a window on systems science research in health promotion and public health. Health Educ Behav. 2013 Oct;40(1 Suppl):5S-8S. doi: 10.1177/1090198113503343. No abstract available.
• Luke DA, Stamatakis KA. Systems science methods in public health: dynamics, networks, and agents. Annu Rev Public Health. 2012 Apr;33:357-76. doi: 10.1146/annurev-publhealth-031210-101222. Epub 2012 Jan 3.
• Moore GF, Evans RE. What theory, for whom and in which context? Reflections on the application of theory in the development and evaluation of complex population health interventions. SSM Popul Health. 2017 Dec;3:132-135. doi: 10.1016/j.ssmph.2016.12.005.
• Monks T. Operational research as implementation science: definitions, challenges and research priorities. Implement Sci. 2016 Jun 6;11(1):81. doi: 10.1186/s13012-016-0444-0.
• May CR, Johnson M, Finch T. Implementation, context and complexity. Implement Sci. 2016 Oct 19;11(1):141. doi: 10.1186/s13012-016-0506-3.
• Brainard J, Hunter PR. Do complexity-informed health interventions work? A scoping review. Implement Sci. 2016 Sep 20;11(1):127. doi: 10.1186/s13012-016-0492-5.

Study record dates

Study Major Dates

• Study Start (ACTUAL): February 1, 2019
• Primary Completion (ANTICIPATED): August 31, 2024
• Study Completion (ANTICIPATED): August 31, 2024

Study Registration Dates

• First Submitted: March 8, 2019
• First Submitted That Met QC Criteria: April 17, 2020
• First Posted (ACTUAL): April 22, 2020

Study Record Updates

• Last Update Posted (ACTUAL): August 11, 2021
• Last Update Submitted That Met QC Criteria: August 3, 2021
• Last Verified: August 1, 2021

More Information

Terms related to this study

• Keywords: addiction; implementation science; quality improvement; evidence-based psychotherapy; evidence-based pharmacotherapy; mental health/behavioral health; audit and feedback; participatory system dynamics
• Additional Relevant MeSH Terms: Mental Disorders; Chemically-Induced Disorders; Pathologic Processes; Alcohol-Related Disorders; Substance-Related Disorders; Narcotic-Related Disorders; Compulsive Behavior; Impulsive Behavior; Alcoholism; Disease; Opioid-Related Disorders; Behavior, Addictive
• Other Study ID Numbers: ZIM_0002

Plan for Individual participant data (IPD)

• Plan to Share Individual Participant Data (IPD)?: NO

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No