Testing implementation support for evidence-based programs in community settings: a replication cluster-randomized trial of Getting To Outcomes®

Matthew Chinman, Patricia Ebener, Patrick S Malone, Jill Cannon, Elizabeth J D'Amico, Joie Acosta, Matthew Chinman, Patricia Ebener, Patrick S Malone, Jill Cannon, Elizabeth J D'Amico, Joie Acosta

Abstract

Background: Community organizations can have difficulty implementing evidence-based prevention programs. More research is needed on implementation support interventions designed to help these organizations implement programs with quality.

Methods: Preparing to Run Effective Programs (PREP) is a randomized controlled trial testing Getting To Outcomes (GTO), a 2-year implementation support intervention. It compares 15 Boys and Girls Club sites implementing CHOICE (control group), a five-session evidence-based alcohol and drug prevention program, with 14 similar sites implementing CHOICE supported by GTO (intervention group). PREP replicates a previous GTO study that had the same design, but featured a teen pregnancy prevention program instead. All sites received typical CHOICE training. Fourteen intervention sites received GTO manuals, training, and onsite technical assistance to help practitioners complete implementation best practices specified by GTO (i.e., GTO steps). During the first year, technical assistance providers helped the intervention group adopt, plan, and deliver CHOICE. Then, this group was trained on evaluation and quality improvement steps of GTO using feedback reports summarizing their own data, which yielded revised plans for subsequent implementation of CHOICE. This paper presents results regarding GTO's impact on CHOICE fidelity (adherence, quality of delivery, dosage) and the proximal outcomes of the youth participants (aged 10-14)-attitudes and intentions regarding cigarettes, alcohol, and marijuana use. Fidelity was assessed at all sites by observer ratings and attendance logs. Proximal outcomes were assessed via survey at baseline, 3, and 6 months.

Results: After 1 year, fidelity and proximal outcomes were similar between Intervention and control groups. After 2 years (which included GTO quality improvement activities that took place between years 1 and 2), intervention sites had higher ratings of CHOICE adherence and quality of delivery (dosage remained similar). Proximal outcomes did not differ between groups in either year, although there was universally high endorsement of prosocial responses to those outcomes from the start.

Conclusions: Findings suggest that systematic implementation support provided by GTO can help community organizations achieve better fidelity. Findings replicate the implementation results from a previous GTO study using the same design, but with a different evidence-based program and different fidelity measures. Although proximal outcomes did not change, in large part due to ceiling effects, the implementation findings suggest GTO can support a variety of programs.

Trial registration: This project is registered at ClinicalTrials.gov with number NCT02135991 . The trial was first registered on May 12, 2014.

Keywords: Community-based; Evidence-based prevention; Fidelity; Implementation support.

Conflict of interest statement

Ethics approval and consent to participate

This study was approved by RAND’s Human Subjects Protection Committee (Federal Assurance Number: FWA00003425), which reviews and monitors all non-exempt research to ensure protection of subjects from research risks, pursuant to federal regulations. All subjects in this research study provided consent to participate.

Consent for publication

This paper does not contain any individually identifiable images or any other data. Thus, consent for publication was not obtained.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Figures

Fig. 1
Fig. 1
Getting To Outcomes logic model

References

    1. Johnston LD, O’Malley PM, Miech RA, Bachman JG, Schulenberg JE. Monitoring the future national survey results on drug use, 1975–2015: overview, key findings on adolescent drug use. Ann Arbor, MI: Institute for Social Research, The University of Michigan; 2016.
    1. U.S . Department of Health and Human Services. Facing Addiction in America. The Surgeon General’s Report on Alcohol, Drugs, and Health. Washington, DC: U.S. Department of Health and Human Services; 2016.
    1. Green L. From research to ‘best practices’ in other settings and populations. Am J Health Behav. 2001;25(3):165–178. doi: 10.5993/AJHB.25.3.2.
    1. Wandersman A, Florin P. Community interventions and effective prevention: bringing researchers/evaluators, funders and practitioners together for accountability. Am Psychol. 2003;58(6/7):441–448. doi: 10.1037/0003-066X.58.6-7.441.
    1. Ennett ST, Ringwalt CL, Thorne J, Rohrbach LA, Vincus A, Simons-Rudolph A, et al. A comparison of current practice in school-based substance abuse prevention programs with meta-analysis findings. Prev Sci. 2003;4:1–14. doi: 10.1023/A:1021777109369.
    1. Hallfors D, Godette D. Will the 'principles of effectiveness' improve prevention practice? Early findings from a diffusion study. Health Educ Res. 2002;17(4):461–470. doi: 10.1093/her/17.4.461.
    1. Kumar R, O'Malley PM, Johnston LD, Laetz VB. Alcohol, tobacco, and other drug use prevention programs in U.S. schools: a descriptive summary. Prev Sci. 2013;14(6):581–592. doi: 10.1007/s11121-012-0340-z.
    1. Ringwalt C, Hanley S, Vincus AA, Ennett ST, Rohrbach LA, Bowling JM. The prevalence of effective substance use prevention curricula in the nation's high schools. J Prim Prev. 2008;29(6):479–488. doi: 10.1007/s10935-008-0158-4.
    1. Ringwalt C, Vincus AA, Hanley S, Ennett ST, Bowling JM, Rohrbach LA. The prevalence of evidence-based drug use prevention curricula in U.S. middle schools in 2005. Prev Sci. 2009;10(1):33–40. doi: 10.1007/s11121-008-0112-y.
    1. Chinman M, Acosta J, Hunter S, Ebener P. Getting To Outcomes®: evidence of empowerment evaluation and evaluation capacity building at work. In: Fetterman DM, Kaftarian S, Wandersman A, editors. Empowerment evaluation: knowledge and tools for self-assessment, evaluation capacity building and accountability. 2. Thousand Oaks, CA: Sage; 2015. pp. 316–336.
    1. D'Amico EJ, Tucker JS, Miles JNV, Zhou AJ, Shih RA, Green HDJ. Preventing alcohol use with a voluntary after school program for middle school students: results from a cluster randomized controlled trial of CHOICE. Prev Sci. 2012;13:415–425. doi: 10.1007/s11121-011-0269-7.
    1. Chinman M, Acosta J, Ebener P, Malone PS, Slaughter ME. Can implementation support help community-based settings better deliver evidence-based sexual health promotion programs? A randomized trial of Getting To Outcomes(R) Implement Sci. 2016;11(1):78. doi: 10.1186/s13012-016-0446-y.
    1. Jemmott JB, Jemmott LS, Fong GT. Abstinence and safer sex HIV risk-reduction interventions for African American adolescents: a randomized controlled trial. J Am Med Assoc. 1998;279:1529–1536. doi: 10.1001/jama.279.19.1529.
    1. Livet M, Wandersman A. Organizational functioning: facilitating effective interventions and increasing the odds of programming success. In: Fetterman DM, Wandersman A, editors. Empowerment evaluation in practice. New York: Guilford; 2005. pp. 123–154.
    1. Ajzen I, Fishbein M. Attitude-behavior relations: a theoretical analysis and review of empirical research. Psychol Bull. 1977;84:888–918. doi: 10.1037/0033-2909.84.5.888.
    1. Bandura A. Health promotion by social cognitive means. Health Educ Behav. 2004;31(2):143–164. doi: 10.1177/1090198104263660.
    1. Fishbein M, Ajzen I. Attitudes toward objects as predictive of single and multiple behavioral criteria. Psychol Rev. 1974;81:59–74. doi: 10.1037/h0035872.
    1. Fishbein M, Ajzen I. Belief, attitude, intention, and behavior: an introduction to theory and research. Reading, MA: Addison-Wesley; 1975.
    1. Acosta J, Chinman M, Ebener P, Malone PS, Paddock S, Phillips A, et al. An intervention to improve program implementation: findings from a two-year cluster randomized trial of assets-Getting To Outcomes. Implement Sci. 2013;8:87. doi: 10.1186/1748-5908-8-87.
    1. Smelson DA, Chinman M, McCarthy S, Hannah G, Sawh L, Glickman M. A cluster randomized hybrid type III trial testing an implementation support strategy to facilitate the use of an evidence-based practice in VA homeless programs. Implement Sci. 2015;10:79. doi: 10.1186/s13012-015-0267-4.
    1. Durlak JA, DuPre EP. Implementation matters: a review of research on the influence of implementation on program outcomes and the factors affecting implementation. Am J Community Psychol. 2008;41:327–350. doi: 10.1007/s10464-008-9165-0.
    1. Chinman M, Imm P, Wandersman A. Getting to Outcomes 2004: Promoting Accountability Through Methods and Tools for Planning, Implementation, and Evaluation. Santa Monica, CA: RAND Corporation; 2004.
    1. Wandersman A, Imm P, Chinman M, Kaftarian S. Getting to outcomes: a results-based approach to accountability. Eval Program Plann. 2000;23:389–395. doi: 10.1016/S0149-7189(00)00028-8.
    1. Chinman M, Hunter SB, Ebener P, Paddock S, Stillman L, Imm P, et al. The Getting To Outcomes demonstration and evaluation: an illustration of the prevention support system. Am J Community Psychol. 2008;41:206–224. doi: 10.1007/s10464-008-9163-2.
    1. Chinman M, Acosta JD, Ebener P, Sigel C, Keith J. Getting To Outcomes: a guide for teen pregnancy prevention. Santa Monica, CA: RAND Corporation; 2016.
    1. Imm P, Chinman M, Wandersman A, Rosenbloom D, Guckenburg S, Leis R. Using the “Getting to Outcomes” Approach to Help Communities Prevent Underage Drinking. Santa Monica, CA: RAND Corporation; 2007.
    1. Fisher D, Imm P, Chinman M, Wandersman A. Getting To Outcomes with developmental assets: ten steps to measuring success in youth programs and communities Minneapolis: MN: Search Institute; 2006.
    1. Chinman M, Tremain B, Imm P, Wandersman A. Strengthening prevention performance using technology: a formative evaluation of interactive Getting To Outcomes. Am J Orthopsychiatry. 2009;79:469–481. doi: 10.1037/a0016705.
    1. Chinman M, Acosta J, Ebener P, Malone PS, Slaughter M. A cluster-randomized trial of Getting To Outcomes’ impact on sexual health outcomes in community-based settings. Prev Sci. In Press.
    1. Hawkins JD, Oesterle S, Brown EC, Arthur MW, Abbott RD, Fagan AA, et al. Results of a type 2 translational research trial to prevent adolescent drug use and delinquency: a test of communities that care. Arch Pediatr Adolesc Med. 2009;163(9):789–798. doi: 10.1001/archpediatrics.2009.141.
    1. Spoth R, Redmond C, Shin C, Greenberg M, Clair S, Feinberg M. Substance-use outcomes at 18 months past baseline: the PROSPER Community-University Partnership Trial. Am J Prev Med. 2007;32(5):395–402. doi: 10.1016/j.amepre.2007.01.014.
    1. Rohrbach LA, Gunning M, Sun P, Sussman S. The project towards no drug abuse (TND) dissemination trial: implementation fidelity and immediate outcomes. Prev Sci. 2010;11(1):77–88. doi: 10.1007/s11121-009-0151-z.
    1. Miller WR, Rollnick S. Motivational interviewing: preparing people for change. 2. New York: Guilford Press; 2002.
    1. Rollnick S, Miller WR, Butler CC. Motivational interviewing in health care: helping patients change behavior. New York: Guilford Press; 2008.
    1. D'Amico EJ, Ellickson PL, Wagner EF, Turrisi R, Fromme K, Ghosh-Dastidar B, et al. Developmental considerations for substance use interventions from middle school through college. Alcohol Clin Exp Res. 2005;29:474–483. doi: 10.1097/01.ALC.0000156081.04560.78.
    1. Chinman M, Acosta J, Ebener P, Burkhart Q, Clifford M, Corsello M, et al. Intervening with practitioners to improve the quality of prevention: one year findings from a randomized trial of assets-Getting To Outcomes. J Prim Prev. 2013;34:173–191. doi: 10.1007/s10935-013-0302-7.
    1. Chinman M, Ebener P, Burkhart Q, Osilla KC, Imm P, Paddock SM, et al. Evaluating the impact of getting to outcomes-underage drinking on prevention capacity and alcohol merchant attitudes and selling behaviors. Prev Sci. 2014;15(4):485–496. doi: 10.1007/s11121-013-0389-3.
    1. Hawkins JD, Oesterle S, Brown EC, Arthur MW, Abbott RD, Fagan AA, et al. Results of a type 2 translational research trial to prevent adolescent drug use and delinquency: a test of communities that care. Arch Pediatr Adolesc Med. 2009;163(9):789–798. doi: 10.1001/archpediatrics.2009.141.
    1. Spoth R, Redmond C, Shin C, Greenberg M, Clair S, Feinberg M. Substance-use outcomes at 18 months past baseline: the PROSPER Community-University Partnership Trial. Am J Prev Med. 2007;32(5):395–402. doi: 10.1016/j.amepre.2007.01.014.
    1. Chinman M, Acosta J, Ebener P, Malone PS, Slaughter ME. Can implementation support help community-based settings better deliver evidence-based sexual health promotion programs? A randomized trial of Getting To Outcomes(R) Implement Sci. 2016;11(1):78. doi: 10.1186/s13012-016-0446-y.
    1. Chinman M, Acosta J, Ebener P, Malone PS, Slaughter ME. A cluster-randomized trial of Getting To Outcomes’ impact on sexual health outcomes in community-based settings. Prev Sci. 2018;19(4):437–448. doi: 10.1007/s11121-017-0845-6.
    1. D'Amico EJ, Green HDJ, Miles JNV, Zhou AJ, Tucker JA, Shih RA. Voluntary after school alcohol and drug programs: if you build it right, they will come. J Res Adolesc. 2012;22(3):571–582. doi: 10.1111/j.1532-7795.2012.00782.x.
    1. Aarons GA, Glisson C, Hoagwood K, Kelleher K, Landsverk J, Cafri G. Psychometric properties and U.S. national norms of the evidence-based practice attitude scale (EBPAS) Psychol Assess. 2010;22(2):356–365. doi: 10.1037/a0019188.
    1. McDonald RP. Test theory: a unified treatment. Mahwah: Lawrence Erlbaum; 1999.
    1. Dunn TJ, Baguley T, Brunsden V. From alpha to omega: a practical solution to the pervasive problem of internal consistency estimation. Br J Psychol. 2014;105(3):399–412. doi: 10.1111/bjop.12046.
    1. Krippendorff K. Content analysis: an introduction to its methodology. 2. Thousand Oaks, CA: Sage; 2004.
    1. Bandura A. Self-efficacy: toward a unifying theory of behavioral change. Psychol Rev. 1977;84:191–215. doi: 10.1037/0033-295X.84.2.191.
    1. Kahneman D, Tversky A. Choices, values, and frames. New York: Cambridge University Press; 2000.
    1. Bandura A. Self-efficacy: the exercise of control. New York: Freeman; 1997.
    1. Miller WR, Rollnick S. Motivational interviewing: helping people change. 3. New York: Guilford Press; 2012.
    1. D'Amico EJ, Edelen M. Pilot test of Project CHOICE: a voluntary after school intervention for middle school youth. Psychol Addict Behav. 2007;21(4):592–598. doi: 10.1037/0893-164X.21.4.592.
    1. Dusenbury L, Brannigan R, Hansen W, Walsh J, Falco M. Quality of implementation: developing measures crucial to understanding the diffusion of preventive interventions. Health Educ Behav. 2005;20:308–313.
    1. Moyers TB, Martin T, Manuel JK, Miller WR, Ernst D. Revised Global Scales: Motivational Interviewing Treatment Integrity 3.1.1 (MITI 3.1.1) Albuquerque: NM University of New Mexico; 2010.
    1. Madson MB, Campbell TC. Measures of fidelity in motivational enhancement: a systematic review. J Subst Abus Treat. 2006;31(1):67–73. doi: 10.1016/j.jsat.2006.03.010.
    1. Moyers TB, Martin T, Manuel JK, Hendrickson SM, Miller WR. Assessing competence in the use of motivational interviewing. J Subst Abus Treat. 2005;28(1):19–26. doi: 10.1016/j.jsat.2004.11.001.
    1. Cambell M, Carr C, DeVellis B, Switzer B, Biddle A, Amamoo MA, et al. A randomized trial of tailoring and motivational interviewing to promote fruit and vegetable consumption for cancer prevention and control. Ann Behav Med. 2009;38(2):71–85. doi: 10.1007/s12160-009-9140-5.
    1. Martino S, Ball SA, Nich C, Frankforter TL, Carroll KM. Community program therapist adherence and competence in motivational enhancement therapy. Drug Alcohol Depend. 2008;96(1):37–48. doi: 10.1016/j.drugalcdep.2008.01.020.
    1. Parsons JT, Golub SA, Rosof E, Holder C. Motivational interviewing and cognitive-behavioral intervention to improve HIV medication adherence among hazardous drinkers: a randomized controlled trial. J Accuir Immune Defic. 2007;46(4):443. doi: 10.1097/QAI.0b013e318158a461.
    1. Turrisi R, Larimer ME, Mallett KA, Kilmer JR, Ray AE, Mastroleo NR, et al. A randomized clinical trial evaluating a combined alcohol intervention for high-risk college students. J Stud Alcohol Drugs. 2009;70(4):555. doi: 10.15288/jsad.2009.70.555.
    1. McCambridge J, Day M, Thomas BA, Strang J. Fidelity to motivational interviewing and subsequent cannabis cessation among adolescents. Addict Behav. 2011;36(7):749–754. doi: 10.1016/j.addbeh.2011.03.002.
    1. Pollak KI, Coffman CJ, Alexander SC, Ostbye T, Lyna P, Tulsky JA, et al. Predictors of weight-related communication during primary care visits with overweight adolescents. Patient Educ Couns. 2014;96(3):327–332. doi: 10.1016/j.pec.2014.07.025.
    1. Woodin EM, Sotskova A, O'Leary KD. Do motivational interviewing behaviors predict reductions in partner aggression for men and women? Behav Res Ther. 2012;50:79–84. doi: 10.1016/j.brat.2011.11.001.
    1. Cicchetti DV. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol Assess. 1994;94:284–290. doi: 10.1037/1040-3590.6.4.284.
    1. Johnston LD, O'Malley PM, Miech RA, Bachman JG, Schulenberg JE. Monitoring the future national survey results on drug use, 1975–2016: overview, key findings on adolescent drug use. Ann Arbor: Institute for Social Research, The University of Michigan; 2017.
    1. D'Amico EJ, Hunter SB, Miles JN, Ewing BA, Osilla KC. A randomized controlled trial of a group motivational interviewing intervention for adolescents with a first time alcohol or drug offense. J Subst Abus Treat. 2013;45(5):400–408. doi: 10.1016/j.jsat.2013.06.005.
    1. D'Amico EJ, Miles JN, Stern SA, Meredith LS. Brief motivational interviewing for teens at risk of substance use consequences: a randomized pilot study in a primary care clinic. J Subst Abus Treat. 2008;35(1):53–61. doi: 10.1016/j.jsat.2007.08.008.
    1. DAmico E. J., Parast L., Meredith L. S., Ewing B. A., Shadel W. G., Stein B. D. Screening in Primary Care: What Is the Best Way to Identify At-Risk Youth for Substance Use? PEDIATRICS. 2016;138(6):e20161717–e20161717. doi: 10.1542/peds.2016-1717.
    1. D'Amico EJ, Miles JN, Tucker JS. Gateway to curiosity: medical marijuana ads and intention and use during middle school. Psychol Addict Behav. 2015;29(3):613–619. doi: 10.1037/adb0000094.
    1. D'Amico EJ, Parast L, Shadel WG, Meredith LS, Seelam R, Stein BD. Brief motivational interviewing intervention to reduce alcohol and marijuana use for at-risk adolescents in primary care: a randomized clinical trial. J Consult Clin Psychol. In Press.
    1. D’Amico EJ, McCarthy DM. Escalation and initiation of younger adolescents’ substance use: the impact of perceived peer use. J Adolesc Health. 2006;39:481–487. doi: 10.1016/j.jadohealth.2006.02.010.
    1. D'Amico EJ, Miles JNV, Stern SA, Meredith LS. Brief motivational interviewing for teens at risk of substance use consequences: a randomized pilot study in a primary care clinic. J Subst Abus Treat. 2008;35:53–61. doi: 10.1016/j.jsat.2007.08.008.
    1. Ellickson PL, Bell RM. Drug prevention in junior high: a multi-site longitudinal test. Science. 1990;247:1265–1372. doi: 10.1126/science.2180065.
    1. Lakens D. Calculate Partial Eta Omega Epsilon Squared.xlsx. 2013. Accessed 15 Apr 2016.
    1. Lakens D. Calculating_Effect_Sizes.xlsx (v.4). 2013. . Accessed 22 Nov 2017.
    1. Hess MR, Kromrey JD. Effect_CI: a SAS® macro for constructing confidence intervals around standardized mean differences. 2003. . Accessed 16 Apr 2016.
    1. Benjamini Y, Hochberg B. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J R Stat Soc. 1995;57(1):289–300.
    1. Fagan AA, Hanson K, Hawkins JD, Arthur MW. Bridging science to practice: achieving prevention program implementation fidelity in the community youth development study. Am J Community Psychol. 2008;41:235–249. doi: 10.1007/s10464-008-9176-x.
    1. Rohrbach LA, Gunning M, Sun P, Sussman S. The project towards no drug abuse (TND) dissemination trial: implementation fidelity and immediate outcomes. Prev Sci. 2010;11:77–88. doi: 10.1007/s11121-009-0151-z.
    1. Spoth R, Guyll M, Lillehoj CJ, Redmond C. PROSPER study of evidence-based intervention implementation quality by community–university partnerships. J Community Psychol. 2007;35(8):981–999. doi: 10.1002/jcop.20207.
    1. D'Amico EJ, Houck JM, Hunter SB, Miles JN, Osilla KC, Ewing BA. Group motivational interviewing for adolescents: change talk and alcohol and marijuana outcomes. J Consult Clin Psychol. 2015;83(1):68–80. doi: 10.1037/a0038155.
    1. Mueller T, Tevendale HD, Fuller TR, House LD, Romero LM, Brittain A, et al. Teen pregnancy prevention: implementation of a multicomponent Community-Wide Approach. J Adolesc Health. 2017;60(3S):S9–S17. doi: 10.1016/j.jadohealth.2016.11.002.
    1. Ellickson PL, Martino SC, Collins RL. Marijuana use from adolescence to young adulthood: multiple developmental trajectories and their associated outcomes. Health Psychol. 2004;23(3):299–307. doi: 10.1037/0278-6133.23.3.299.
    1. Jackson KM, Sher KJ, Schulenberg JE. Conjoint developmental trajectories of young adult substance use. Alcohol Clin Exp Res. 2008;32(5):723–737. doi: 10.1111/j.1530-0277.2008.00643.x.
    1. Schulenberg JE, Merline AC, Johnston LD, O'Malley PM, Bachman JG, Laetz VB. Trajectories of marijuana use during the transition to adulthood: the big picture based on National Panel Data. J Drug Issues. 2005;35(2):255–279. doi: 10.1177/002204260503500203.
    1. Setoh P, Qin L, Zhang X, Pomerantz EM. The social self in early adolescence: two longitudinal investigations in the United States and China. Dev Psychol. 2015;51(7):949–961. doi: 10.1037/a0039354.
    1. Tucker JS, Ellickson PL, Orlando M, Klein DJ. Cigarette smoking from adolescence to young adulthood: women's developmental trajectories and associates outcomes. Womens Health Issues. 2006;16(1):30–37. doi: 10.1016/j.whi.2005.08.004.
    1. Tucker JS, Orlando M, Ellickson PL. Patterns and correlates of binge drinking trajectories from early adolescence to young adulthood. Health Psychol. 2003;22(1):79–87. doi: 10.1037/0278-6133.22.1.79.
    1. Vega WA, Chen KW, Williams J. Smoking, drugs, and other behavioral health problems among multiethnic adolescents in the NHSDA. Addict Behav. 2007;32(9):1949–1956. doi: 10.1016/j.addbeh.2006.12.022.
    1. Chervin DD, Philliber S, Brindis CD, Chadwick AE, Revels ML, Kamin SL, et al. Community capacity building in CDC’s community coalition partnership programs for the prevention of teen pregnancy. J Adolesc Health. 2005;37:S11–SS9. doi: 10.1016/j.jadohealth.2005.06.001.
    1. Lesesne CA, Lewis KM, White CP, White DC, Green DC, Duffy JL, et al. Promoting science-based approaches to teen pregnancy prevention: proactively engaging the three systems of the Interactive Systems Framework. Am J Community Psychol. 2008;41:379–392. doi: 10.1007/s10464-008-9175-y.
    1. Philliber S, Nolte K. Implementation science: promoting science-based approaches to prevent teen pregnancy. Prev Sci. 2008;9(3):166–177. doi: 10.1007/s11121-008-0094-9.
    1. Zief S, Shapiro R, Strong D. How states are implementing evidence-based teen pregnancy prevention programs through the personal responsibility education program. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families; 2014.

Source: PubMed

Szponzorok és közreműködők

Egészségi állapot

Kábítószer-beavatkozások

3
Iratkozz fel