Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda

Enola Proctor, Hiie Silmere, Ramesh Raghavan, Peter Hovmand, Greg Aarons, Alicia Bunger, Richard Griffey, Melissa Hensley, Enola Proctor, Hiie Silmere, Ramesh Raghavan, Peter Hovmand, Greg Aarons, Alicia Bunger, Richard Griffey, Melissa Hensley

Abstract

An unresolved issue in the field of implementation research is how to conceptualize and evaluate successful implementation. This paper advances the concept of "implementation outcomes" distinct from service system and clinical treatment outcomes. This paper proposes a heuristic, working "taxonomy" of eight conceptually distinct implementation outcomes-acceptability, adoption, appropriateness, feasibility, fidelity, implementation cost, penetration, and sustainability-along with their nominal definitions. We propose a two-pronged agenda for research on implementation outcomes. Conceptualizing and measuring implementation outcomes will advance understanding of implementation processes, enhance efficiency in implementation research, and pave the way for studies of the comparative effectiveness of implementation strategies.

Figures

Fig. 1
Fig. 1
Types of outcomes in implementation research

References

    1. Aarons GA. Mental health provider attitudes toward adoption of evidence-based practice: The evidence-based practice attitude scale (EBPAS) Mental Health Service Research. 2004;6(2):61–74. doi: 10.1023/B:MHSR.0000024351.12294.65.
    1. Aarons GA, Palinkas LA. Implementation of evidence-based practice in child welfare: Service provider perspectives. Mental Health Services Research. 2007;34:411–419. doi: 10.1007/s10488-007-0121-3.
    1. Adily A, Westbrook J, Coiera E, Ward J. Use of on-line evidence databases by Australian public health practitioners. Medical Informatics & the Internet in Medicine. 2004;29(2):127–136. doi: 10.1080/14639230410001723437.
    1. Bartholomew NG, Joe GW, Rowan-Szal GA, Simpson DD. Counselor assessments of training and adoption barriers. Journal of Substance Abuse Treatment. 2007;33:193–199. doi: 10.1016/j.jsat.2007.01.005.
    1. Bates DM, Sarkar D. lme4: Linear mixed-effects models using S4 classes. R Package Version. 2007;0:99875–99876.
    1. Berwick DM. Disseminating innovations in health care. Journal of the American Medical Association. 2003;289(15):1969–1975. doi: 10.1001/jama.289.15.1969.
    1. Bond GR, McHugo GJ, Becker DR, Rapp CA, Whitley R. Fidelity of supported employment: Lessons learned from the national evidence-based practice project. Psychiatric Rehabilitation Journal. 2008;31(4):300–305. doi: 10.2975/31.4.2008.300.305.
    1. Callum P, Iger J, Ray M, Sims CA, Falk RE. Outcome and experience of implementing spinal muscular atrophy carrier screening on sperm donors. Fertility and Sterility. 2010;94(5):1912–1914. doi: 10.1016/j.fertnstert.2009.12.071.
    1. Chamberlain P, Brown CH, Saldana L, Reid J, Wang W, Marsenich L, et al. Engaging and recruiting counties in an experiment on implementing evidence-based practice in California. Administration and Policy In Mental Health. 2008;35:250–260. doi: 10.1007/s10488-008-0167-x.
    1. Cooke M, Mattick RP, Walsh RA. Implementation of the ‘fresh start’ smoking cessation programme to 23 antenatal clinics: A randomized controlled trial investigating two methods of dissemination. Drug and Alcohol Review. 2001;20:19–28. doi: 10.1080/09595230020029536.
    1. Dane AV, Schneider BH. Program integrity in primary and early secondary prevention: Are implementation effects out of control? Clinical Psychology Review. 1998;18:23–45. doi: 10.1016/S0272-7358(97)00043-3.
    1. Davis F. User acceptance of information technology: System characteristics, user perceptions and behavioral impacts. International Journal of Man-Machine Studies. 1993;38:475–487. doi: 10.1006/imms.1993.1022.
    1. Dusenbury L, Brannigan R, Falco M, Hansen WB. A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Education Research. 2003;18(2):237–256. doi: 10.1093/her/18.2.237.
    1. Fischer MA, Vogeli C, Stedman MR, Ferris TG, Weissman JS. Uptake of electronic prescribing in community-based practices. Journal of General Internal Medicine. 2008;23(4):358–363. doi: 10.1007/s11606-007-0383-1.
    1. Fixsen, D. L., Naoom, S. F., Blasé, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature (No. FMHI Publication #231). Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, National Implementation Research Network.
    1. Frambach RT, Schillewaert N. Organizational innovation adoption: A multi-level framework of determinants and opportunities for future research. Journal of Business Research Special Issue: Marketing Theory in the Next Millennium. 2002;55:163–176.
    1. Glasgow RE. eHealth evaluation and dissemination research. American Journal of Preventive Medicine. 2007;32(5S):S119–S126. doi: 10.1016/j.amepre.2007.01.023.
    1. Glasgow, R. E. (2007b). The RE-AIM model for planning, evaluation and reporting on implementation and dissemination research. Paper presented at the NIH Conference on Building the Science of Dissemination and Implementation in the Service of Public Health: Bethesda, MD.
    1. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: The RE-AIM framework. American Journal of Public Health. 1999;89:1322–1327. doi: 10.2105/AJPH.89.9.1322.
    1. Goodman RM, McLeroy KR, Steckler AB, Hoyle RH. Development of level of institutionalization scales for health promotion programs. Health Education Quarterly. 1993;20(2):161–179.
    1. Graham ID, Brouwers M, Davies C, Tetroe J. Ontario doctors’ attitudes toward and use of clinical practice guidelines in oncology. Journal of Evaluation in Clinical Practice. 2007;13(4):607–615. doi: 10.1111/j.1365-2753.2006.00670.x.
    1. Grimshaw J, Eccles M, Thomas R, MacLennan G, Ramsay C, Fraser C, et al. Toward evidence-based quality improvement: Evidence (and its limitations) of the effectiveness of guideline dissemination and implementation strategies 1966–1998. Journal of General Internal Medicine. 2006;21:S14–S20.
    1. Haug NA, Shopshire M, Tajima B, Gruber V, Guydish J. Adoption of evidence-based practices among substance abuse treatment providers. Journal of Drug Education. 2008;38(2):181–192. doi: 10.2190/DE.38.2.f.
    1. Hayes SC. President’s message: Dissemination research now. The Behavior Therapist. 1998;21(9):166–169.
    1. Henggeler SW, Lee T, Burns JA. What happens after the innovation is identified? Clinical Psychology: Science & Practice. 2002;9(2):191–194. doi: 10.1093/clipsy/9.2.191.
    1. Henggeler SW, Chapman JE, Rowland MD, Halliday-Boykinds CA, Randall J, Shackelford J, et al. Statewide adoption and initial implementation of contingency management for substance-abusing adolescents. Journal of Consulting and Clinical Psychology. 2008;76(4):556–567. doi: 10.1037/0022-006X.76.4.556.
    1. Hides L, Lubman DI, Elkins K, Catania LS, Rogers N. Feasibility and acceptability of a mental health screening tool and training programme in the youth alcohol and other drug (AOD) sector. Drug and Alcohol Review. 2007;26:509–515. doi: 10.1080/09595230701499126.
    1. Hogue A, Liddle HA, Rowe C. Treatment adherence process research in family therapy: A rationale and some practical guidelines. Psychotherapy. 1996;33(2):332–345.
    1. Hogue A, Dauber S, Chincilla P, Fried A, Henderson C, Inclan J, et al. Assessing fidelity in individual and family therapy for adolescent substance abuse. Journal of Substance Abuse Treatment. 2008;35:137–147. doi: 10.1016/j.jsat.2007.09.002.
    1. Hovmand PS, Gillespie DF. Implementation of evidence based practice and organizational performance. Journal of Behavioral Health Services & Research. 2010;37(1):79–94. doi: 10.1007/s11414-008-9154-y.
    1. Institute of Medicine Committee on Crossing the Quality Chasm . Adaption to mental health and addictive disorder: Improving the quality of health care for mental and substance-use conditions. Washington, DC: Institute of Medicine, National Academies Press; 2006.
    1. Institute of Medicine Committee on Quality of Health Care in America . Crossing the quality chasm: A new health system for the 21st century. Washington, DC: Institute of Medicine, National Academy Press; 2001.
    1. Johnson K, Hays C, Center H, Daley C. Building capacity and sustainable prevention innovations: A sustainability planning model. Evaluation Program Planning. 2004;27:135–149. doi: 10.1016/j.evalprogplan.2004.01.002.
    1. Jones K. Mission drift in qualitative research, or moving toward a systematic review of qualitative studies. The Qualitative Report. 1997;9(1):95–112.
    1. Karlsson A, Bendtsen P. Acceptability of a computerized alcohol screening and advice routine in an emergency department setting―a patient perspective. Addictive Behaviors. 2005;30:767–776. doi: 10.1016/j.addbeh.2004.08.022.
    1. Karsh BT. Beyond usability: Designing effective technology implementation systems to promote patient safety. Quality and Safety in Health Care. 2004;13:388–394. doi: 10.1136/qshc.2004.010322.
    1. Kirkevoid M. Integrative nursing research—an important strategy to further the development of nursing science and practice. Journal of Advanced Nursing. 1997;25:977–984. doi: 10.1046/j.1365-2648.1997.1997025977.x.
    1. Klein KJ, Knight AP. Innovation implementation: Overcoming the challenge. Current Directions in Psychological Science. 2005;14(5):243–246. doi: 10.1111/j.0963-7214.2005.00373.x.
    1. Klein KJ, Sorra JS. The challenge of innovation implementation. (Special topic forum on the management of innovation) Academy of Management Review. 1996;21(4):1055–1081. doi: 10.2307/259164.
    1. Luke DA. Multilevel modeling. Sage University Papers Series on Quantitative Applications in the Social Sciences. Thousand Oaks, CA: Sage; 2004.
    1. McHugh RK, Otto MW, Barlow DH, Gorman JM, Shear MK, Woods SW. Cost-efficacy of individual and combined treatments for panic disorder. The Journal of Clinical Psychiatry. 2007;68(7):1038–1044. doi: 10.4088/JCP.v68n0710.
    1. McPheeters ML, Briss P, Teutsch SJ, Truman B. Systematic reviews in public health. In: Brownson RC, Petitti DB, editors. Applied epidemiology: Theory to practice. 2. New York: Oxford University Press; 2006. pp. 99–124.
    1. Michie S, Johnson M, Abraham C, Lawton R, Parker D, Walker A. Making psychological theory useful for implementing evidence based practice: A consensus approach. Quality and Safety in Health Care. 2005;14(1):26–33. doi: 10.1136/qshc.2004.011155.
    1. Michie S, Pilling S, Garety P, Whitty P, Eccles MP, Johnston M, et al. Difficulties implementing a mental health guideline: An exploratory investigation using psychosocial theory. Implementation Science. 2007;2(8):1–8.
    1. Michie S, Fixsen D, Grimshaw J, Eccles M. Specifying and reporting complex behavior change interventions: The need for a scientific method. Implementation Science. 2009;4(1):40. doi: 10.1186/1748-5908-4-40.
    1. Mihalic S. The importance of implementation fidelity. Emotional and Behavioral Disorders in Youth. 2004;4:83–86.
    1. Proctor EK, Landsverk J, Aarons G, Chambers D, Glisson C, Mittman B. Implementation research in mental health services: An emerging science with conceptual, methodological, and training challenges. Administration and Policy in Mental Health and Mental Health Services Research. 2009;36(1):24–34. doi: 10.1007/s10488-008-0197-4.
    1. Rabin BA, Brownson RC, Haire-Joshu D, Kreuter MW, Weaver NL. A glossary for dissemination and implementation research in health. Journal of Public Health Management and Practice. 2008;14(2):117–123.
    1. Raghavan R, Bright C, Shadoin A. Toward a policy ecology of implementation of evidence-based practices in public mental health settings. Implementation Science. 2008;3:26. doi: 10.1186/1748-5908-3-26.
    1. Repenning NP. A simulation-based approach to understanding the dynamics of innovation implementation. Organization Science. 2002;13(2):109–127. doi: 10.1287/orsc.13.2.109.535.
    1. Rogers EM. Diffusion of innovations. 4. New York: The Free Press; 1995.
    1. Ronckers E, Groot W, Steenbakkers M, Ruland E, Ament A. Costs of the ‘hartslag limburg’ community heart health intervention. BMC Public Health. 2006;6(1):51. doi: 10.1186/1471-2458-6-51.
    1. Rosen A, Proctor EK. Distinctions between treatment outcomes and their implications for treatment evaluation. Journal of Consulting and Clinical Psychology. 1981;49(3):418–425. doi: 10.1037/0022-006X.49.3.418.
    1. Rye CB, Kimberly JR. The adoption of innovations by provider organizations in health care. Medical Care Research and Review. 2007;64(3):235–278. doi: 10.1177/1077558707299865.
    1. Schoenwald SK, Letourneau EJ, Halliday-Boykins CA. Predicting therapist adherence to a transported family-based treatment for youth. Journal of Clinical Child & Adolescent Psychology. 2005;34:658–670. doi: 10.1207/s15374424jccp3404_8.
    1. Shumway M, Saunders T, Shern D, Pines E, Downs A, Burbine T, et al. Preferences for schizophrenia treatment outcomes among public policy makers, consumers, families, and provides. Psychiatric Services. 2003;54:1124–1128. doi: 10.1176/appi.ps.54.8.1124.
    1. Steckler A, Goodman R, McElory K. Measuring the diffusion of innovative health promotion programs. American Journal of Health Promotion. 1992;6:214–224.
    1. Stiles PG, Boothroyd RA, Snyder K, Zong X. Service penetration by persons with severe mental illness: How should it be measured? Journal of Behavioral Health Services & Research. 2002;29(2):198. doi: 10.1007/BF02287706.
    1. Turner KMT, Sanders MR. Dissemination of evidence-based parenting and family support strategies: Learning from the triple p-positive parenting program system approach. Aggression and Violent Behavior. 2006;11(2):176–193. doi: 10.1016/j.avb.2005.07.005.
    1. Waldorff F, Steenstrup A, Nielsen B, Rubak J, Bro F. Diffusion of an e-learning programme among Danish general practitioners: A nation-wide prospective survey. BMC Family Practice. 2008;9(1):24. doi: 10.1186/1471-2296-9-24.
    1. Wilkie DJ, Judge MKM, Berry DL, Dell J, Zong S, Gilespie R, et al. Usability of a computerized PAINReportIt in the general public with pain and people with cancer pain. Journal of Pain and Symptom Management. 2003;25(3):213–224. doi: 10.1016/S0885-3924(02)00638-3.
    1. Woolf SH. The meaning of translational research and why it matters. Journal of the American Medical Association. 2008;299(2):211–213. doi: 10.1001/jama.2007.26.

Source: PubMed

Szponzorok és közreműködők

Egészségi állapot

Kábítószer-beavatkozások

3
Iratkozz fel