Assessing mental health clinicians' intentions to adopt evidence-based treatments: reliability and validity testing of the evidence-based treatment intentions scale

Nathaniel J Williams, Nathaniel J Williams

Abstract

Background: Intentions play a central role in numerous empirically supported theories of behavior and behavior change and have been identified as a potentially important antecedent to successful evidence-based treatment (EBT) implementation. Despite this, few measures of mental health clinicians' EBT intentions exist and available measures have not been subject to thorough psychometric evaluation or testing. This paper evaluates the psychometric properties of the evidence-based treatment intentions (EBTI) scale, a new measure of mental health clinicians' intentions to adopt EBTs.

Methods: The study evaluates the reliability and validity of inferences made with the EBTI using multi-method, multi-informant criterion variables collected over 12 months from a sample of 197 mental health clinicians delivering services in 13 mental health agencies. Structural, predictive, and discriminant validity evidence is assessed.

Results: Findings support the EBTI's factor structure (χ (2) = 3.96, df = 5, p = .556) and internal consistency reliability (α = .80). Predictive validity evidence was provided by robust and significant associations between EBTI scores and clinicians' observer-reported attendance at a voluntary EBT workshop at a 1-month follow-up (OR = 1.92, p < .05), self-reported EBT adoption at a 12-month follow-up (R (2) = .17, p < .001), and self-reported use of EBTs with clients at a 12-month follow-up (R (2) = .25, p < .001). Discriminant validity evidence was provided by small associations with clinicians' concurrently measured psychological work climate perceptions of functionality (R (2) = .06, p < .05), engagement (R (2) = .06, p < .05), and stress (R (2) = .00, ns).

Conclusions: The EBTI is a practical and theoretically grounded measure of mental health clinicians' EBT intentions. Scores on the EBTI provide a basis for valid inferences regarding mental health clinicians' intentions to adopt EBTs. Discussion focuses on research and practice applications.

Keywords: Adoption; Evidence-based treatment; Intention; Measurement; Mental health.

References

    1. Chaudoir SR, Dugan AG, Barr CH. Measuring factors affecting implementation of health innovations: a systematic review of structural, organizational, provider, patient, and innovation level measures. Implement Sci. 2013;8:22. doi: 10.1186/1748-5908-8-22.
    1. Proctor EK, Brownson RC. Measurement issues in dissemination and implementation research. In: Brownson RC, Colditz GA, Proctor EK, editors. Dissemination and implementation research in health: translating science into practice. New York, NY: Guildford Press; 2014. pp. 261–280.
    1. Fishbein M, Triandis HC, Kanfer FH, Becker M, Middlestadt SE, Eichler A. Factors influencing behavior and behavior change. In: Baum A, Revenson TA, Singer JE, editors. Handbook of health psychology. Mahwah, NJ: Lawrence Erlbaum Associates; 2001. pp. 3–17.
    1. Riemer M, Rosof-Williams J, Bickman L. Theories related to changing clinician practice. Child Adolesc Psychiatr Clin North Am. 2005;14:241–254. doi: 10.1016/j.chc.2004.05.002.
    1. Michie S, Johnston M, Abraham C, Lawton R, Parker D, Walker A. Making psychological theory useful for implementing evidence based practice: a consensus approach. Qual Saf Health Care. 2005;14:26–33. doi: 10.1136/qshc.2004.011155.
    1. Fishbein M, Ajzen I. Belief, attitude, intention and behavior: an introduction to theory and research. Reading, MA: Addison Wesley; 1975.
    1. Ajzen I. The theory of planned behavior. Organ Behav Hum Decis Process. 1991;50:179–211. doi: 10.1016/0749-5978(91)90020-T.
    1. Bandura A. Social foundations of thought and action: a social cognitive theory. Englewood Cliffs, NJ: Prentice-Hall; 1986.
    1. Triandis C. Values, attitudes, and interpersonal behavior. In: Howe HE, Page M, editors. Nebraska symposium of motivation. Lincoln, NE: University of Nebraska Press; 1980. pp. 195–259.
    1. Locke EA, Latham GP. Building a practically useful theory of goal setting and task motivation: a 35-year odyssey. Am Psychol. 2002;57:705–717. doi: 10.1037/0003-066X.57.9.705.
    1. Michie S, van Stralen MM, West R. The behavior change wheel: a new method for characterizing and designing behavior change interventions. Implement Sci. 2011;6:42. doi: 10.1186/1748-5908-6-42.
    1. Perkins MB, Jensen PS, Jaccard J, Gollwitzer P, Oettingen G, Pappadopulos E, et al. Applying theory-driven approaches to understanding and modifying clinicians’ behavior: what do we know? Psychiatr Serv. 2007;58:342–348. doi: 10.1176/ps.2007.58.3.342.
    1. Martinez RG, Lewis CC, Weiner BJ. Instrumentation issues in implementation science. Implement Sci. 2014;9:118. doi: 10.1186/s13012-014-0118-8.
    1. Webb TL, Sheeran P. Does changing behavioral intentions engender behavior change? A meta-analysis of the experimental evidence. Psychol Bull. 2006;132:249–268. doi: 10.1037/0033-2909.132.2.249.
    1. Carr JZ, Schmidt AM, Ford JK, DeShon RP. Climate perceptions matter: a meta-analytic path analysis relating molar climate, cognitive and affective states, and individual level work outcomes. J Appl Psychol. 2003;88:605–619. doi: 10.1037/0021-9010.88.4.605.
    1. James LR, Choi CC, Ko CI, McNeil PK, Minton MK, Wright MA, et al. Organizational and psychological climate: a review of theory and research. Eur J Work Organ Psychol. 2008;17:5–32. doi: 10.1080/13594320701662550.
    1. Jaccard J, Litardo HA, Wan CK. Subjective culture and social behavior. In: Adamopoulos J, Kashima Y, editors. Social Psychology and Cultural Context. Thousand Oaks, CA: Sage Publications; 1999. pp. 95–106.
    1. Sheeran P. Intention-behavior relations: a conceptual and empirical review. In: Stroebe W, Hewstone M, editors. European review of social psychology. London: Wiley; 2002. pp. 1–36.
    1. Eccles MP, Hrisos S, Francis J, Kaner EF, Dickinson HO, Beyer F, et al. Do self-reported intentions predict clinicians’ behavior: a systematic review. Implement Sci. 2006;1:28. doi: 10.1186/1748-5908-1-28.
    1. Godin G, Bélanger-Gravel A, Eccles M, Grimshaw J. Healthcare professionals’ intentions and behaviors: a systematic review of studies based on social cognitive theories. Implement Sci. 2008;3:36. doi: 10.1186/1748-5908-3-36.
    1. Aarons GA, Cafri G, Lugo L, Sawitzky A. Expanding the domains of attitudes toward evidence-based practice: the evidence based practice attitude scale-50. Adm Policy Ment Health. 2012;39:331–340. doi: 10.1007/s10488-010-0302-3.
    1. Borntrager CF, Chorpita BF, Higa-McMillan C, Weisz J. Provider attitudes toward evidence-based practices: are the concerns with the evidence or with the manuals? Psychiatr Serv. 2009;60:677–681. doi: 10.1176/ps.2009.60.5.677.
    1. Williams NJ, Glisson C. The role of organizational culture and climate in the dissemination and implementation of empirically supported treatments for youth. In: Beidas RS, Kendall PC, editors. Dissemination and implementation of evidence-based practices in child and adolescent mental health. New York, NY: Oxford University Press; 2014. pp. 61–81.
    1. Aarons GA, Ehrhart MG, Farahnak LR, Hurlburt MS. Leadership and organizational change for implementation (LOCI): a randomized mixed method pilot study of a leadership and organization development intervention for evidence-based practice implementation. Implement Sci. 2015;10:11. doi: 10.1186/s13012-014-0192-y.
    1. Glisson C, Williams NJ. Assessing and changing organizational social contexts for effective mental health services. Annu Rev Public Health. 2015;36:507–523. doi: 10.1146/annurev-publhealth-031914-122435.
    1. Williams NJ. Multilevel mechanisms of implementation strategies in mental health: integrating theory, research, and practice. Adm Policy Ment Health. 2015; doi: 10.1007/s10488-015-0693-2.
    1. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011;38:65–76. doi: 10.1007/s10488-010-0319-7.
    1. Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Health. 2011;38:4–23. doi: 10.1007/s10488-010-0327-7.
    1. Francis JJ, Eccles MP, Johnston M, Walker A, Grimshaw J, Foy R, et al. Constructing questionnaires based on the theory of planned behavior: a manual for health services researchers. Newcastle upon Tyne. UK: University of Newcastle; 2004.
    1. Armitage CJ, Conner M. Efficacy of the theory of planned behavior: a meta-analytic review. Br J Soc Psychol. 2001;40:471–499. doi: 10.1348/014466601164939.
    1. DeVellis FF. Scale development: theory and applications. Thousand Oaks, CA: Sage Publications; 2011.
    1. Glisson C, Landsverk J, Schoenwald S, Kelleher K, Hoagwood KE, Mayberg S, et al. Assessing the organizational social context (OSC) of mental health services: implications for research and practice. Adm Policy Ment Health. 2008;35:98–113. doi: 10.1007/s10488-007-0148-5.
    1. Ehrhart MG, Aarons GA, Farahnak LR. Assessing the organizational context for EBP implementation: the development and validity testing of the implementation climate scale (ICS) Implement Sci. 2014;9:157. doi: 10.1186/s13012-014-0157-1.
    1. Aarons GA, Glisson C, Green PD, Hoagwood K, Kelleher KJ, Landsverk JA, et al. The organizational social context of mental health services and clinician attitudes toward evidence-based practice: a United States national study. Implement Sci. 2012;7:56. doi: 10.1186/1748-5908-7-56.
    1. Weisz JR, Chorpita BF, Palinkas LA, Schoenwald SK, Miranda J, Bearman SK, et al. Testing standard and modular designs for psychotherapy treating depression, anxiety, and conduct problems in youth: a randomized effectiveness trial. Arch Gen Psychiatry. 2012;69:274–282. doi: 10.1001/archgenpsychiatry.2011.147.
    1. Walrath C, Sheehan D, Holden EW, Hernandez M, Blau G. Evidence-based treatment: provider knowledge, training, and practice. J Behav Health Serv Res. 2006;33:1–10. doi: 10.1007/s11414-005-9008-9.
    1. Aarons GA, Sommerfeld DH, Walrath-Greene CM. Evidence-based practice implementation: the impact of public versus private sector organization type on organizational support, provider attitudes, and adoption of evidence-based practice. Implement Sci. 2009;4:83. doi: 10.1186/1748-5908-4-83.
    1. Williams NJ, Glisson C. Testing a theory of organizational culture, climate and youth outcomes in child welfare systems: a United States national study. Child Abuse Negl. 2014;38:757–767. doi: 10.1016/j.chiabu.2013.09.003.
    1. Glisson C, Hemmelgarn A, Green P, Williams NJ. Randomized trial of the availability, responsiveness and continuity (ARC) organizational intervention for improving youth outcomes in community mental health programs. J Am Acad Child Adolesc Psychiatry. 2013;52:493–500. doi: 10.1016/j.jaac.2013.02.005.
    1. Williams NJ, Glisson C. Reducing turnover is not enough: the need for proficient organizational cultures to support positive youth outcomes in child welfare. Child Youth Serv Rev. 2013;35:1871–1877. doi: 10.1016/j.childyouth.2013.09.002.
    1. Olin SS, Williams NJ, Pollock M, Armusewicz K, Kutash K, Glisson C, et al. Quality indicators for family support services and their relationship to organizational social context. Adm Policy Ment Health. 2014;41:43–54. doi: 10.1007/s10488-013-0499-z.
    1. Muthén LK, Muthén BO. Mplus user’s guide. 7th ed. Los Angeles, CA: Muthén & Muthén; 1998–2012.
    1. Kline RB. Principles and practice of structural equation modeling. 3. New York, NY: Guildford Press; 2011.
    1. Raudenbush SW, Bryk AS, Congdon R. HLM 6 for Windows [Computer software] Skokie, IL: Scientific Software International, Inc; 2004.
    1. Raudenbush SW, Bryk AS. Hierarchical linear models: applications and data analysis. 2. Thousand Oaks, CA: Sage Publications; 2002.
    1. Klein KJ, Sorra JS. The challenge of innovation implementation. Acad Manage Rev. 1996;21:1055–1083.
    1. Weiner BJ, Belden CM, Bergmire DM, Johnston M. The meaning and measurement of implementation climate. Implement Sci. 2011;6:78. doi: 10.1186/1748-5908-6-78.
    1. Lewis CC, Scott K, Marti CN, Marriott BR, Kroenke K, Putz JW, et al. Implementing measurement-based care (iMBC) for depression in community mental health: a dynamic cluster randomized trial study protocol. Implement Sci. 2015;10:127. doi: 10.1186/s13012-015-0313-2.
    1. American Psychological Association: Policy statement on evidence-based practice in psychology. (2005). Accessed 28 Jan 2016.
    1. Podsakoff PM, MacKenzie SB, Lee J, Podsakoff NP. Common method bias in behavioral research: a critical review of the literature and recommended remedies. J Appl Psychol. 2003;88:879–903. doi: 10.1037/0021-9010.88.5.879.

Source: PubMed

Sponsorer og samarbejdspartnere

Medicinske tilstande

Narkotikainterventioner

3
Abonner