Context matters: measuring implementation climate among individuals and groups

Sara R Jacobs, Bryan J Weiner, Alicia C Bunger, Sara R Jacobs, Bryan J Weiner, Alicia C Bunger

Abstract

Background: It has been noted that implementation climate is positively associated with implementation effectiveness. However, issues surrounding the measurement of implementation climate, or the extent to which organizational members perceive that innovation use is expected, supported and rewarded by their organization remain. Specifically, it is unclear whether implementation climate can be measured as a global construct, whether individual or group-referenced items should be used, and whether implementation climate can be assessed at the group or organizational level.

Methods: This research includes two cross-sectional studies with data collected via surveys at the individual level. The first study assessed the implementation climate perceptions of physicians participating in the National Cancer Institute's (NCI) Community Clinical Oncology Program (CCOP), and the second study assessed the perceptions of children's behavioral health clinicians implementing a treatment innovation. To address if implementation climate is a global construct, we used confirmatory factor analysis. To address how implementation climate should be measured and at what level, we followed a five-step framework outlined by van Mierlo and colleagues. This framework includes exploratory factor analysis and correlations to assess differences between individual and group-referenced items and intraclass correlations, interrater agreements, and exploratory factor analysis to determine if implementation climate can be assessed at the organizational level.

Results: The confirmatory factor analysis demonstrated that implementation climate is a global construct consisting of items related to expectations, support and rewards. There are mixed results, however, as to whether implementation climate should be measured using individual or group-referenced items. In our first study, where physicians were geographically dispersed and practice independently, there were no differences based on the type of items used, and implementation climate was an individual level construct. However, in the second study, in which clinicians practice in a central location and interact more frequently, group-referenced items may be appropriate. In addition, implementation climate could be considered an organizational level construct.

Conclusions: The results are context-specific. Researchers should carefully consider the study setting when measuring implementation climate. In addition, more opportunities are needed to validate this measure and understand how well it predicts and explains implementation effectiveness.

Figures

Figure 1
Figure 1
Five-step process to determine group-level construct from individual data.
Figure 2
Figure 2
Example of second order CFA model for individually referenced items.
Figure 3
Figure 3
Example of second order CFA model for group-referenced items.

References

    1. Klein KJ, Sorra JS. The challenge of innovation implementation. Acad Manage Rev. 1996;21:1055–1080.
    1. Greenhalgh T. Diffusion of innovations in health service organisations: a systematic literature review. Malden, Mass: BMJ Books/Blackwell Pub; 2005.
    1. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50. doi: 10.1186/1748-5908-4-50.
    1. Holahan PJ, Aronson ZH, Jurkat MP, Schoorman FD. Implementing computer technology: a multiorganizational test of Klein and Sorra’s model. J Eng Technol Manag. 2004;21:31–50. doi: 10.1016/j.jengtecman.2003.12.003.
    1. Dong LY, Neufeld DJ, Higgins C. Testing Klein and Sorra’s innovation implementation model: an empirical examination. J Eng Technol Manag. 2008;25:237–255. doi: 10.1016/j.jengtecman.2008.10.006.
    1. Osei-Bryson KM, Dong LY, Ngwenyama O. Exploring managerial factors affecting ERP implementation: an investigation of the Klein-Sorra model using regression splines. Inf Syst J. 2008;18:499–527. doi: 10.1111/j.1365-2575.2008.00309.x.
    1. Damschroder LM, Goodrich DE, Robinson CH, Fletcher CE, Lowery JC. A systematic exploration of differences in contextual factors related to implementing the MOVE! weight management program in the VA: a mixed methods study. BMC Health Serv Res. 2011;11:248. doi: 10.1186/1472-6963-11-248.
    1. Damschroder LM, Lowery JC. Evaluation of a large-scale weight management program using the consolidated framework for implementation research. Implement Sci. 2013;8:51. doi: 10.1186/1748-5908-8-51.
    1. Teal R, Bergmire DM, Johnston M, Weiner BJ. Implementing community-based provider participation in research: an empirical study. Implement Sci. 2012;7:41. doi: 10.1186/1748-5908-7-41.
    1. Weiner BJ, Haynes-Maslow L, Kahwati LC, Kinsinger L, Campbell MK. Implementing the MOVE! Weight- management program in the Veterans health administration, 2007-2010: a qualitative study. Prev Chronic Dis. 2012;9:E16.
    1. Helfrich CD, Weiner BJ, McKinney MM, Minasian L. Determinants of implementation effectiveness: exploring an adapted model for complex innovations. Med Care Res Rev. 2007;64:279–303. doi: 10.1177/1077558707299887.
    1. Weiner BJ, Belden CM, Bergmire DM, Johnston M. The meaning and measurement of implementation climate. Implement Sci. 2011;6:78. doi: 10.1186/1748-5908-6-78.
    1. Klein KJ, Conn AB, Sorra JS. Implementing computerized technology: an organizational analysis. J Appl Psychol. 2001;86:811–824.
    1. Glick WH. Conceptualizing and measuring organizational and psychological climate - pitfalls in multilevel research. Acad Manag Rev. 1985;10:601–616.
    1. Glisson C, James LR. The cross-level effects of culture and climate in human service teams. J Organ Behav. 2002;23:767–794. doi: 10.1002/job.162.
    1. Glisson C, Green P, Williams NJ. Assessing the Organizational Social Context (OSC) of child welfare systems: implications for research and practice. Child Abuse Negl. 2012;36:621–632. doi: 10.1016/j.chiabu.2012.06.002.
    1. Glisson C, Green P. Organizational climate, services, and outcomes in child welfare systems. Child Abuse Negl. 2011;35:582–591. doi: 10.1016/j.chiabu.2011.04.009.
    1. Glisson C, Dukes D, Green P. The effects of the ARC organizational intervention on caseworker turnover, climate, and culture in children’s service systems. Child Abuse Negl. 2006;30:855–880. doi: 10.1016/j.chiabu.2005.12.010.
    1. Aarons GA, Glisson C, Green PD, Hoagwood K, Kelleher KJ, Landsverk JA. Research Network on Youth Mental Health. The organizational social context of mental health services and clinician attitudes toward evidence-based practice: a United States national study. Implement Sci. 2012;7:56. doi: 10.1186/1748-5908-7-56.
    1. Klein KJ, Conn AB, Smith DB, Sorra JS. Is everyone in agreement? An exploration of within-group agreement in employee perceptions of the work environment. J Appl Psychol. 2001;86:3–16.
    1. Baltes BB, Zhdanova LS, Parker CP. Psychological climate: a comparison of organizational and individual level referents. Human Relations. 2009;62:669–700. doi: 10.1177/0018726709103454.
    1. Minasian LM, Carpenter WR, Weiner BJ, Anderson DE, McCaskill-Stevens W, Nelson S, Whitman C, Kelaghan J, O’Mara AM, Kaluzny AD. Translating research into evidence-based practice: the National Cancer Institute Community Clinical Oncology Program. Cancer. 2010;116:4440–4449. doi: 10.1002/cncr.25248.
    1. Kaluzny AD, Lacey LM, Warnecke R, Morrissey JP, Sondk EJ, Ford L. Accrual of patients to randomized clinical trials. Factors affecting cancer prevention and control research. Int J Technol Assess Health Care. 1994;10:506–516. doi: 10.1017/S0266462300006723.
    1. Carpenter WR, Fortune-Greeley AK, Zullig LL, Lee SY, Weiner BJ. Sustainability and performance of the National Cancer Institute's Community Clinical Oncology Program. Contemp Clin Trials. 2012;33:46–54. doi: 10.1016/j.cct.2011.09.007.
    1. McKinney MM, Weiner BJ, Carpenter WR. Building community capacity to participate in cancer prevention research. Cancer Control. 2006;13:295–302.
    1. McKinney MM, Weiner BJ, Wang V. Recruiting participants to cancer prevention clinical trials: lessons from successful community oncology networks. Oncol Nurse Forum. 2006;33:951–959. doi: 10.1188/06.ONF.951-959.
    1. Cohen JA, Mannarino AP, Knudsen K. Treating sexually abused children: 1 year follow-up of a randomized controlled trial. Child Abuse Negl. 2005;29:135–145. doi: 10.1016/j.chiabu.2004.12.005.
    1. Deblinger E, Stauffer LB, Steer RA. Comparative efficacies of supportive and cognitive behavioral group therapies for young children who have been sexually abused and their nonoffending mothers. Child Maltreat. 2001;6:332–343. doi: 10.1177/1077559501006004006.
    1. Cary CE, McMillen JC. The data behind the dissemination: a systematic review of trauma-focused cognitive behavioral therapy for use with children and youth. Child Youth Serv Rev. 2012;34:748–757. doi: 10.1016/j.childyouth.2012.01.003.
    1. Kilo CM. A framework for collaborative improvement: lessons from the institute for healthcare improvement’s breakthrough series. Qual Manag Health Care. 1998;8:1–13.
    1. Markiewicz J, Ebert L, Ling D, Amaya-Jackson L, Kisiel C. Learning Collaborative Toolkit. Los Angeles, CA, and Durham, NC: National Center for Child Traumatic Stress; .
    1. van Mierlo H, Vermunt JK, Rutte CG. Composing group-level constructs from individual-level survey data. Organ Res Methods. 2009;12:368–392.
    1. StataCorp. Stata Statistical Software: Release 12. College Station, TX: StataCorp LP; 2009.
    1. Norman GR, Streiner DL, editor. PDQ Statistics. 3. London: BC Decker Inc; 2003. Chapter 17: Path Analysis and Structural Equation Modeling; pp. 156–176.
    1. Hox JJ, Bechger TM. An introduction to structural equation modeling. Fam Sci Rev. 2007;11:354–373.
    1. Schreiber JB, Stage FK, King J. Reporting structural equation modeling and confirmatory factor analysis results: a review. J Educ Res. 2006;99:323–337. doi: 10.3200/JOER.99.6.323-338.
    1. Lance CE, Butts MM, Michels LC. The sources of four commonly reported cutoff criteria - What did they really say? Organ Res Methods. 2006;9:202–220. doi: 10.1177/1094428105284919.
    1. Bliese PD. In: Multilevel theory, research, and methods in organizations. Klein KJ, Kozlowski SWJ, editor. San Francisco: Jossey-Bass; 2000. Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis; pp. 349–381.
    1. Cohen A, Doveh E, Eick U. Statistical properties of the rwg(J) index of agreement. Psychol Methods. 2001;6:297–310.
    1. Dunlap WP, Burke MJ, Smith-Crowe K. Accurate tests of statistical significance for RWG and average deviation interrater agreement indexes. J Appl Psychol. 2003;88:356–362.
    1. Dansereau F, Yammarino FJ. Is more discussion about levels of analysis really necessary? When is such discussion sufficient? Leadership Quarterly. 2003;17:537–552.
    1. Altmann R, Huff J, Baltes B, LaCost H, Young S, Parker C. Psychological and organizational climate perceptions: A field experiment of a contemporary distinction. Dallas, TX: The Thirteenth Annual Meeting of the Society for Industrial Organizational Psychology; 1998.
    1. Castro SL. Data analytic methods for the analysis of multilevel questions: a comparison of intraclass correlation coefficients, rwg(j), hierarchical linear modeling, within- and between-analysis, and random group resampling. Leadersh Q. 2002;13:69–93. doi: 10.1016/S1048-9843(01)00105-9.
    1. Gagne P, Hancock GR. Measurement model quality, sample size, and solution propriety in confirmatory factor models. Multivar Behav Res. 2006;41:65–83. doi: 10.1207/s15327906mbr4101_5.

Source: PubMed

スポンサーと協力者

医学的状態

薬物療法

3
購読する