Developing measures to assess constructs from the Inner Setting domain of the Consolidated Framework for Implementation Research

Maria E Fernandez, Timothy J Walker, Bryan J Weiner, William A Calo, Shuting Liang, Betsy Risendal, Daniela B Friedman, Shin Ping Tu, Rebecca S Williams, Sara Jacobs, Alison K Herrmann, Michelle C Kegler, Maria E Fernandez, Timothy J Walker, Bryan J Weiner, William A Calo, Shuting Liang, Betsy Risendal, Daniela B Friedman, Shin Ping Tu, Rebecca S Williams, Sara Jacobs, Alison K Herrmann, Michelle C Kegler

Abstract

Background: Scientists and practitioners alike need reliable, valid measures of contextual factors that influence implementation. Yet, few existing measures demonstrate reliability or validity. To meet this need, we developed and assessed the psychometric properties of measures of several constructs within the Inner Setting domain of the Consolidated Framework for Implementation Research (CFIR).

Methods: We searched the literature for existing measures for the 7 Inner Setting domain constructs (Culture Overall, Culture Stress, Culture Effort, Implementation Climate, Learning Climate, Leadership Engagement, and Available Resources). We adapted items for the healthcare context, pilot-tested the adapted measures in 4 Federally Qualified Health Centers (FQHCs), and implemented the revised measures in 78 FQHCs in the 7 states (N = 327 respondents) with a focus on colorectal cancer (CRC) screening practices. To psychometrically assess our measures, we conducted confirmatory factor analysis models (CFA; structural validity), assessed inter-item consistency (reliability), computed scale correlations (discriminant validity), and calculated inter-rater reliability and agreement (organization-level construct reliability and validity).

Results: CFAs for most constructs exhibited good model fit (CFI > 0.90, TLI > 0.90, SRMR < 0.08, RMSEA < 0.08), with almost all factor loadings exceeding 0.40. Scale reliabilities ranged from good (0.7 ≤ α < 0.9) to excellent (α ≥ 0.9). Scale correlations fell below 0.90, indicating discriminant validity. Inter-rater reliability and agreement were sufficiently high to justify measuring constructs at the clinic-level.

Conclusions: Our findings provide psychometric evidence in support of the CFIR Inner Setting measures. Our findings also suggest the Inner Setting measures from individuals can be aggregated to represent the clinic-level. Measurement of the Inner Setting constructs can be useful in better understanding and predicting implementation in FQHCs and can be used to identify targets of strategies to accelerate and enhance implementation efforts in FQHCs.

Keywords: CFIR; Colorectal cancer screening implementation; Consolidated Framework for Implementation Research; Implementation science; Inner Setting; Measurement of implementation.

Conflict of interest statement

Ethics approval and consent to participate

All study procedures were approved by the Coordinating Center at the University of North Carolina at Chapel Hill, UNC Office of Human Research Ethics 12-1586. The Institutional Review Boards of each CPCRN also approved the study procedures (UCLA Office of the Human Research Protection Program, Colorado Multiple Institution Review Board, Emory University Institutional Review Board, Washington University Institutional Review Board, University of South Carolina Office of Research Compliance, University of Texas Health Science Center at Houston Committee for the Protection of Human Subjects, University of Washington Human Subjects Division).

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Figures

Fig. 1
Fig. 1
Example of multilevel confirmatory factor model for the Leadership Engagement Scale. The item number with B represents clinic-level items

References

    1. Balas EA, Weingarten S, Garb CT, Blumenthal D, Boren SA, Brown GD. Improving preventive care by prompting physicians. Arch Intern Med. 2000;160:301–308. doi: 10.1001/archinte.160.3.301.
    1. McGlynn EA, Asch SM, Adams J, Keesey J, Hicks J, DeCristofaro A, et al. The quality of health care delivered to adults in the United States. N Engl J Med. 2003;348:2635–2645. doi: 10.1056/NEJMsa022615.
    1. Glasgow RE, Klesges LM, Dzewaltowski DA, Bull SS, Estabrooks P. The future of health behavior change research: what is needed to improve translation of research into health promotion practice? Ann Behav Med. 2004;27:3–12. doi: 10.1207/s15324796abm2701_2.
    1. Klabunde CN, Riley GF, Mandelson MT, Frame PS, Brown ML. Health plan policies and programs for colorectal cancer screening: a national profile. Am J Manag Care. 2004;10:273–279.
    1. Emmons KM, Weiner B, Fernandez ME, Tu SP. Systems antecedents for dissemination and implementation: a review and analysis of measures. Health Educ Behav. 2012;39:87–105. doi: 10.1177/1090198111409748.
    1. Chor K, Wisdom J, Olin S, Hoagwood K, Horwitz SM. Measures for predictors of innovation adoption. Admin Pol Ment Health. 2015;42(5):545-73.
    1. Weiner BJ, Amick H, Lee SY. Conceptualization and measurement of organizational readiness for change: a review of the literature in health services research and other fields. Med Care Res Rev. 2008;65:379–436. doi: 10.1177/1077558708317802.
    1. Lewis CC, Fischer S, Weiner BJ, Stanick C, Kim M, Martinez RG. Outcomes for implementation science: an enhanced systematic review of instruments using evidence-based rating criteria. Implement Sci. 2015;10:155. doi: 10.1186/s13012-015-0342-x.
    1. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4:50. doi: 10.1186/1748-5908-4-50.
    1. Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004;82:581–629. doi: 10.1111/j.0887-378X.2004.00325.x.
    1. Lash SJ, Timko C, Curran GM, McKay JR, Burden JL. Implementation of evidence-based substance use disorder continuing care interventions. Psychol Addict Behav. 2011;25:238–251. doi: 10.1037/a0022608.
    1. Harris JR, Brown PK, Coughlin S, Fernandez ME, Hebert JR, Kerner J, et al. The cancer prevention and control research network. Prev Chronic Dis. 2005;2:A21.
    1. Ribisl KM, Fernandez ME, Friedman DB, Hannon PA, Leeman J, Moore A, et al. Impact of the cancer prevention and control research network: accelerating the translation of research into practice. Am J Prev Med. 2017;52:S233–S240. doi: 10.1016/j.amepre.2016.08.026.
    1. Fernandez ME, Melvin CL, Leeman J, Ribisl KM, Allen JD, Kegler MC, et al. The cancer prevention and control research network: an interactive systems approach to advancing cancer control implementation research and practice. Cancer Epidemiol Biomark Prev. 2014;23:2512–2521. doi: 10.1158/1055-9965.EPI-14-0097.
    1. Liang S, Kegler M, Carvalho M, Fernandez M, Weiner B, Jacobs S, et al. Measuring constructs from the consolidated framework for implementation research in the context of increasing colorectal cancer screening at community health centers. Implement Sci. 2015;10(Suppl 1):A10.
    1. Sohng H, Kuniyuki A, Edelson J, Weir R, Song H, Tu S-P. Capability for change at community health centers: an exploratory study on the implementation of an evidence-based intervention. Asian Pac J Cancer Prev. 2013;14:7451–7457. doi: 10.7314/APJCP.2013.14.12.7451.
    1. Nutting PA, Crabtree BF, Miller WL, Stewart EE, Stange KC, Jaen CR. Journey to the patient-centered medical home: a qualitative analysis of the experiences of practices in the National Demonstration Project. Ann Fam Med. 2010;8(Suppl 1):S45–S56. doi: 10.1370/afm.1075.
    1. Nutting PA, Crabtree BF, Stewart EE, Miller WL, Palmer RF, Stange KC, et al. Effect of facilitation on practice outcomes in the National Demonstration Project model of the patient-centered medical home. Ann Fam Med. 2010;8(Suppl 1):S33–S44. doi: 10.1370/afm.1119.
    1. Jaen CR, Crabtree BF, Palmer RF, Ferrer RL, Nutting PA, Miller WL, et al. Methods for evaluating practice change toward a patient-centered medical home. Ann Fam Med. 2010;8(Suppl 1):S9–20. doi: 10.1370/afm.1108.
    1. Lehman WE, Greener JM, Simpson DD. Assessing organizational readiness for change. J Subst Abus Treat. 2002;22:197–209. doi: 10.1016/S0740-5472(02)00233-7.
    1. Patterson MG, West M, Shackleton V, Dawson J, Lawthom R, Maitlis S, et al. Validating the organizational climate measures: links to managerial practices, productivity and innovation. J Organ Behav. 2005;26:379–408. doi: 10.1002/job.312.
    1. Tu S-P, Young VM, Coombs LJ, Williams RS, Kegler MC, Kimura AT, et al. Practice adaptive reserve and colorectal cancer screening best practices at community health center clinics in 7 states. Cancer. 2015;121:1241–1248. doi: 10.1002/cncr.29176.
    1. Brown TA. Confirmatory factor analysis for applied research. 1. New York: The Guilford Press; 2015.
    1. Bentler PM. Comparative fit indexes in structural models. Psychol Bull. 1990;107:238–246. doi: 10.1037/0033-2909.107.2.238.
    1. DeVellis R. Scale development: theory and applications. 3. Thousand Oaks, CA: Sage Publications, Inc; 2012.
    1. Hu L, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct Equ Model Multidiscip J. 1999;6:1–55. doi: 10.1080/10705519909540118.
    1. Muthen BO. Multilevel covariance structure analysis. Sociol Methods Res. 1994;22:376–398. doi: 10.1177/0049124194022003006.
    1. Hox J. Multilevel analysis: techniques and applications. Mahwah, NJ: Lawrence Erlbaum Associates; 2002.
    1. Satorra A, Bentler PM. A scaled difference chi-square test statistic for moment structure analysis. Psychometrika. 2001;66:507–514. doi: 10.1007/BF02296192.
    1. Klein KJ, Kozlowski SW. Multilevel theory, research, and methods in organziations: foundations, extensions, and new directions. San Francisco: Jossey-Bass; 2000.
    1. Klein KJ, Conn AB, Sorra JS. Implementing computerized technology: an organizational analysis. J Appl Psychol. 2001;86:811–824. doi: 10.1037/0021-9010.86.5.811.
    1. Castro SL. Data analytic methods for the analysis of multilevel questions: a comparison of intraclass correlation coefficients, rwg(i), hierarchical lineal modeling, within- and between-analysis, and random group resampling. Leadersh Q. 2002;13:69–93. doi: 10.1016/S1048-9843(01)00105-9.
    1. Lance CE, Butts MM, Michaels LC. The sources of four commonly reported cutoff criteria. Organ Res Methods. 2006;9:202–220. doi: 10.1177/1094428105284919.
    1. Muthen L, Muthen B. Statistical analyses with latent variables. User’s guide 3. 1998.
    1. Weiner BJ, Belden C, Bergmire D, Johnston M. The meaning and measurement of implementation climate. Implement Sci. 2011;6:78. doi: 10.1186/1748-5908-6-78.
    1. Dedrick RF, Greenbaum PE. Multilevel confirmatory factor analysis of a scale measuring interagency collaboration of children’s mental health agencies. J Emot Behav Disord. 2011;19:27–40. doi: 10.1177/1063426610365879.
    1. Ehrhart MG, Torres EM, Wright LA, Martinez SY, Aarons GA. Validating the implementation climate scale (ICS) in child welfare organizations. Child Abuse Neglect. 2016;53:17–26. doi: 10.1016/j.chiabu.2015.10.017.
    1. Clinton-McHarg T, Yoong SL, Tzelepis F, Regan T, Fielding A, Skelton E, et al. Psychometric properties of implementation measures for public health and community settings and mapping of constructs against the Consolidated Framework for Implementation Research: a systematic review. Implement Sci. 2016;11:148. doi: 10.1186/s13012-016-0512-5.
    1. Kalkan A, Roback K, Hallert E, Carlsson P. Factors influencing rheumatologists’ prescription of biological treatment in rheumatoid arthritis: an interview study. Implement Sci. 2014;9:153. doi: 10.1186/s13012-014-0153-5.
    1. Ditty MS, Landes SJ, Doyle A, Beidas RS. It takes a village: a mixed method analysis of Inner Setting variables and dialectical behavior therapy implementation. Admin Pol Ment Health. 2015;42(6):672-81.
    1. Acosta J, Chinman M, Ebener P, Malone PS, Paddock S, Phillips A, et al. An intervention to improve program implementation: findings from a two-year cluster randomized trial of Assets-Getting To Outcomes. Implement Sci. 2013;8:87. doi: 10.1186/1748-5908-8-87.
    1. Helfrich CD, Li YF, Sharp ND, Sales AE. Organizational readiness to change assessment (ORCA): development of an instrument based on the Promoting Action on Research in Health Services (PARIHS) framework. Implementation Science. 2009;4(1):38. doi: 10.1186/1748-5908-4-38.

Source: PubMed

Sponsorzy i współpracownicy

Warunki medyczne

Interwencje lekowe

3
Subskrybuj