Predicting implementation from organizational readiness for change: a study protocol

Christian D Helfrich, Dean Blevins, Jeffrey L Smith, P Adam Kelly, Timothy P Hogan, Hildi Hagedorn, Patricia M Dubbert, Anne E Sales, Christian D Helfrich, Dean Blevins, Jeffrey L Smith, P Adam Kelly, Timothy P Hogan, Hildi Hagedorn, Patricia M Dubbert, Anne E Sales

Abstract

Background: There is widespread interest in measuring organizational readiness to implement evidence-based practices in clinical care. However, there are a number of challenges to validating organizational measures, including inferential bias arising from the halo effect and method bias - two threats to validity that, while well-documented by organizational scholars, are often ignored in health services research. We describe a protocol to comprehensively assess the psychometric properties of a previously developed survey, the Organizational Readiness to Change Assessment.

Objectives: Our objective is to conduct a comprehensive assessment of the psychometric properties of the Organizational Readiness to Change Assessment incorporating methods specifically to address threats from halo effect and method bias.

Methods and design: We will conduct three sets of analyses using longitudinal, secondary data from four partner projects, each testing interventions to improve the implementation of an evidence-based clinical practice. Partner projects field the Organizational Readiness to Change Assessment at baseline (n = 208 respondents; 53 facilities), and prospectively assesses the degree to which the evidence-based practice is implemented. We will conduct predictive and concurrent validities using hierarchical linear modeling and multivariate regression, respectively. For predictive validity, the outcome is the change from baseline to follow-up in the use of the evidence-based practice. We will use intra-class correlations derived from hierarchical linear models to assess inter-rater reliability. Two partner projects will also field measures of job satisfaction for convergent and discriminant validity analyses, and will field Organizational Readiness to Change Assessment measures at follow-up for concurrent validity (n = 158 respondents; 33 facilities). Convergent and discriminant validities will test associations between organizational readiness and different aspects of job satisfaction: satisfaction with leadership, which should be highly correlated with readiness, versus satisfaction with salary, which should be less correlated with readiness. Content validity will be assessed using an expert panel and modified Delphi technique.

Discussion: We propose a comprehensive protocol for validating a survey instrument for assessing organizational readiness to change that specifically addresses key threats of bias related to halo effect, method bias and questions of construct validity that often go unexplored in research using measures of organizational constructs.

Figures

Figure 1
Figure 1
ORCA scales, subscales and outcomes. This figure illustrates the composition of the ORCA scales and their hypothesized relationship to organizational readiness for change, and subsequently to implementation outcomes.

References

    1. Berwick DM. Disseminating innovations in health care. Jama. 2003;289(15):1969–75. doi: 10.1001/jama.289.15.1969.
    1. Institute of Medicine Committee on Quality of Health Care in America. Crossing the quality chasm: a new health system for the 21st century. Washington, D.C.: National Academy Press; 2001.
    1. Balas EA, Boren SA. In: Managing clinical knowledge for health care improvement. Yearbook of Medical Informatics. Medicine NLo. Bethesda MD, editor. 2000. pp. 65–70.
    1. Dopson S, Locock L, Chambers D, Gabbay J. Implementation of evidence-based medicine: evaluation of the Promoting Action on Clinical Effectiveness programme. Journal of Health Services Research and Policy. 2001;6:23–31. doi: 10.1258/1355819011927161.
    1. Weiner BJ, Savitz LA, Bernard S, Pucci LG. How do integrated delivery systems adopt and implement clinical information systems? Health Care Management Review. 2004;29(1):51–66.
    1. Nutting PA, Miller WL, Crabtree BF, Jaen CR, Stewart EE, Stange KC. Initial Lessons From the First National Demonstration Project on Practice Transformation to a Patient-Centered Medical Home. Ann Fam Med. 2009;7(3):254–260. doi: 10.1370/afm.1002.
    1. Smith ME. Success rates for different types of organizational change. Performance Improvement. 2002;41(1):26–33. doi: 10.1002/pfi.4140410107.
    1. Armenakis AA, Harris SG, Mossholder KW. Creating Readiness for Organizational Change. Human Relations. 1993;46(6):681–703. doi: 10.1177/001872679304600601.
    1. Holt D, Armenakis A, Harris S, Feild H. Research in Organizational Change and Development. JAI Press: Amsterdam, Netherlands; 2006. Toward a comprehensive definition of readiness for change: a review of research and instrumentation.
    1. Weiner BJ, Amick H, Lee SYD. Conceptualization and Measurement of Organizational Readiness for Change: A Review of the Literature in Health Services Research and Other Fields. Med Care Res Rev. 2008;65(4):379–436. doi: 10.1177/1077558708317802.
    1. Rosenzweig P. Misunderstanding the Nature of Company Performance: The Halo Effect and Other Business Delusions. California Management Review. 2007;49(4):6–20.
    1. Spector PE. Method Variance in Organizational Research. Organizational Research Methods. 2006;9(2):221–232. doi: 10.1177/1094428105284955.
    1. Hinkin TR. A Review of Scale Development Practices in the Study of Organizations. Journal of Management. 1995;21(5):967–988.
    1. Helfrich C, Li Y-F, Sharp N, Sales A. Organizational readiness to change assessment (ORCA): Development of an instrument based on the Promoting Action on Research in Health Services (PARIHS) framework. Implementation Science. 2009;4(1):38. doi: 10.1186/1748-5908-4-38.
    1. Weiner BJ, Amick H, Lee SYD. Conceptualization and measurement of organizational readiness for change: A review of the literature in health services research and other fields. Medical Care Research and Review. 2008;65(4):379–436. doi: 10.1177/1077558708317802.
    1. Scott SD, Plotnikoff RC, Karunamuni N, Bize R, Rodgers W. Factors influencing the adoption of an innovation: An examination of the uptake of the Canadian Heart Health Kit (HHK) Implement Sci. 2008;3(1):41. doi: 10.1186/1748-5908-3-41.
    1. Hedrick SC, Chaney EF, Felker B, Liu CF, Hasenberg N, Heagerty P, Buchanan J, Bagala R, Greenberg D, Paden G, Fihn SD, Katon W. Effectiveness of collaborative care depression treatment in Veterans' Affairs primary care. J Gen Intern Med. 2003;18(1):9–16. doi: 10.1046/j.1525-1497.2003.11109.x.
    1. Klein K, Sorra J. The challenge of innovation implementation. Academy of Management Review. 1996;21(4):1055–1080.
    1. Lehman WEK, Greener JM, Simpson DD. Assessing organizational readiness for change. Journal of Substance Abuse Treatment. 2002;22(4):197–209. doi: 10.1016/S0740-5472(02)00233-7.
    1. Gustafson DH, Sainfort F, Eichler M, Adams L, Bisognano M, Steudel H. Developing and Testing a Model to Predict Outcomes of Organizational Change. Health Services Research. 2003;38(2):751–776. doi: 10.1111/1475-6773.00143.
    1. Molfenter T, Gustafson D, Kilo C, Bhattacharya A, Olsson J. Prospective evaluation of a Bayesian model to predict organizational change. Health Care Manage Rev. 2005;30(3):270–9.
    1. Fuller BE, Rieckmann T, Nunes EV, Miller M, Arfken C, Edmundson E, McCarty D. Organizational Readiness for Change and opinions toward treatment innovations. Journal of Substance Abuse Treatment. 2007;33(2):183–192. doi: 10.1016/j.jsat.2006.12.026.
    1. Jansen KJ. From persistence to pursuit: A longitudinal examination of momentum during the early stages of strategic change. Organization Science. 2004;15(3):276–294. doi: 10.1287/orsc.1040.0064.
    1. Ingersoll GL, Kirsch JC, Merk SE, Lightfoot J. Relationship of Organizational Culture and Readiness for Change to Employee Commitment to the Organization. Journal of Nursing Administration. 2000;30(1):11–20. doi: 10.1097/00005110-200001000-00004.
    1. Herscovitch L, Meyer JP. Commitment to organizational change: Extension of a three-component model. Journal of Applied Psychology. 2002;87(3):474–487.
    1. Holt DT, Armenakis AA, Feild HS, Harris SG. Readiness for Organizational Change: The Systematic Development of a Scale. Journal of Applied Behavioral Science. 2007;43(2):232–255. doi: 10.1177/0021886306295295.
    1. Molla A, Licker PS. eCommerce adoption in developing countries: a model and instrument. Information & Management. 2005;42(6):877–899. doi: 10.1016/j.im.2004.09.002.
    1. Sen A, Sinha AP, Ramamurthy K. Data warehousing process maturity: An exploratory study of factors influencing user perceptions. IEEE Transactions on Engineering Management. 2006;53(3):440–455.
    1. Scientific Advisory Committee of the Medical Outcomes Trust. Assessing Health Status and Quality-of-Life Instruments: Attributes and Review Criteria. Quality of Life Research. 2002;11(3):193–205. doi: 10.1023/A:1015291021312.
    1. Nunnally JC, Bernstein IH. Psychometric Theory. New York, NY: McGraw-Hill Inc.; 1994.
    1. Tesluk P, Mathieu JE, Zaccaro SJ, Marks M. In: Task and aggregation issues in the analysis and assessment of team performance, in Team performance assessment and measurement: theory, methods, and applications. Brannick MT, Salas E, Prince C, editor. Lawrence Erlbaum Associates: Mahwah, N.J; 1997. pp. 197–224.
    1. Hinkin TR. A Brief Tutorial on the Development of Measures for Use in Survey Questionnaires. Organizational Research Methods. 1998;1(1):104–121. doi: 10.1177/109442819800100106.
    1. Bahtsevani C, Willman A, Khalaf A, Östman M. Developing an instrument for evaluating implementation of clinical practice guidelines: a test-retest study. Journal of Evaluation in Clinical Practice. 2008;9999(9999)
    1. Cummings GG, Estabrooks CA, Midodzi WK, Wallin L, Hayduk L. Influence of organizational characteristics and context on research utilization. Nurs Res. 2007;56(4 Suppl):S24–39.
    1. Trochim WMK. The Research Methods Knowledge Base. Atomic Dog ; 2000.
    1. Lance CE, Dawson B, Birkelbach D, Hoffman BJ. Method Effects, Measurement Error, and Substantive Conclusions. Organizational Research Methods. 2010;13(3):435–455. doi: 10.1177/1094428109352528.
    1. Podsakoff P, MacKenzie S, Lee J, Podsakoff N. Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology. 2003;88(5):879.
    1. Thorndike EL. A constant error in psychological ratings. Journal of Applied Psychology. 1920. pp. 25–29.
    1. Staw BM. Attribution of the "causes" of performance: a general alternative interpretation of cross-sectional research on organizations. [Urbana]: College of Commerce and Business Administration, University of Illinois at Urbana-Champaign; 1974.
    1. Hagedorn HJ, Heideman PW. The relationship between baseline Organizational Readiness to Change Assessment subscale scores and implementation of hepatitis prevention services in substance use disorders treatment clinics: a case study. Implement Sci. 2010;5(1):46. doi: 10.1186/1748-5908-5-46.
    1. Yu W, Ravelo A, Wagner TH, Phibbs CS, Bhandari A, Chen S, Barnett PG. Prevalence and Costs of Chronic Conditions in the VA Health Care System. Med Care Res Rev. 2003;60(3_suppl):146S–167.
    1. Kirchner JE, Curran GM, Aikens J. Datapoints: detecting depression in VA primary care clinics. Psychiatr Serv. 2004;55(4):350. doi: 10.1176/appi.ps.55.4.350.
    1. Kauth MR, Sullivan G, Blevins D, Cully JA, Landes RD, Said Q, Teasdale TA. Employing external facilitation to implement cognitive behavioral therapy in VA clinics: a pilot study. Implement Sci. 2010;5(10):75.
    1. Cohen J. Statistical power analysis for the behavioral sciences. Hillsdale, N.J.: L. Erlbaum Associates; 1988.
    1. Blow FC, McCarthy JF, Valenstein M, Visnic S, Gillon L. Care for Veterans with Psychosis in the Veterans Health Administration, FY06 8th annual National Psychosis Registry. 2007.
    1. Duncan E, Dunlop BW, Boshoven W, Woolson SL, Hamer RM, Phillips LS. Relative risk of glucose elevation during antipsychotic exposure in a Veterans Administration population. Int Clin Psychopharmacol. 2007;22(1):1–11. doi: 10.1097/01.yic.0000224794.29029.67.
    1. Lambert BL, Cunningham FE, Miller DR, Dalack GW, Hur K. Diabetes risk associated with use of olanzapine, quetiapine, and risperidone in veterans health administration patients with schizophrenia. Am J Epidemiol. 2006;164(7):672–81. doi: 10.1093/aje/kwj289.
    1. Leslie DL, Rosenheck RA. Incidence of newly diagnosed diabetes attributable to atypical antipsychotic medications. Am J Psychiatry. 2004;161(9):1709–11. doi: 10.1176/appi.ajp.161.9.1709.
    1. Newcomer JW. Second-generation (atypical) antipsychotics and metabolic effects: a comprehensive literature review. CNS Drugs. 2005;19(Suppl 1):1–93.
    1. Sernyak MJ, Gulanski B, Rosenheck R. Undiagnosed hyperglycemia in patients treated with atypical antipsychotics. J Clin Psychiatry. 2005;66(11):1463–7. doi: 10.4088/JCP.v66n1117.
    1. Stroup TS, Lieberman JA, McEvoy JP, Swartz MS, Davis SM, Rosenheck RA, Perkins DO, Keefe RS, Davis CE, Severe J, Hsiao JK. Effectiveness of olanzapine, quetiapine, risperidone, and ziprasidone in patients with chronic schizophrenia following discontinuation of a previous atypical antipsychotic. Am J Psychiatry. 2006;163(4):611–22. doi: 10.1176/appi.ajp.163.4.611.
    1. Consensus development conference on antipsychotic drugs and obesity and diabetes. J Clin Psychiatry. 2004;65(2):267–72.
    1. Washington DOoQPaPCS, Department of Defense, editor. Department of Veterans Affairs and Health Affairs DoD. VA/DoD Clinical Practice Guideline for the Diagnosis and Management of Dyslipidemia. 2006.
    1. Washington DVEES, Offices of Quality & Performance and Patient Care Services, Department of Defense, editor. Department of Veterans Affairs D. VA/DoD Clinical Practice Guideline for Screening and Management of Overweight and Obesity. 2006.
    1. Marder SR, Essock SM, Miller AL, Buchanan RW, Casey DE, Davis JM, Kane JM, Lieberman JA, Schooler NR, Covell N, Stroup S, Weissman EM, Wirshing DA, Hall CS, Pogach L, Pi-Sunyer X, Bigger JT Jr, Friedman A, Kleinberg D, Yevich SJ, Davis B, Shon S. Physical health monitoring of patients with schizophrenia. Am J Psychiatry. 2004;161(8):1334–49. doi: 10.1176/appi.ajp.161.8.1334.
    1. Washington DOoQaPp, editor. Veterans Health Administration DoVAaHA, Department of Defense. VA/DoD Clinical Practice Guideline for the Management of Diabetes Mellitus (DM) in Primary Care. 2003.
    1. VA/DoD Evidence-Based Clinical Practice Guideline Working Group. Washington DCOoQaPP, editor. Veterans Health Administration DoVAaHA, Department of Defense. Management of Persons with Psychoses. 2004.
    1. Jennex A, Gardner DM. Monitoring and management of metabolic risk factors in outpatients taking antipsychotic drugs: a controlled study. Can J Psychiatry. 2008;53(1):34–42.
    1. Morrato EH, Newcomer JW, Allen RR, Valuck RJ. Prevalence of baseline serum glucose and lipid testing in users of second-generation antipsychotic drugs: a retrospective, population-based study of Medicaid claims data. J Clin Psychiatry. 2008;69(2):316–22. doi: 10.4088/JCP.v69n0219.
    1. Reid LD. Lipid Profile Monitoring in Veterans Living with Schizophrenia-related disorders and treated with second-generation antipsychotics: Findings from a VA-based population. 2007.
    1. Weissman EM, Zhu CW, Schooler NR, Goetz RR, Essock SM. Lipid monitoring in patients with schizophrenia prescribed second-generation antipsychotics. J Clin Psychiatry. 2006;67(9):1323–6. doi: 10.4088/JCP.v67n0901.
    1. Department of Veterans Affairs Office of the Inspector General. Healthcare inspection: Atypical antipsychotic medications and diabetes screening and management. 2007.
    1. Rubenstein LV, Parker LE, Meredith LS, Altschuler A, dePillis E, Hernandez J, Gordon NP. Understanding team-based quality improvement for depression in primary care. Health Serv Res. 2002;37(4):1009–29. doi: 10.1034/j.1600-0560.2002.63.x.
    1. Smith J, Spollen J, Owen R. In: Facilitation in implementing evidence-based practices for schizophrenia: Research and clinical leader perspectives. AcademyHealth: Orlando, FL, editor. 2007.
    1. Smith PC, Kendall LM, Hulin CL. The measurement of satisfaction in work and retirement; a strategy for the study of attitudes. Chicago, Ill.: Rand McNally; 1969.
    1. Dhopesh VP, Taylor KR, Burke WM. Survey of hepatitis B and C in addiction treatment unit. Am J Drug Alcohol Abuse. 2000;26(4):703–7. doi: 10.1081/ADA-100101903.
    1. Abraham HD, Degli-Esposti S, Marino L. Seroprevalence of hepatitis C in a sample of middle class substance abusers. J Addict Dis. 1999;18(4):77–87. doi: 10.1300/J069v18n04_07.
    1. Hagedorn H, Dieperink E, Dingmann D, Durfee J, Ho SB, Isenhart C, Rettmann N, Willenbring M. Integrating hepatitis prevention services into a substance use disorder clinic. Journal of Substance Abuse Treatment. 2007;32(4):391–398. doi: 10.1016/j.jsat.2006.10.004.
    1. Almasio PL, Amoroso P. HAV infection in chronic liver disease: a rationale for vaccination. Vaccine. 2003;21(19-20):2238–41. doi: 10.1016/S0264-410X(03)00139-7.
    1. Reiss G, Keeffe EB. Review article: hepatitis vaccination in patients with chronic liver disease. Aliment Pharmacol Ther. 2004;19(7):715–27. doi: 10.1111/j.1365-2036.2004.01906.x.
    1. Stetler CB, Legro MW, Rycroft-Malone J, Bowman C, Curran G, Guihan M, Hagedorn H, Pineros S, Wallace CM. Role of "external facilitation" in implementation of research findings: a qualitative evaluation of facilitation experiences in the Veterans Health Administration. Implement Sci. 2006;1:23. doi: 10.1186/1748-5908-1-23.
    1. Streiner DL, Norman GR. Health measurement scales: a practical guide to their development and use. Oxford; New York: Oxford University Press; 2003.
    1. Bernard HR. Social Research Methods: Qualitative and Quantitative Approaches. Thousand Oaks, CA: Sage; 2000.
    1. Schriesheim CA, Powers KJ, Scandura TA, Gardiner CC, Lankau MJ. Improving construct measurement in management research: Comments and a quantitative approach for assessing the theoretical content adequacy of paper-and-pencil survey-type instruments. Journal of Management. 1993;19(2):385–417.
    1. Jöreskog KG, Sörbom D. LISREL 8: structural equation modeling with the SIMPLIS command language. Chicago, Ill.; Hillsdale, N.J.: Scientific Software International; distributed by L. Erlbaum Associates; 1995.
    1. Haidet P, Kelly PA, Chou C. Characterizing the patient-centeredness of hidden curricula in medical schools: development and validation of a new measure. Acad Med. 2005;80(1):44–50. doi: 10.1097/00001888-200501000-00012.
    1. Nagy M. Using a single-item approach to measure facet job satisfaction. Journal of Occupational and Organizational Psychology. 2002;75(1):77–86. doi: 10.1348/096317902167658.
    1. Reckase MD. The past and future of multidimensional item response theory. Applied Psychological Measurement. 1997;21:25–36. doi: 10.1177/0146621697211002.
    1. Molla A, Licker PS. Perceived e-readiness factors in e-commerce adoption: An empirical investigation in a developing country. International Journal of Electronic Commerce. 2005;10(1):83–110.
    1. Saldana L, Chapman JE, Henggeler SW, Rowland MD. The Organizational Readiness for Change scale in adolescent programs: Criterion validity. Journal of Substance Abuse Treatment. 2007;33(2):159–169. doi: 10.1016/j.jsat.2006.12.029.
    1. Rampazzo L, Angeli MD, Serpelloni G, Simpson DD, Flynn PM. Italian Survey of Organizational Functioning and Readiness for Change: A Cross-Cultural Transfer of Treatment Assessment Strategies. European Addiction Research. 2006;12:76–181.

Source: PubMed

Sponsorer og samarbeidspartnere

Medisinsk tilstand

Legemiddelintervensjoner

3
Abonnere