Application of balanced scorecard in the evaluation of a complex health system intervention: 12 months post intervention findings from the BHOMA intervention: a cluster randomised trial in Zambia

Wilbroad Mutale, Jeffrey Stringer, Namwinga Chintu, Roma Chilengi, Margaret Tembo Mwanamwenge, Nkatya Kasese, Dina Balabanova, Neil Spicer, James Lewis, Helen Ayles, Wilbroad Mutale, Jeffrey Stringer, Namwinga Chintu, Roma Chilengi, Margaret Tembo Mwanamwenge, Nkatya Kasese, Dina Balabanova, Neil Spicer, James Lewis, Helen Ayles

Abstract

Introduction: In many low income countries, the delivery of quality health services is hampered by health system-wide barriers which are often interlinked, however empirical evidence on how to assess the level and scope of these barriers is scarce. A balanced scorecard is a tool that allows for wider analysis of domains that are deemed important in achieving the overall vision of the health system. We present the quantitative results of the 12 months follow-up study applying the balanced scorecard approach in the BHOMA intervention with the aim of demonstrating the utility of the balanced scorecard in evaluating multiple building blocks in a trial setting.

Methods: The BHOMA is a cluster randomised trial that aims to strengthen the health system in three rural districts in Zambia. The intervention aims to improve clinical care quality by implementing practical tools that establish clear clinical care standards through intensive clinic implementations. This paper reports the findings of the follow-up health facility survey that was conducted after 12 months of intervention implementation. Comparisons were made between those facilities in the intervention and control sites. STATA version 12 was used for analysis.

Results: The study found significant mean differences between intervention(I) and control (C) sites in the following domains: Training domain (Mean I:C; 87.5.vs 61.1, mean difference 23.3, p = 0.031), adult clinical observation domain (mean I:C; 73.3 vs.58.0, mean difference 10.9, p = 0.02 ) and health information domain (mean I:C; 63.6 vs.56.1, mean difference 6.8, p = 0.01. There was no gender differences in adult service satisfaction. Governance and motivation scores did not differ between control and intervention sites.

Conclusion: This study demonstrates the utility of the balanced scorecard in assessing multiple elements of the health system. Using system wide approaches and triangulating data collection methods seems to be key to successful evaluation of such complex health intervention.

Trial number: ClinicalTrials.gov NCT01942278.

Conflict of interest statement

Competing Interests: The authors have declared that no competing interests exist.

Figures

Figure 1. Summary of the BHOMA Intervention…
Figure 1. Summary of the BHOMA Intervention cascade.
Figure 2. BHOMA intervention randomisation chart.
Figure 2. BHOMA intervention randomisation chart.
Figure 3. BHOMA intervention step wedged rollout…
Figure 3. BHOMA intervention step wedged rollout over time.

References

    1. Swanson RC, Cattaneo A, Bradley E, Chunharas S, Atun R, et al. (2012) Rethinking health systems strengthening: key systems thinking tools and strategies for transformational change. Health Policy Plan 27 Suppl 4iv54–61.
    1. English M, Nzinga J, Mbindyo P, Ayieko P, Irimu G, et al. (2011) Explaining the effects of a multifaceted intervention to improve inpatient care in rural Kenyan hospitals–interpretation based on retrospective examination of data from participant observation, quantitative and qualitative studies. Implement Sci 6: 124.
    1. Adam T, De Savigny D (2012) Systems thinking for strengthening health systems in LMICs: Need for a paradigm shift. Health Policy and Planning 27: iv1–iv3.
    1. Agyepong IA, Ko A, Adjei S, Adam T (2012) When “Solution of yesterday become problems of today”Crisis-ridden decision maling in a complex adaptive sytem(CAS)-the additional Duty Hours Allowance in Ghana. Health Policy 27.
    1. Julio Frenk (2010) The Global Health System: Strengthening National Health Systems as the Next Step for Global Progress. PlosMedicine 7.
    1. English M, Schellenberg J, Todd J (2011) Assessing health system interventions: key points when considering the value of randomization. Bull World Health Organ 89: 907–912.
    1. Paina L, Peters DH (2012) Understanding pathways for scaling up health services through the lens of complex adaptive systems. Health Policy Plan 27: 365–373.
    1. Targreed A, Hsu J, de Savigny D, Lavis NJ, Rottingen JA, Bennet S, (2012) Evalauting health systems strengthening Interventions in low-income countries:Are we asking the right questions? Health Policy 27.
    1. Targreed A, Hsu J, de Savigny D (2012) Systems thinking for strengthening health systems in LMICS:need for a paradigm shift. Health Policy 27.
    1. Taghreed A, Hsu J, de Savigny D, Lavis NJ, Rottingen JA, Bennet S, (2012) Evalauting health systems strengthening Interventions in low-income countries:Are we asking the right questions? Health Policy 27.
    1. Yap C, Siu E, Baker GR, Brown AD (2005) A comparison of systemwide and hospital-specific performance measurement tools. Journal of Health Care Management 50: 4.
    1. Kunz H, Schaaf T (2011) General and specific formalization approach for a Balanced Scorecard: An expert system with application in health care. Expert Systems with Applications 38: 1947–1955.
    1. Kaplan RS, Norton DP (1992) The balanced scorecard–measures that drive performance. Harv Bus Rev 70: 71–79.
    1. Gauld R, Al-wahaibi S, Chisholm J, Crabbe R, Kwon B, et al. (2011) Scorecards for health system performance assessment: the New Zealand example. Health Policy 103: 200–208.
    1. Lupi S, Verzola A, Carandina G, Salani M, Antonioli P, et al. (2011) Multidimensional evaluation of performance with experimental application of balanced scorecard: a two year experience. Cost Eff Resour Alloc 9: 7.
    1. Inamdar SN, Kaplan RS, Jones ML, Menitoff R (2000) The Balanced Scorecard: a strategic management system for multi-sector collaboration and strategy implementation. Qual Manag Health Care 8: 21–39.
    1. Bisbe J, Barrubes J (2012) The Balanced Scorecard as a Management Tool for Assessing and Monitoring Strategy Implementation in Health Care Organizations. Rev Esp Cardiol S0300–8932(12)00383–1 [pii]/j.recesp.2012.05.014 [doi].
    1. Khan MM, Hotchkiss D, Dmytraczenko T, Zunaid Ahsan K (2012) Use of a Balanced Scorecard in strengthening health systems in developing countries: an analysis based on nationally representative Bangladesh Health Facility Survey. Int J Health Plann Manage 10.1002/hpm.2136 [doi].
    1. Inamdar N, Kaplan RS, Bower M (2002) Applying the balanced scorecard in healthcare provider organizations. J Healthc Manag 47: 179–195 discussion 195–176.
    1. Jeffs L, Merkley J, Richardson S, Eli J, McAllister M (2011) Using a nursing balanced scorecard approach to measure and optimize nursing performance. Nurs Leadersh (Tor Ont) 24: 47–58.
    1. WHO (2010) Monitoring the Building Blocks of healh systems: A handbook of Indiactors and their measurement stragegies.
    1. Edward A, Kumar B, Kakar F, Salehi AS, Burnham G, et al. (2011) Configuring balanced scorecards for measuring health system performance: evidence from 5 years’ evaluation in Afghanistan. PLoS Med 8: e1001066.
    1. El-Jardali F, Saleh S, Ataya N, Jamal D (2011) Design, implementation and scaling up of the balanced scorecard for hospitals in Lebanon: policy coherence and application lessons for low and middle income countries. Health Policy 103: 305–314.
    1. Bouland DL, Fink E, Fontanesi J (2011) Introduction of the Balanced Scorecard into an academic department of medicine: creating a road map to success. J Med Pract Manage 26: 331–335.
    1. Mutale W, Godfrey-Fausset P, Mwanamwenge MT, Kasese N, Chintu N, et al. (2013) Measuring health system strengthening: application of the balanced scorecard approach to rank the baseline performance of three rural districts in Zambia. PLoS One 8: e58650.
    1. Mutale W, Chintu N, Stringer JS, Chilengi R, Mwanamwenge MT, et al... (2013) Application of system thinking: Evaluation of the BHOMA health system strengthening intervention in Zambia:.
    1. Sherr K, Requejo J, Basinga P (2013) Implementation research to catalyze advances in health systems strengthening in sub-Saharan Africa: the African Health Initiative. BMC Health Services Research 13: S1.
    1. Stringer J, Chisembele-Taylor A, Chibwesha C, Chi H, Ayles H, et al. (2013) Protocol-driven primary care and community linkages to improve population health in rural Zambia: the Better Health Outcomes through Mentoring and Assessment (BHOMA) project. BMC Health Services Research 13: S7.
    1. Mutale W, Mwanamwenge TM, Chintu N, Stringer J, Balabanova D, et al... (2013) Application of system thinking concepts in health system strengthening in low income settings: A proposed conceptual framework for the evaluation of a complex health system intervention: The case of the BHOMA intervention in Zambia:. Lusaka: ZAMBART.
    1. Jeffrey M Wooldridge (2010) Econometric Analysis of Cross Section and Panel Data, Second Edition: MIT Press.
    1. Adam T, Hsu J, De Savigny D, Lavis JN, Rottingen JA, et al. (2012) Evaluating health systems strengthening interventions in low-income and middle-income countries: Are we asking the right questions? Health Policy and Planning 27: iv9–iv19.
    1. WHO (2009) Systems Thinking for Health Systems Strengthening.
    1. Atun R (2012) Health systems, systems thinking and innovation. Health Policy and Planning 27: iv4–iv8.
    1. Taghreed A, Hsu J, de Savigny D (2012) Systems thinking for strengthening health systems in LMICS:need for a paradigm shift. Health Policy 27.
    1. BeLue R, Carmack C, Myers KR, Weinreb-Welch L, Lengerich EJ (2012) Systems thinking tools as applied to community-based participatory research: a case study. Health Educ Behav 39: 745–751.
    1. Wasunna B, Zurovac D, Bruce J, Jones C, Webster J, et al. (2010) Health worker performance in the management of paediatric fevers following in-service training and exposure to job aids in Kenya. Malar J 9: 261.
    1. Byrne A, Morgan A, Soto EJ, Dettrick Z (2012) Context-specific, evidence-based planning for scale-up of family planning services to increase progress to MDG 5: health systems research. Reprod Health 9: 27.
    1. de Savigny D, Webster J, Agyepong IA, Mwita A, Bart-Plange C, et al. (2012) Introducing vouchers for malaria prevention in Ghana and Tanzania: context and adoption of innovation in health systems. Health Policy Plan 27 Suppl 4iv32–43.
    1. Grant A, Treweek S, Dreischulte T, Foy R, Guthrie B (2013) Process evaluations for cluster-randomised trials of complex interventions: a proposed framework for design and reporting. Trials 14: 15.
    1. English M (2013) Designing a theory-informed, contextually appropriate intervention strategy to improve delivery of paediatric services in Kenyan hospitals. Implementation Science 8.
    1. Horwitz O, Lysgaard-Hansen B (1975) Medical observations and bias. Am J Epidemiol 101: 391–399.
    1. Van Ryckeghem DM, Crombez G, Goubert L, De Houwer J, Onraedt T, et al. (2013) The predictive value of attentional bias towards pain-related information in chronic pain patients: a diary study. Pain 154: 468–475.

Source: PubMed

Sponsors and Collaborators

Medical Conditions

Drug Interventions

3
Subscribe