Using path analysis to test theory of change: a quantitative process evaluation of the MapSan trial

Sarah Bick, Helen Buxton, Rachel P Chase, Ian Ross, Zaida Adriano, Drew Capone, Jackie Knee, Joe Brown, Rassul Nalá, Oliver Cumming, Robert Dreibelbis, Sarah Bick, Helen Buxton, Rachel P Chase, Ian Ross, Zaida Adriano, Drew Capone, Jackie Knee, Joe Brown, Rassul Nalá, Oliver Cumming, Robert Dreibelbis

Abstract

Background: Although theory-driven evaluations should have empirical components, few evaluations of public health interventions quantitatively test the causal model made explicit in the theory of change (ToC). In the context of a shared sanitation trial (MapSan) in Maputo, Mozambique, we report findings of a quantitative process evaluation assessing intervention implementation, participant response and impacts on hypothesised intermediary outcomes on the pathway to trial health outcomes. We examine the utility of path analysis in testing intervention theory using process indicators from the intervention's ToC.

Methods: Process data were collected through a cross-sectional survey of intervention and control compounds of the MapSan trial > 24-months post-intervention, sampling adult residents and compound leaders. Indicators of implementation fidelity (dose received, reach) and participant response (participant behaviours, intermediary outcomes) were compared between trial arms. The intervention's ToC (formalised post-intervention) was converted to an initial structural model with multiple alternative pathways. Path analysis was conducted through linear structural equation modelling (SEM) and generalised SEM (probit model), using a model trimming process and grouped analysis to identify parsimonious models that explained variation in outcomes, incorporating demographics of respondents and compounds.

Results: Among study compounds, the MapSan intervention was implemented with high fidelity, with a strong participant response in intervention compounds: improvements were made to intermediary outcomes related to sanitation 'quality' - latrine cleanliness, maintenance and privacy - but not to handwashing (presence of soap / soap residue). These outcomes varied by intervention type: single-cabin latrines or multiple-cabin blocks (designed for > 20 users). Path analysis suggested that changes in intermediary outcomes were likely driven by direct effects of intervention facilities, with little contribution from hygiene promotion activities nor core elements expected to mediate change: a compound sanitation committee and maintenance fund. A distinct structural model for two compound size subgroups (≤ 20 members vs. > 20 members) explained differences by intervention type, and other contextual factors influenced specific model parameters.

Conclusions: While process evaluation found that the MapSan intervention achieved sufficient fidelity and participant response, the path analysis approach applied to test the ToC added to understanding of possible 'mechanisms of change', and has value in disentangling complex intervention pathways.

Trial registration: MapSan trial registration: NCT02362932 Feb-13-2015.

Keywords: Complex intervention; Mozambique; Path analysis; Process evaluation; Sanitation; Structural equation modelling; Theory of change.

Conflict of interest statement

The authors declare that they have no competing interests.

© 2021. The Author(s).

Figures

Fig. 1
Fig. 1
Theory of Change for the MapSan intervention in Maputo, Mozambique. Abbreviations: CBO, community-based organisation; HWF, handwashing facility; HWWS, handwashing with soap; MHM, menstrual hygiene management; MISAU, Ministério da Saúde (Ministry of Health) Mozambique; STH, soil-transmitted helminths
Fig. 2
Fig. 2
Initial structural model constructed from the MapSan intervention theory of change. This model formed the basis of path analysis of cross-sectional data from MapSan trial compounds. Sections of the model corresponding to process evaluation domains (implementation fidelity, participant response, context) and subdomains (dose received, participant behaviours, intermediary outcomes) indicated by dashed lines. Abbreviations: ‘chefe’, chefe de composto (informal compound leader); CSB, communal sanitation block; HWF, handwashing facility; HWWS, handwashing with soap
Fig. 3
Fig. 3
Prevalence of implementation fidelity and participant response indicators between intervention and control compounds of the MapSan trial. Associated 95% confidence intervals and significance level of the chi-squared test comparing prevalence between trial arms indicated. Abbreviations: Con, control; HWF, handwashing facility; HWWS, handwashing with soap; Int, intervention
Fig. 4
Fig. 4
Reported topics discussed at household-level behaviour change visits to intervention compounds of the MapSan trial. Prevalence, 95% confidence interval, and the intraclass correlation coefficient (ICC (1,k)) associated with each response are displayed. Abbreviations: HWWS, handwashing with soap; ICC, intraclass correlation coefficient
Fig. 5
Fig. 5
Linear probability path analysis of data from small compounds (≤ 20 members) of the MapSan trial. Path model for small compounds (≤ 20 members) subgroup assessing effects of a sanitation intervention on intermediary outcomes. Unstandardised path coefficients (bold type) represent the increase in absolute probability of the outcome as a result of a unit change in the exposure. Sections of the model corresponding to process evaluation domains and subdomains indicated by dashed lines. The model structure was less complex than that of the large compounds subgroup. Large direct effects on latrine maintenance (condition of slab/floor), privacy (working lock) and cleanliness were observed. Indirect effects on latrine cleanliness via cleaning behaviours were minimal. Intervention latrines were less likely to be accessible to all members of the compound. Abbreviations: HWF, handwashing facility; HWWS, handwashing with soap
Fig. 6
Fig. 6
Linear probability path analysis of data from large compounds (> 20 members) of the MapSan trial. Path model for large compounds (> 20 members) subgroup assessing effects of a sanitation intervention on intermediary outcomes. Unstandardised path coefficients (bold type) represent the increase in absolute probability of the outcome as a result of a unit change in the exposure. Sections of the model corresponding to process evaluation domains and subdomains indicated by dashed lines. The model structure was more complex than that of the small compounds subgroup. Large direct effects on latrine maintenance (condition of slab/floor), privacy (working lock) and cleanliness were observed. Indirect effects on latrine cleanliness via cleaning behaviours (supported by a compound cleaning rota system) and indirect effects on latrine privacy were both minimal. A small negative effect on handwashing (soap residue) was observed. Abbreviations: ‘chefe’, chefe de composto (informal compound leader); HWF, handwashing facility; HWWS, handwashing with soap

References

    1. Oakley A, Strange V, Bonell C, Allen E, Stephenson J, Team RS. Process evaluation in randomised controlled trials of complex interventions. BMJ. 2006;332(7538):413–416. doi: 10.1136/bmj.332.7538.413.
    1. Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, Moore L, O'Cathain A, Tinati T, Wight D, Baird J. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015;350:h1258. doi: 10.1136/bmj.h1258.
    1. Coryn CLS, Noakes LA, Westine CD, Schröter DC. A systematic review of theory-driven evaluation practice from 1990 to 2009. Am J Eval. 2010;32(2):199–226. 10.1177/1098214010389321.
    1. Astbury B, Leeuw FL. Unpacking black boxes: mechanisms and theory building in evaluation. Am J Eval. 2010;31(3):363–381. doi: 10.1177/1098214010371972.
    1. Bonell C, Fletcher A, Morton M, Lorenc T, Moore L. Realist randomised controlled trials: a new approach to evaluating complex public health interventions. Soc Sci Med. 2012;75(12):2299–2306. doi: 10.1016/j.socscimed.2012.08.032.
    1. Pawson R, Tilley N. Realistic evaluation. London: Sage; 1997.
    1. Weiss C. Nothing as practical as a good theory: exploring theory-based evaluation in complex community initiatives for children and families. In: Connell J, Kubisch A, Schorr L, C W, editors. New approaches to evaluating community initiatives: volume 1, concepts, methods, and contexts. Washington, DC: The Aspen Institute; 1995.
    1. De Silva MJ, Breuer E, Lee LA, Asher L, Chowdhary N, Lund C, et al. Theory of change: a theory-driven approach to enhance the Medicial research Council's framework for complex interventions. Trials. 2014;15(1):267. doi: 10.1186/1745-6215-15-267.
    1. Breuer E, Lee L, De Silva M, Lund C. Using theory of change to design and evaluate public health interventions: a systematic review. Implement Sci. 2016;11(1):63. 10.1186/s13012-016-0422-6.
    1. Chen H-T. Practical program evaluation: assessing and improving planning, implementation, and effectiveness. Thousand Oaks: Sage; 2005.
    1. Fletcher A, Jamal F, Moore G, Evans RE, Murphy S, Bonell C. Realist complex intervention science: applying realist principles across all phases of the Medical Research Council framework for developing and evaluating complex interventions. Evaluation (Lond) 2016;22(3):286–303. doi: 10.1177/1356389016652743.
    1. Lipsey MW. Theory as method: small theories of treatments. New Direct Prog Eval. 1993;1993(57):5–38. doi: 10.1002/ev.1637.
    1. National Institute for Health and Care Excellence . Behaviour change: general approaches. London: NICE; 2007.
    1. Smith NL. Using path analysis to develop and evaluate program theory and impact. New Direct Prog Eval. 1990;1990(47):53–57. doi: 10.1002/ev.1554.
    1. Pedhazur EJ. Multiple Regression in Behavioral Research: Explanation and Prediction. 3rd ed. Fort Worth: Harcourt Brace College Publishers; 1997.
    1. Wright S. The method of path coefficients. Ann Math Stat. 1934;5(3):161–215. doi: 10.1214/aoms/1177732676.
    1. Hennessy M, Greenberg J. Bringing it all together: modeling intervention processes using structural equation modeling. Am J Eval. 1999;20(3):471–480. doi: 10.1177/109821409902000306.
    1. Pruss-Ustun A, Wolf J, Bartram J, Clasen T, Cumming O, Freeman MC, et al. Burden of disease from inadequate water, sanitation and hygiene for selected adverse health outcomes: an updated analysis with a focus on low- and middle-income countries. Int J Hyg Environ Health. 2019;222(5):765–777. doi: 10.1016/j.ijheh.2019.05.004.
    1. UNICEF, World Health Organization. Progress on household drinking water, sanitation and hygiene 2000–2017: special focus on inequalities. New York: United Nations Children’s Fund (UNICEF) and World Health Organization; 2019.
    1. UN-HABITAT. The challenge of slums: global report on human settlements 2003. London: Earthscan Publications; 2003.
    1. UN-HABITAT. Mozambique Cities Profile: Maputo, Nacala and Manica. Nairobi: United Nations Human Settlements Programme (UN-HABITAT); 2010. Report No. 9789211322675.
    1. Peal A, Evans B, Blackett I, Hawkins P, Heymans C. Fecal sludge management: a comparative analysis of 12 cities. J Water Sanit Hyg Dev. 2014;4(4):563–75. 10.2166/washdev.2014.026.
    1. Simiyu S, Swilling M, Cairncross S, Rheingans R. Determinants of quality of shared sanitation facilities in informal settlements: case study of Kisumu, Kenya. BMC Public Health. 2017;17(1):68. doi: 10.1186/s12889-016-4009-6.
    1. Kwiringira J, Atekyereza P, Niwagaba C, Gunther I. Descending the sanitation ladder in urban Uganda: evidence from Kampala slums. BMC Public Health. 2014;14(1):624. doi: 10.1186/1471-2458-14-624.
    1. Tumwebaze IK, Mosler HJ. Shared toilet users' collective cleaning and determinant factors in Kampala slums, Uganda. BMC Public Health. 2014;14(1):1260. doi: 10.1186/1471-2458-14-1260.
    1. Brown J, Cumming O, Bartram J, Cairncross S, Ensink J, Holcomb D, Knee J, Kolsky P, Liang K, Liang S, Nala R, Norman G, Rheingans R, Stewart J, Zavale O, Zuin V, Schmidt WP. A controlled, before-and-after trial of an urban sanitation intervention to reduce enteric infections in children: research protocol for the Maputo sanitation (MapSan) study, Mozambique. BMJ Open. 2015;5(6):e008215. doi: 10.1136/bmjopen-2015-008215.
    1. Water & Sanitation for the Urban Poor (WSUP). An integrated approach to peri-urban sanitation and hygiene in Maputo. Working with city authorities to improve services and practices. Policy Brief. Maputo: WSUP; 2018. .
    1. Knee J, Sumner T, Adriano Z, Anderson C, Bush F, Capone D, et al. Effects of an urban sanitation intervention on childhood enteric infection and diarrhea in Maputo, Mozambique: a controlled before-and-after trial. eLife. 2021;10. 10.7554/eLife.62278.
    1. Knee J, Sumner T, Adriano Z, Berendes D, de Bruijn E, Schmidt WP, Nalá R, Cumming O, Brown J. Risk factors for childhood enteric infection in urban Maputo, Mozambique: a cross-sectional study. Plos Neglect Trop D. 2018;12(11):e0006956. doi: 10.1371/journal.pntd.0006956.
    1. Mattson K. Final evaluation of the Water & Sanitation for the urban poor JSDF funded Maputo Peri-urban sanitation project. 2016.
    1. Shiras T, Cumming O, Brown J, Muneme B, Nala R, Dreibelbis R. Shared Sanitation Management and the Role of Social Capital: Findings from an Urban Sanitation Intervention in Maputo, Mozambique. Int J Environ Res Public Health. 2018;15(10):2222. 10.3390/ijerph15102222.
    1. Carroll C, Patterson M, Wood S, Booth A, Rick J, Balain S. A conceptual framework for implementation fidelity. Implement Sci. 2007;2(1):40. doi: 10.1186/1748-5908-2-40.
    1. Linnan L, Steckler A. Process evaluation for public health interventions and research. San Francisco: Jossey-Boss: A Wiley Imprint; 2002.
    1. Capone D, Buxton H, Cumming O, Dreibelbis R, Knee J, Nala R, et al. Impact of an intervention to improve pit latrine emptying practices in low income urban neighborhoods of Maputo, Mozambique. Int J Hyg Environ Health. 2020;226:113480. doi: 10.1016/j.ijheh.2020.113480.
    1. Cicchetti DV. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol Assess. 1994;6(4):284–290. doi: 10.1037/1040-3590.6.4.284.
    1. Schreiner M. Simple poverty scorecard-assessment tool Mozambique. 2013.
    1. Rheingans R, Anderson JD, Luyendijk R, Cumming O. Measuring disparities in sanitation access: does the measure matter? Trop Med Int Health. 2014;19(1):2–13. 10.1111/tmi.12220.
    1. Evans B, Hueso A, Johnston R, Norman G, Pérez E, Slaymaker T, Trémolet S. Limited services? The role of shared sanitation in the 2030 agenda for sustainable development. J Water Sanit Hyg Dev. 2017;7(3):349–351. doi: 10.2166/washdev.2017.023.
    1. Sinharoy SS, Pittluck R, Clasen T. Review of drivers and barriers of water and sanitation policies for urban informal settlements in low-income and middle-income countries. Util Policy. 2019;60:100957. doi: 10.1016/j.jup.2019.100957.
    1. Tidwell JB, Chipungu J, Bosomprah S, Aunger R, Curtis V, Chilengi R. Effect of a behaviour change intervention on the quality of peri-urban sanitation in Lusaka, Zambia: a randomised controlled trial. Lancet Planet Health. 2019;3(4):e187–ee96. doi: 10.1016/S2542-5196(19)30036-1.
    1. Tumwebaze IK, Mosler HJ. Effectiveness of group discussions and commitment in improving cleaning behaviour of shared sanitation users in Kampala, Uganda slums. Soc Sci Med. 2015;147:72–79. doi: 10.1016/j.socscimed.2015.10.059.
    1. Alam MU, Winch PJ, Saxton RE, Nizame FA, Yeasmin F, Norman G, et al. Behaviour change intervention to improve shared toilet maintenance and cleanliness in urban slums of Dhaka: a cluster-randomised controlled trial. Trop Med Int Health. 2017;22(8):1000–11. 10.1111/tmi.12902.
    1. Isunju JB, Schwartz K, Schouten MA, Johnson WP, van Dijk MP. Socio-economic aspects of improved sanitation in slums: a review. Public Health. 2011;125(6):368–376. doi: 10.1016/j.puhe.2011.03.008.
    1. Tumwebaze IK. Prevalence and determinants of the cleanliness of shared toilets in Kampala slums, Uganda. J Public Health. 2013;22(1):33–9. 10.1007/s10389-013-0590-7.
    1. Hanssen CE. Structural equation modeling as a tool for multisite evaluation. 2004.
    1. Bagozzi RP, Yi Y. Specification, evaluation, and interpretation of structural equation models. J Acad Market Sci. 2012;40(1):8–34. doi: 10.1007/s11747-011-0278-x.
    1. MacCallum RC, Austin JT. Applications of structural equation modeling in psychological research. Annu Rev Psychol. 2000;51(1):201–226. doi: 10.1146/annurev.psych.51.1.201.
    1. Bollen KA, Noble MD. Structural equation models and the quantification of behavior. Proc Natl Acad Sci U S A. 2011;108(Suppl 3):15639–46. 10.1073/pnas.1010661108.
    1. VanderWeele TJ. Mediation analysis: a Practitioner's guide. Annu Rev Public Health. 2016;37(1):17–32. doi: 10.1146/annurev-publhealth-032315-021402.
    1. Aunger R, Curtis V. Behaviour Centred design: towards an applied science of behaviour change. Health Psychol Rev. 2016;10(4):425–446. doi: 10.1080/17437199.2016.1219673.
    1. Petraitis PS, Dunham AE, Niewiarowski PH. Inferring multiple causality: the limitations of path analysis. Funct Ecol. 1996;10(4):421–431. doi: 10.2307/2389934.
    1. Bonell C, Jamal F, Melendez-Torres GJ, Cummins S. 'Dark logic': theorising the harmful consequences of public health interventions. J Epidemiol Community Health. 2015;69(1):95–98. doi: 10.1136/jech-2014-204671.

Source: PubMed

Sponsors en medewerkers

Medische omstandigheden

Geneesmiddelinterventies

3
Abonneren