The ENCOMPASS framework: a practical guide for the evaluation of public health programmes in complex adaptive systems

Angie Luna Pinzon, Karien Stronks, Coosje Dijkstra, Carry Renders, Teatske Altenburg, Karen den Hertog, Stef P J Kremers, Mai J M Chinapaw, Arnoud P Verhoeff, Wilma Waterlander, Angie Luna Pinzon, Karien Stronks, Coosje Dijkstra, Carry Renders, Teatske Altenburg, Karen den Hertog, Stef P J Kremers, Mai J M Chinapaw, Arnoud P Verhoeff, Wilma Waterlander

Abstract

Background: Systems thinking embraces the complexity of public health problems, including childhood overweight and obesity. It aids in understanding how factors are interrelated, and it can be targeted to produce favourable changes in a system. There is a growing call for systems approaches in public health research, yet limited practical guidance is available on how to evaluate public health programmes within complex adaptive systems. The aim of this paper is to present an evaluation framework that supports researchers in designing systems evaluations in a comprehensive and practical way.

Methods: We searched the literature for existing public health systems evaluation studies. Key characteristics on how to conduct a systems evaluation were extracted and compared across studies. Next, we overlaid the identified characteristics to the context of the Lifestyle Innovations Based on Youth Knowledge and Experience (LIKE) programme evaluation and analyzed which characteristics were essential to carry out the LIKE evaluation. This resulted in the Evaluation of Programmes in Complex Adaptive Systems (ENCOMPASS) framework.

Results: The ENCOMPASS framework includes five iterative stages: (1) adopting a system dynamics perspective on the overall evaluation design; (2) defining the system boundaries; (3) understanding the pre-existing system to inform system changes; (4) monitoring dynamic programme output at different system levels; and (5) measuring programme outcome and impact in terms of system changes.

Conclusions: The value of ENCOMPASS lies in the integration of key characteristics from existing systems evaluation studies, as well as in its practical, applied focus. It can be employed in evaluating public health programmes in complex adaptive systems. Furthermore, ENCOMPASS provides guidance for the entire evaluation process, all the way from understanding the system to developing actions to change it and to measuring system changes. By the nature of systems thinking, the ENCOMPASS framework will likely evolve further over time, as the field expands with more completed studies.

Keywords: Complex systems; Evaluation; Overweight and obesity; Participatory action research; Practice; Public health; Systems thinking; Whole-of-systems approaches.

Conflict of interest statement

The authors declare that they have no competing interests.

© 2022. The Author(s).

Figures

Fig. 1
Fig. 1
Overview of the LIKE programme
Fig. 2
Fig. 2
Overview of the various iterative stages in the ENCOMPASS framework*

References

    1. McGlashan J, Hayward J, Brown A, Owen B, Millar L, Johnstone M, et al. Comparing complex perspectives on obesity drivers: action‐driven communities and evidence‐oriented experts. Obes Sci Pract. 2018;4(6):575–581. doi: 10.1002/osp4.306.
    1. Swinburn BA, Kraak VI, Allender S, Atkins VJ, Baker PI, Bogard JR, et al. The global syndemic of obesity, undernutrition, and climate change: the Lancet Commission report. Lancet. 2019;393(10173):791–846. doi: 10.1016/S0140-6736(18)32822-8.
    1. Carey G, Malbon E, Carey N, Joyce A, Crammond B, Carey A. Systems science and systems thinking for public health: a systematic review of the field. BMJ Open. 2015;5(12)
    1. Egan M, McGill E, Penney T, Anderson de Cuevas R, Er V, Orton L, et al. NIHR SPHR Guidance on Systems Approaches to Local Public Health Evaluation. Part 1: Introducing Systems Thinking. NIHR SPHR Guidance on Systems Approaches to Local Public Health Evaluation Part 1: Introducing systems thinking. 2019.
    1. McGill E, Er V, Penney T, Egan M, White M, Meier P, et al. Evaluation of public health interventions from a complex systems perspective: a research methods review. Soc Sci Med. 2021;113697. 10.1016/j.socscimed.2021.113697.
    1. Stansfield J, South J, Mapplethorpe T. What are the elements of a whole system approach to community-centred public health? A qualitative study with public health leaders in England’s local authority areas. BMJ Open. 2020;10(8):e036044. doi: 10.1136/bmjopen-2019-036044.
    1. Meadows D, Wright D. Thinking in Systems: A Primer. White River Junction: Chelsea Green Publishing; 2008.
    1. Patton MQ. Developmental evaluation: Applying complexity concepts to enhance innovation and use. New York: Guilford Press; 2011. pp. 375–xxi.
    1. Garcia LM, Hunter RF, de la Haye K, Economos CD, King AC. An action‐oriented framework for systems‐based solutions aimed at childhood obesity prevention in US Latinx and Latin American populations. Obes Rev. 2021:e13241. 10.1111/obr.13241.
    1. Ford DN. A system dynamics glossary. Syst Dyn Rev. 2019;35(4):369–379. doi: 10.1002/sdr.1641.
    1. Foster-Fishman PG, Behrens TR. Systems change reborn: rethinking our theories, methods, and efforts in human services reform and community-based change. Am J Commun Psychol. 2007;39(3-4):191–196. doi: 10.1007/s10464-007-9104-5.
    1. Walton M. Applying complexity theory: a review to inform evaluation design. Eval Program Plan. 2014;45:119–126. doi: 10.1016/j.evalprogplan.2014.04.002.
    1. Byrne D, Callaghan G. Complexity theory and the social sciences: The state of the art. London: Routledge; 2013.
    1. Resnicow K, Vaughan R. A chaotic view of behavior change: a quantum leap for health promotion. Int J Behav Nutr Phys Act. 2006;3(1):1–7. doi: 10.1186/1479-5868-3-25.
    1. Bagnall A-M, Radley D, Jones R, Gately P, Nobles J, Van Dijk M, et al. Whole systems approaches to obesity and other complex public health challenges: a systematic review. BMC Public Health. 2019;19(1):1–14. doi: 10.1186/s12889-018-6274-z.
    1. Eoyang G, Oakden J. Adaptive evaluation: a synergy between complexity theory and evaluation practice. Emerg Complex Organ. 2016;18(3/4):1–14.
    1. Egan M, Penney T, Anderson de Cuevas R, Er V, Orton L, White M, et al. NIHR SPHR Guidance on Systems Approaches to Local Public Health Evaluation. Part 2: What to consider when planning a systems evaluation. 2019.
    1. Skivington K, Matthews L, Simpson SA, Craig P, Baird J, Blazeby JM, et al. A new framework for developing and evaluating complex interventions: update of Medical Research Council guidance. Bmj. 2021;374:n2061. doi: 10.1136/bmj.n2061.
    1. Hargreaves MB. Evaluating system change: A planning guide. 2010.
    1. Hennessy E, Economos CD, Hammond RA. Team SUSSM, the CT. Integrating Complex Systems Methods to Advance Obesity Prevention Intervention Research. Health Educ Behav. 2020. 10.1177/1090198119898649.
    1. Waterlander WE, Luna Pinzon A, Verhoeff A, Den Hertog K, Altenburg T, Dijkstra C, et al. A System Dynamics and Participatory Action Research Approach to Promote Healthy Living and a Healthy Weight among 10–14-Year-Old Adolescents in Amsterdam: The LIKE Programme. Int J Environ Res Public Health. 2020;17(14):4928. doi: 10.3390/ijerph17144928.
    1. Sawyer A, den Hertog K, Verhoeff AP, Busch V, Stronks K. Developing the logic framework underpinning a whole‐systems approach to childhood overweight and obesity prevention: Amsterdam Healthy Weight Approach. Obes Sci Pract. 2021. 10.1002/osp4.505.
    1. Karacabeyli D, Allender S, Pinkney S, Amed S. Evaluation of complex community-based childhood obesity prevention interventions. Obes Rev. 2018;19(8):1080–1092. doi: 10.1111/obr.12689.
    1. Best A. Greater than the sum: systems thinking in tobacco control: National Cancer Institute, US Department of Health and Human Services. 2007.
    1. Moore GF, Evans RE, Hawkins J, Littlecott H, Melendez-Torres G, Bonell C, et al. From complex social interventions to interventions in complex social systems: future directions and unresolved questions for intervention development and evaluation. Evaluation. 2019;25(1):23–45. doi: 10.1177/1356389018803219.
    1. Hawe P, Shiell A, Riley T. Theorising interventions as events in systems. Am J Commun Psychol. 2009;43(3-4):267–276. doi: 10.1007/s10464-009-9229-9.
    1. Matheson A, Walton M, Gray R, Lindberg K, Shanthakumar M, Fyfe C, et al. Evaluating a community-based public health intervention using a complex systems approach. J Public Health. 2018;40(3):606–613. doi: 10.1093/pubmed/fdx117.
    1. Foster-Fishman PG, Nowell B, Yang H. Putting the system back into systems change: A framework for understanding and changing organizational and community systems. Am J Commun Psychol. 2007;39(3-4):197–215. doi: 10.1007/s10464-007-9109-0.
    1. Stern E, Stame N, Mayne J, Forss K, Davies R, Befani B. Broadening the range of designs and methods for impact evaluations. 2012.
    1. Jones M, Verity F, Warin M, Ratcliffe J, Cobiac L, Swinburn B, et al. OPALesence: Epistemological pluralism in the evaluation of a systems-wide childhood obesity prevention program. Evaluation. 2016;22(1):29–48. doi: 10.1177/1356389015623142.
    1. MacMynowski DP. Pausing at the brink of interdisciplinarity: power and knowledge at the meeting of social and biophysical science. Ecol Soc. 2007;12(1)
    1. Miller TR, Baird TD, Littlefield CM, Kofinas G, Chapin FS III, Redman CL. Epistemological pluralism: reorganizing interdisciplinary research. Ecol Soc. 2008;13(2)
    1. Midgley G. Systems thinking for evaluation. Systems concepts in evaluation: An expert anthology. 2006. pp. 11–34.
    1. Williams B, Imam I. Systems concepts in evaluation: An expert anthology: EdgePress of Inverness Point Reyes, CA. 2007.
    1. Forrest J. Evolutionary and behavioral characteristics of systems. Systems Concepts in Evaluation. 2006.
    1. Waterlander WE, Singh A, Altenburg T, Dijkstra C, Luna Pinzon A, Anselma M, et al. Understanding obesity-related behaviors in youth from a systems dynamics perspective: the use of causal loop diagrams. Obes Rev. 2021;22(7):e13185.
    1. Maitland N, Wardle K, Whelan J, Jalaludin B, Creighton D, Johnstone M, et al. Tracking implementation within a community-led whole of system approach to address childhood overweight and obesity in south west Sydney, Australia. BMC Public Health. 2021;21(1):1-11. doi: 10.1186/s12889-021-11288-5.
    1. McGlashan J, Nichols M, Korn A, Millar L, Marks J, Sanigorski A, et al. Social network analysis of stakeholder networks from two community-based obesity prevention interventions. PLoS One. 2018;13(4):e0196211. doi: 10.1371/journal.pone.0196211.
    1. Johnston LM, Matteson CL, Finegood DT. Systems science and obesity policy: a novel framework for analyzing and rethinking population-level planning. Am J Public Health. 2014;104(7):1270–1278. doi: 10.2105/AJPH.2014.301884.
    1. Nobles JD, Radley D, Mytton OT, team WSOp. The Action Scales Model: A conceptual tool to identify key points for action within complex adaptive systems. Perspectives. Public Health. 2021. 10.1177/17579139211006747.
    1. Malhi L, Karanfil Ö, Merth T, Acheson M, Palmer A, Finegood DT. Places to intervene to make complex food systems more healthy, green, fair, and affordable. J Hunger Environ Nutr. 2009;4(3-4):466–476. doi: 10.1080/19320240903346448.
    1. Brown AD, Whelan J, Bolton KA, Nagorcka-Smith P, Hayward J, Fraser P, et al. A Theory of Change for Community-Based Systems Interventions to Prevent Obesity. Am J Prev Med. 2021;
    1. McGlashan J, Johnstone M, Creighton D, de la Haye K, Allender S. Quantifying a systems map: network analysis of a childhood obesity causal loop diagram. PloS One. 2016;11(10):e0165459. doi: 10.1371/journal.pone.0165459.
    1. Ling T. Evaluating complex and unfolding interventions in real time. Evaluation. 2012;18(1):79–91. doi: 10.1177/1356389011429629.
    1. Kreger M, Brindis CD, Manuel DM, Sassoubre L. Lessons learned in systems change initiatives: benchmarks and indicators. Am J Commun Psychol. 2007;39(3-4):301–320. doi: 10.1007/s10464-007-9108-1.
    1. Van der Merwe SE, Biggs R, Preiser R, Cunningham C, Snowden DJ, O’Brien K, et al. Making sense of complexity: Using sensemaker as a research tool. Systems. 2019;7(2):25. doi: 10.3390/systems7020025.
    1. Fowler B, Dunn E. Evaluating Systems and Systemic Change for Inclusive Market Development. Leveraging Economic Opportunities. Report. 2014;3
    1. Nobles J, Wheeler J, Dunleavy-Harris K, Holmes R, Inman-Ward A, Potts A, et al. Ripple effects mapping: capturing the wider impacts of systems change efforts in public health. 2021.

Source: PubMed

Szponzorok és közreműködők

Egészségi állapot

Kábítószer-beavatkozások

3
Iratkozz fel