How to "Start Small and Just Keep Moving Forward": Mixed Methods Results From a Stepped-Wedge Trial to Support Evidence-Based Processes in Local Health Departments

Abstract

Background: Local health departments (LHDs) in the United States are charged with preventing disease and promoting health in their respective communities. Understanding and addressing what supports LHD's need to foster a climate and culture supportive of evidence-based decision making (EBDM) processes can enhance delivery of effective practices and services.

Methods: We employed a stepped-wedge trial design to test staggered delivery of implementation supports in 12 LHDs (Missouri, USA) to expand capacity for EBDM processes. The intervention was an in-person training in EBDM and continued support by the research team over 24 months (March 2018-February 2020). We used a mixed-methods approach to evaluate: (1) individuals' EBDM skills, (2) organizational supports for EBDM, and (3) administered evidence-based interventions. LHD staff completed a quantitative survey at 4 time points measuring their EBDM skills, organizational supports, and evidence-based interventions. We selected 4 LHDs with high contact and engagement during the intervention period to interview staff (n = 17) about facilitators and barriers to EBDM. We used mixed-effects linear regression to examine quantitative survey outcomes. Interviews were transcribed verbatim and coded through a dual independent process.

Results: Overall, 519 LHD staff were eligible and invited to complete quantitative surveys during control periods and 593 during intervention (365 unique individuals). A total of 434 completed during control and 492 during intervention (83.6 and 83.0% response, respectively). In both trial modes, half the participants had at least a master's degree (49.7-51.7%) and most were female (82.1-83.8%). No significant intervention effects were found in EBDM skills or in implementing evidence-based interventions. Two organizational supports scores decreased in intervention vs. control periods: awareness (-0.14, 95% CI -0.26 to -0.01, p < 0.05) and climate cultivation (-0.14, 95% CI -0.27 to -0.02, p < 0.05) but improved over time among all participants. Interviewees noted staff turnover, limited time, resources and momentum as challenges to continue EBDM work. Setting expectations, programmatic reviews, and pre-existing practices were seen as facilitators.

Conclusions: Challenges (e.g., turnover, resources) may disrupt LHDs' abilities to fully embed organizational processes which support EBDM. This study and related literature provides understanding on how best to support LHDs in building capacity to use and sustain evidence-based practices.

Keywords: evidence-based decision making; evidence-based decision making competency; evidence-based public health; local health department; organizational capacity.

Conflict of interest statement

SK was employed by RAND Corporation. DD was employed by National Association of County and City Health Officials. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2022 Jacob, Parks, Allen, Mazzucca, Yan, Kang, Dekker and Brownson.

Figures

Figure 1

Stepped-wedge design. (A) This stepped-wedge design featured 12 units (local health departments) randomly assigned into one of three groups. Shaded cells represent intervention periods. Clear cells represent control periods. Group 1's intervention period was 24 months, Group 2's intervention period was 16 months and Group 3's intervention period was 8 months. (B) Baseline measures for all units were taken during the pre-intervention period. Groups crossed over from control to receive intervention activities with measurements at 8-month intervals.

Figure 2

Participation flow diagram. This stepped-wedge design featured 12 units (local health departments) randomly assigned into one of three groups. Within each unit, individuals were invited to participate in a quantitative survey at four separate time points. Each time point included returning survey invitees and newly-invited individuals (open cohort design) where turnover warranted replacements with new hires.

Figure 3

Mean EBDM skill gaps by time and trial mode. At each time point, mean and 95% confidence intervals for skill gaps in evidence-based decision making (EBDM) are displayed for individuals during control and intervention phases. EBDM skill gaps come from importance and availability of 10 skills measured on an 11-point ordered scale. For each skill, a skill gap was calculated by subtracting the availability rating from the importance rating. Time 0 represents baseline where all units (local health departments) were in control period. Time 3 is the final data collection point and all individuals are in intervention period. Total mean skill gap score was created as an average of the 10 individual competency gaps.

Figure 4

Mean EBI score by time and trial mode. At each time point, mean and 95% confidence intervals for evidence-based interventions (EBI) score are displayed for individuals during control and intervention phases. For EBI Score, participants selected from a list of eight evidence-based programs or policies related to chronic disease prevention which were currently implemented at their respective local health department. We summed all 8 possible EBIs to calculate the EBI score which had a possible range of 0–8. Time 0 represents baseline where all units (local health departments) were in control period. Time 3 is the final data collection point and all individuals are in intervention period. Total mean skill gap score was created as an average of the 10 individual competency gaps.

Figure 5

Mean EBDM culture items by time and trial mode. At each time point, mean and 95% confidence intervals for organizational culture supportive of evidence-based decision making (EBDM) items are displayed for individuals during control and intervention phases. All items on organizational culture supportive of EBDM were measured on a 7-point Likert scale. A summary score was created as an average of the items within each domain. Time 0 represents baseline where all units (local health departments) were in control period. Time 3 is the final data collection point and all individuals are in intervention period. Total mean skills gap score was created as an average of the 10 individual competency gaps.