Fidelity is not easy! Challenges and guidelines for assessing fidelity in complex interventions

Liane R Ginsburg, Matthias Hoben, Adam Easterbrook, Ruth A Anderson, Carole A Estabrooks, Peter G Norton, Liane R Ginsburg, Matthias Hoben, Adam Easterbrook, Ruth A Anderson, Carole A Estabrooks, Peter G Norton

Abstract

Background: Fidelity in complex behavioural interventions is underexplored and few comprehensive or detailed fidelity studies report on specific procedures for monitoring fidelity. Using Bellg's popular Treatment Fidelity model, this paper aims to increase understanding of how to practically and comprehensively assess fidelity in complex, group-level, interventions.

Approach and lessons learned: Drawing on our experience using a mixed methods approach to assess fidelity in the INFORM study (Improving Nursing home care through Feedback On perfoRMance data-INFORM), we report on challenges and adaptations experienced with our fidelity assessment approach and lessons learned. Six fidelity assessment challenges were identified: (1) the need to develop succinct tools to measure fidelity given tools tend to be intervention specific, (2) determining which components of fidelity (delivery, receipt, enactment) to emphasize, (3) unit of analysis considerations in group-level interventions, (4) missing data problems, (5) how to respond to and treat fidelity 'failures' and 'deviations' and lack of an overall fidelity assessment scheme, and (6) ensuring fidelity assessment doesn't threaten internal validity.

Recommendations and conclusions: Six guidelines, primarily applicable to group-level studies of complex interventions, are described to help address conceptual, methodological, and practical challenges with fidelity assessment in pragmatic trials. The current study offers guidance to researchers regarding key practical, methodological, and conceptual challenges associated with assessing fidelity in pragmatic trials. Greater attention to fidelity assessment and publication of fidelity results through detailed studies such as this one is critical for improving the quality of fidelity studies and, ultimately, the utility of published trials.

Trial registration: ClinicalTrials.gov NCT02695836. Registered on February 24, 2016.

Keywords: Complex interventions; Fidelity; Fidelity assessment; Fidelity methodology; Pragmatic group-level trials.

Conflict of interest statement

All authors declare no competing interests.

References

    1. Dusenbury L, Brannigan R, Falco F, Hansen WB. A review of research on fidelity of implementation: implications for drug abuse prevention in school settings. Health Educ Res. 2003;18(May):237-256. doi:org/10.1093/her/18.2.237
    1. Walton H, Spector A, Tombor I, Michie S. Measures of fidelity of delivery of, and engagement with, complex, face-to-face health behaviour change interventions: a systematic review of measure quality. Br J Health Psychol. 2017;22(4):1–19. doi: 10.1111/bjhp.12260.
    1. Bellg AJ, Resnick B, Minicucci DS, et al. Enhancing treatment fidelity in health behavior change studies: best practices and recommendations from the NIH Behavior Change Consortium. Heal Psychol. 2004;23(5):443–451. doi: 10.1037/0278-6133.23.5.443.
    1. Mowbray CT, Holter MC, Gregory B, Bybee D. Fidelity criteria: development , measurement , and validation. Am J Eval. 2003;24(3):315–340. doi: 10.1177/109821400302400303.
    1. Walton H, Spector A, Williamson M, Tombor I, Michie S. Developing quality fidelity and engagement measures for complex health interventions. Br J Health Psychol. 2020;25(1):39–60. doi: 10.1111/bjhp.12394.
    1. Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance. Br Med J. 2015;350(mar19 6):h1258. 10.1136/bmj.h1258.
    1. Carroll C, Patterson M, Wood S, Booth A, Rick J, Balain S. A conceptual framework for implementation fidelity. Implement Sci. 2007;2(1):40. doi: 10.1186/1748-5908-2-40.
    1. Lichstein KL, Riedel BW, Grieve R. Fair tests of clinical trials: a treatment implementation model. Adv Behav Res Ther. 1994;16(1):1–29. doi: 10.1016/0146-6402(94)90001-9.
    1. Century J, Rudnick M, Freeman C. A framework for measuring fidelity of implementation: a foundation for shared language and accumulation of knowledge. Am J Eval. 2010;31(2):199–218. doi: 10.1177/1098214010366173.
    1. Keith RE, Hopp FP, Subramanian U, Wiitala W, Lowery JC. Fidelity of implementation: development and testing of a measure. Implement Sci. 2010;5(1):99. doi: 10.1186/1748-5908-5-99.
    1. Sprange K, Beresford-Dent J, Mountain G, et al. Assessing fidelity of a community based psychosocial intervention for people with mild dementia within a large randomised controlled trial. BMC Geriatr. 2021;21(1). 10.1186/s12877-021-02070-8.
    1. Carpenter JS, Burns DS, Wu J, Yu M, Ryker K, Tallman E, et al. Strategies used and data obtained during treatment fidelity monitoring. Nurs Res. 2013;62(1):59–65. 10.1097/NNR.0b013e31827614fd.
    1. Hasson H. Systematic evaluation of implementation fidelity of complex interventions in health and social care. Implement Sci. 2010;5(67):67. doi: 10.1186/1748-5908-5-67.
    1. Toomey E, Hardeman W, Hankonen N, Byrne M, McSharry J, Matvienko-Sikar K, et al. Focusing on fidelity: narrative review and recommendations for improving intervention fidelity within trials of health behaviour change interventions. Heal Psychol Behav Med. 2020;8(1):132–51. 10.1080/21642850.2020.1738935.
    1. Breitenstein SM, Gross D, Garvey C, Hill C, Fogg L, Resnick B. Implementation fidelity in community-based interventions. Res Nurs Health. 2010;33(2):164–173. doi: 10.1002/nur.20373.Implementation.
    1. Toomey E, Matthews J, Guerin S, Hurley DA. Development of a feasible implementation fidelity protocol within a complex physical therapy-led self-management intervention.; 2016. .
    1. Breitenstein SM, Fogg L, Garvey C, Hill C, Resnick B, Gross D. Measuring implementation fidelity in a community-based parenting intervention. 2010;59(3):158–65. 10.1097/NNR.0b013e3181dbb2e2.Measuring.
    1. Goldberg J, Bumgarner E, Jacobs F. Measuring program- and individual-level fidelity in a home visiting program for adolescent parents. Eval Program Plann. 2016;55:163–173. doi: 10.1016/j.evalprogplan.2015.12.007.
    1. Feely M, Seay KD, Lanier P, Auslander W, Kohl PL. Measuring fidelity in research studies: a field guide to developing a comprehensive fidelity measurement system. Child Adolesc Soc Work J. 2018;35(2):139–152. doi: 10.1007/s10560-017-0512-6.
    1. Butel J, Braun KL, Novotny R, Acosta M, Castro R, Fleming T, et al. Assessing intervention fidelity in a multi-level, multi-component, multi-site program: the Children’s Healthy Living (CHL) program. Transl Behav Med. 2015;5(4):460–9. 10.1007/s13142-015-0334-z.
    1. Hoben M, Norton PG, Ginsburg LR, Anderson RA, Cummings GG, Lanham HJ, et al. Improving Nursing Home Care through Feedback On PerfoRMance Data (INFORM): protocol for a cluster-randomized trial. Trials. 2017;18(1):9. 10.1186/s13063-016-1748-8.
    1. Hoben M, Ginsburg LR, Easterbrook A, et al. Comparing effects of two higher intensity feedback interventions with simple feedback on improving staff communication in nursing homes - the INFORM cluster-randomized controlled trial. Implement Sci. 2020;15(1). 10.1186/s13012-020-01038-3.
    1. Ginsburg LR, Hoben M, Easterbrook A, Andersen E, Anderson RA, Cranley L, et al. Examining fidelity in the INFORM trial: a complex team-based behavioral intervention. Implement Sci. 2020;15(1):1–11. 10.1186/s13012-020-01039-2.
    1. Berta W, Laporte A, Deber R, Baumann A, Gamble B. The evolving role of health care aides in the long-term care and home and community care sectors in Canada. Hum Resour Health. 2013;11(1):25. doi: 10.1186/1478-4491-11-25.
    1. PHI. U.S. Nursing Assistants Employed on Nursing Homes: Key Facts. Bronx, NY: PHI; 2018; 2018. .
    1. Chamberlain SA, Hoben M, Squires JE, Cummings GG, Norton P, Estabrooks CA. Who is (still) looking after mom and dad? Few improvements in care aides’ quality-of-work life. Can J Aging. 2019;38(1):35–50. doi: 10.1017/S0714980818000338.
    1. Estabrooks CA, Squires JE, Hayduk L, et al. The influence of organizational context on best practice use by care aides in residential long-term care settings. J Am Med Dir Assoc. 2015;16(6):537.e1–537.e10. doi: 10.1016/j.jamda.2015.03.009.
    1. Estabrooks CA, Hoben M, Poss JW, Chamberlain SA, Thompson GN, Silvius JL, et al. Dying in a nursing home: treatable symptom burden and its link to modifiable features of work context. J Am Med Dir Assoc. 2015;16(6):515–20. 10.1016/j.jamda.2015.02.007.
    1. Parsons SK, Simmons WP, Penn K, Furlough M. Determinants of satisfaction and turnover among nursing assistants. The results of a statewide survey. J Gerontol Nurs. 2003;29(3):51–58. doi: 10.3928/0098-9134-20030301-11.
    1. Walton AL, Rogers B. Workplace hazards faced by nursing assistants in the united states: a focused literature review. Int J Environ Res Public Health. 2017;14(5). 10.3390/ijerph14050544.
    1. Ginsburg L, Easterbrook A, Berta W, Norton P, Doupe M, Knopp-Sihota J, et al. Implementing Frontline-worker–led quality improvement in nursing homes: getting to “how”. Jt Comm J Qual Patient Saf. 2018;44(9):526–35. 10.1016/j.jcjq.2018.04.009.
    1. Kessler I, Heron P, Dopson S. The nature and consequences of support workers in a hospital setting. Final report NIHR Serv Deliv Organ Program. 2010;(June):229. .
    1. Bailey S, Scales K, Lloyd J, Schneider J, Jones R. The emotional labour of health-care assistants in inpatient dementia care. Ageing Soc. 2015;35(2):246–269. doi: 10.1017/S0144686X13000573.
    1. Spilsbury K, Meyer J. Use, misuse and non-use of health care assistants: understanding the work of health care assistants in a hospital setting. J Nurs Manag. 2004;12(6):411–418. doi: 10.1111/j.1365-2834.2004.00515.x.
    1. Kolanowski A, Van Haitsma K, Penrod J, Hill N, Yevchak A. “Wish we would have known that!” Communication breakdown impedes person-centered care. Gerontologist. 2015;55(Suppl 1):s50–s60. doi: 10.1093/geront/gnv014.
    1. Locke EA, Latham GP. Building a practically useful theory of goal setting and task motivation: a 35-year odyssey. Am Psychol. 2002;57(9):705–717. doi: 10.1037/0003-066X.57.9.705.
    1. Regehr G, MacRae H, Reznick RK, Szalay D. Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination. Acad Med. 1998;73(9):993–997. doi: 10.1097/00001888-199809000-00020.
    1. van der Vleuten CP, Schuwirth LW. Assessing professional competence: from methods to programmes. Med Educ. 2005;39(3):309–317. doi: 10.1111/j.1365-2929.2005.02094.x.
    1. Morgan DL. Focus Groups as Qualitative Research/David L. Morgan. Thousand Oaks, Calif.: Sage Publications; 1997.
    1. Walton H, Tombor I, Burgess J, et al. Measuring fidelity of delivery of the Community Occupational Therapy in Dementia-UK intervention. BMC Geriatr. 2019;19(1). 10.1186/s12877-019-1385-7.
    1. Toomey E, Matthews J, Hurley DA. Using mixed methods to assess fidelity of delivery and its influencing factors in a complex self-management intervention for people with osteoarthritis and low back pain. BMJ Open. 2017;7(8). 10.1136/bmjopen-2016-015452.
    1. Denzin NK. Sociological methods. Second. New York: McGraw-Hill; 1978. doi:10.4324/9781315129945
    1. Ljunggren G, Perera NKP, Hägglund M. Inter-rater reliability in assessing exercise fidelity for the injury prevention exercise programme knee control in youth football players. Sport Med - Open. 2019;5(1). 10.1186/s40798-019-0209-9.
    1. Chan D. Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models. J Appl Psychol. 1998;83(2):234–246. doi: 10.1037/0021-9010.83.2.234.
    1. Peterson L, Homer AL, Wonderlich SA. The integrity of independent variables in behavior analysis. J Appl Behav Anal. 1982;15(4):477–492. doi: 10.1901/jaba.1982.15-477.
    1. Ruud T, Drake RE, Bond GR. Measuring fidelity to evidence-based practices: psychometrics. Adm Policy Ment Heal Ment Heal Serv Res. 2020;47(6):871–873. doi: 10.1007/s10488-020-01074-7.
    1. McGee D, Lorencatto F, Matvienko-Sikar K, Toomey E. Surveying knowledge, practice and attitudes towards intervention fidelity within trials of complex healthcare interventions 11 Medical and Health Sciences 1117 Public Health and Health Services. Trials. 2018;19(1). 10.1186/s13063-018-2838-6.
    1. Kirkpatrick DL. Evaluation of training. In: Craig R, Bittel L, editors. Training and Development Handbook. New York: McGraw Hill; 1967.
    1. Miller GE. The assessment of clinical skills/competence/performance. Acad Med. 1990;65(9 Suppl):S63–S67. doi: 10.1097/00001888-199009000-00045.

Source: PubMed

Sponsorer og samarbejdspartnere

Medicinske tilstande

Narkotikainterventioner

3
Abonner