Leveraging routine clinical materials and mobile technology to assess CBT fidelity: the Innovative Methods to Assess Psychotherapy Practices (imAPP) study

Shannon Wiltsey Stirman, Luana Marques, Torrey A Creed, Cassidy A Gutner, Robert DeRubeis, Paul G Barnett, Eric Kuhn, Michael Suvak, Jason Owen, Dawne Vogt, Booil Jo, Sonja Schoenwald, Clara Johnson, Kera Mallard, Matthew Beristianos, Heidi La Bash, Shannon Wiltsey Stirman, Luana Marques, Torrey A Creed, Cassidy A Gutner, Robert DeRubeis, Paul G Barnett, Eric Kuhn, Michael Suvak, Jason Owen, Dawne Vogt, Booil Jo, Sonja Schoenwald, Clara Johnson, Kera Mallard, Matthew Beristianos, Heidi La Bash

Abstract

Background: Identifying scalable strategies for assessing fidelity is a key challenge in implementation science. However, for psychosocial interventions, the existing, reliable ways to test treatment fidelity quality are often labor intensive, and less burdensome strategies may not reflect actual clinical practice. Cognitive behavioral therapies (CBTs) provide clinicians with a set of effective core elements to help treat a multitude of disorders, which, evidence suggests, need to be delivered with fidelity to maximize potential client impact. The current "gold standard" for rating CBTs is rating recordings of therapy sessions, which is extremely time-consuming and requires a substantial amount of initial training. Although CBTs can vary based on the target disorder, one common element employed in most CBTs is the use of worksheets to identify specific behaviors and thoughts that affect a client's ability to recover. The present study will develop and evaluate an innovative new approach to rate CBT fidelity, by developing a universal CBT scoring system based on worksheets completed in therapy sessions.

Methods: To develop a scoring system for CBT worksheets, we will compile common CBT elements from a variety of CBT worksheets for a range of psychiatric disorders and create adherence and competence measures. We will collect archival worksheets from past studies to test the scoring system and assess test-retest reliability. To evaluate whether CBT worksheet scoring accurately reflects clinician fidelity, we will recruit clinicians who are engaged in a CBT for depression, anxiety, and/or posttraumatic stress disorder. Clinicians and clients will transmit routine therapy materials produced in session (e.g., worksheets, clinical notes, session recordings) to the study team after each session. We will compare observer-rated fidelity, clinical notes, and fidelity-rated worksheets to identify the most effective and efficient method to assess clinician fidelity. Clients will also be randomly assigned to either complete the CBT worksheets on paper forms or on a mobile application (app) to learn if worksheet format influences clinician and client experience or differs in terms of reflecting fidelity.

Discussion: Scoring fidelity using CBT worksheets may allow clinics to test fidelity in a short and effective manner, enhancing continuous quality improvement in the workplace. Clinicians and clinics can use such data to improve clinician fidelity in real time, leading to improved patient outcomes.

Trial registration: ClinicalTrials.gov NCT03479398 . Retrospectively registered March 20, 2018.

Keywords: Behavioral health; Fidelity; Measurement.

Conflict of interest statement

Ethics approval and consent to participate

This study was approved by the institutional review board at the Stanford University and VA Palo Alto Healthcare System, Palo Alto, CA, the City of Philadelphia Department of Public Health Institutional Review Board, Philadelphia, PA, the institutional review board at VA Boston Healthcare System and Boston University, the institutional review board of Massachusetts General Hospital and the Partners Human Research Committee, Somerville, MA, and the institutional review board at University of Pennsylvania, Philadelphia, PA.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

References

    1. Beck AT. The current state of cognitive therapy: a 40-year retrospective. Arch Gen Psychiatry. 2005;62(9):953–959. doi: 10.1001/archpsyc.62.9.953.
    1. Clark DM. Implementing NICE guidelines for the psychological treatment of depression and anxiety disorders: the IAPT experience. Int Rev Psychiatry. 2011;23(4):318–327. doi: 10.3109/09540261.2011.606803.
    1. Karlin BE, Cross G. From the laboratory to the therapy room: national dissemination and implementation of evidence-based psychotherapies in the US Department of Veterans Affairs Health Care System. Am Psychol. 2014;69(1):19–33. doi: 10.1037/a0033888.
    1. Karlin BE, Ruzek JI, Chard KM, Eftekhari A, Monson CM, Hembree EA, Resick PA, Foa EB. Dissemination of evidence-based psychological treatments for posttraumatic stress disorder in the veterans health administration. J Trauma Stress. 2010;23(6):663–673. doi: 10.1002/jts.20588.
    1. Foa EB, Keane TM, Friedman MJ, Cohen JA. Effective treatments for PTSD: practice guidelines from the International Society for Traumatic Stress Studies. 2. New York, NY: Guilford Press; 2009.
    1. McHugh RK, Barlow DH. The dissemination and implementation of evidence-based psychological treatments: a review of current efforts. Am Psychol. 2010;65:73–84. doi: 10.1037/a0018121.
    1. Stirman SW, Buchhofer R, McLaulin JB, Evans AC, Beck AT. The Beck initiative: a partnership to implement cognitive therapy in a community behavioral health system. Psychiatr Serv. 2009;60:1302–1310. doi: 10.1176/ps.2009.60.10.1302.
    1. Mendel P, Meredith LS, Schoenbaum M, Sherbourne CD, Wells KB. Interventions in organizational and community context: a framework for building evidence on dissemination and implementation in health services research. Adm Policy Ment Health Ment Health Serv Res. 2008;35(1):21–37. doi: 10.1007/s10488-007-0144-9.
    1. Schoenwald SK. It’s a bird, it’s a plane, it’s … fidelity measurement in the real world. Clin Psychol Sci Pract. 2011;18(2):142–147. doi: 10.1111/j.1468-2850.2011.01245.x.
    1. Chambers DA, Glasgow RE, Stange KC. The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change. Implement Sci. 2013;8(1):117. doi: 10.1186/1748-5908-8-117.
    1. Schoenwald SK, Garland AF, Chapman JE, Frazier SL, Sheidow AJ, Southam-Gerow MA. Toward the effective and efficient measurement of implementation fidelity. Adm Policy Ment Health Ment Health Serv Res. 2011;38(1):32–43. doi: 10.1007/s10488-010-0321-0.
    1. Santa Ana EJ, Martino S, Ball SA, Nich C, Frankforter TL, Carroll KM. What is usual about ‘treatment-as-usual’? Data from two multisite effectiveness trials. J Subst Abus Treat. 2008;35(4):369–379. doi: 10.1016/j.jsat.2008.01.003.
    1. Stirman SW, Kimberly JR. Monitoring fidelity in longitudinal implementation research. Bethesda, MD: 2nd annual NIH Conference on the Science of Dissemination and Implementation: 2009; 2009.
    1. Stirman SW, Kimberly JR, Calloway A, Cook N, Castro F, Charns MP. The sustainability of new programs and interventions: a review of the empirical literature and recommendations for future research. Implement Sci. 2012;7:17. doi: 10.1186/1748-5908-7-17.
    1. Creed TA, Wolk CB, Feinberg B, Evans AC, Beck AT. Beyond the label: relationship between community therapists’ self-report of a cognitive behavioral therapy orientation and observed skills. Admin Pol Ment Health. 2016;43(1):36–43. doi: 10.1007/s10488-014-0618-5.
    1. Aarons G, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Health Ment Health Serv Res. 2011;38(1):4–23. doi: 10.1007/s10488-010-0327-7.
    1. Shediac-Rizkallah MC, Bone LR. Planning for the sustainability of community-based health programs: conceptual frameworks and future directions for research, practice and policy. Health Educ Res. 1998;13(1):87–108. doi: 10.1093/her/13.1.87.
    1. Racine DP. Reliable effectiveness: a theory on sustaining and replicating worthwhile innovations. Admin Pol Ment Health. 2006;33(3):356–387. doi: 10.1007/s10488-006-0047-1.
    1. Feldstein AC, Glasgow RE, Smith DH. A practical, robust implementation and sustainability model (PRISM) for integrating research findings into practice. Joint Comm J Qual Patient Saf. 2008;34(4):228–243. doi: 10.1016/S1553-7250(08)34030-6.
    1. Wells KB, Miranda J. Promise of interventions and services research: can it transform practice? Clin Psychol Sci Pract. 2006;13(1):99–104. doi: 10.1111/j.1468-2850.2006.00011.x.
    1. Nishith P, Resick PA. Cognitive processing therapy (CPT): therapist adherence and competence protocol. St. Louis: Unpublished manuscript, University of Missouri; 1994.
    1. Webb CA, Derubeis RJ, Barber JP. Therapist adherence/competence and treatment outcome: a meta-analytic review. J Consult Clin Psychol. 2010;78(2):200–211. doi: 10.1037/a0018912.
    1. Farmer CC, Mitchell KS, Parker-Guilbert K, Galovski TE. Fidelity to the cognitive processing therapy protocol: evaluation of critical elements. Behav Ther. 2017;48(2):195–206. doi: 10.1016/j.beth.2016.02.009.
    1. Carroll KM, Martino S, Rounsaville BJ. No train, no gain? Clin Psychol Sci Pract. 2010;17(1):36–40. doi: 10.1111/j.1468-2850.2009.01190.x.
    1. Decker SE, Jameson MT, Naugle AE. Therapist training in empirically supported treatments: a review of evaluation methods for short- and long-term outcomes. Adm Policy Ment Health Ment Health Serv Res. 2011;38(4):254–86.
    1. Perepletchikova F, Treat TA, Kazdin AE. Treatment integrity in psychotherapy research: analysis of the studies and examination of the associated factors. J Consult Clin Psychol. 2007;75(6):829–841. doi: 10.1037/0022-006X.75.6.829.
    1. Kauth MR, Sullivan G, Blevins D, Cully JA, Landes RD, Said Q, Teasdale TA. Employing external facilitation to implement cognitive behavioral therapy in VA clinics: a pilot study. Implement Sci. 2010;5(75):1–11.
    1. Hrisos S, Eccles M, Francis J, Dickinson H, Kaner E, Beyer F, Johnston M. Are there valid proxy measures of clinical behaviour? A systematic review. Implement Sci. 2009;4:37. doi: 10.1186/1748-5908-4-37.
    1. Eccles M, Hrisos S, Francis J, Kaner E, Dickinson H, Beyer F, Johnston M. Do self-reported intentions predict clinicians’ behaviour: a systematic review. Implement Sci. 2006;1(1):28. doi: 10.1186/1748-5908-1-28.
    1. Liddle HA, Rowe C, Gonzalez A, Henderson CE, Dakof G, Greenbaum P. Changing provider practices, program environment, and improving outcomes by transporting multidimensional family therapy to an adolescent drug treatment setting. Am J Addict. 2006;15:102–112. doi: 10.1080/10550490601003698.
    1. Jensen-Doss A, Cusack KJ, de Arellano M. Workshop-based training in trauma-focused CBT: an in-depth analysis of impact on provider practices. Community Ment Health J. 2008;44:227–244. doi: 10.1007/s10597-007-9121-8.
    1. Jacobson NS, Dobson KS, Truax PA, Addis ME, Koerner K, Gollan JK, Gortner E, Prince SE. A component analysis of cognitive-behavioral treatment for depression. J Consult Clin Psychol. 1996;64(2):295. doi: 10.1037/0022-006X.64.2.295.
    1. Acierno R, Gros DF, Ruggiero KJ, Hernandez-Tejada BMA, Knapp RG, Lejuez CW, Muzzy W, Frueh CB, Egede LE, Tuerk PW. Behavioral activation and therapeutic exposure for posttraumatic stress disorder: a noninferiority trial of treatment delivered in person versus home-based telehealth. Depress Anxiety. 2016;33(5):415–423. doi: 10.1002/da.22476.
    1. Keefe JR, Webb CA, DeRubeis RJ. Cognitive therapy for depression, early focus on maladaptive beliefs may be especially efficacious for patients with personality disorders. J Consult Clin Psychol. 2016;84(4):353–364. doi: 10.1037/ccp0000071.
    1. Vogt D, Shipherd JC, Resick PA. Posttraumatic maladaptive beliefs scale: evolution of the personal beliefs and reactions scale. Assessment. 2012;19(3):308–317. doi: 10.1177/1073191110376161.
    1. Evans-Jones C. The CBT handbook: cognitive behavior therapy Speechmark. 2010.
    1. Resick PA, Nishith P, Griffin MG. How well does cognitive-behavioral therapy treat symptoms of complex PTSD? An examination of child sexual abuse survivors within a clinical trial. CNS Spectrums. 2003;8:340–355. doi: 10.1017/S1092852900018605.
    1. Ben-Zeev D, Davis KE, Kaiser S, Krzsos I, Drake RE. Mobile technologies among people with serious mental illness: opportunities for future services. Adm Policy Ment Health Ment Health Serv Res. 2013;40(4):340–343. doi: 10.1007/s10488-012-0424-x.
    1. Poushter J. Smartphone ownership and internet usage continues to climb in emerging economies. Pew Research Center. 2016;22:1-44.
    1. Wiltsey Stirman S, Gutner C, Gamarra J, Vogt D, Suvak M, Wachen J, Dondanville K, Yarvis J, Mintz J, Peterson A, et al. Leveraging routine clinical materials to assess fidelity to CPT. In: Proceedings of the 3rd Biennial Conference of the Society for Implementation Research Collaboration (SIRC) 2015: advancing efficient methodologies through community partnerships and team science. Implement Sci. 2016;11(1):1.
    1. Jackson DN. The dynamics of structured personality tests. Psychol Rev. 1971;78:229–248. doi: 10.1037/h0030852.
    1. Nunnally JC. Psychometric theory 2nd edn. New York: McGraw-Hill; 1978.
    1. Vogt D, Colvin R. Assessment of accurate self-knowledge. J Pers Assess. 2005;84(3):239–251. doi: 10.1207/s15327752jpa8403_03.
    1. Valentine SE, Borba CP, Dixon L, Vaewsorn AS, Guajardo JG, Resick PA, Wiltsey Stirman S, Marques L. Cognitive processing therapy for Spanish-speaking Latinos: a formative study of a model-driven cultural adaptation of the manual to enhance implementation in a usual care setting. J Clin Psychol. 2017;73(3):239–256. doi: 10.1002/jclp.22337.
    1. Creed TA, Frankel SA, German RE, Green KL, Jager-Hyman S, Taylor KP, Adler AD, Wolk CB, Stirman SW, Waltman SH, Williston MA. Implementation of transdiagnostic cognitive therapy in community behavioral health: The Beck Community Initiative. J Consult Clin Psychol. 2016;84(12):1116.
    1. Stirman SW, Pontoski K, Creed TA, Xhezo R, Evans AC, Beck AT, Crits-Christoph P. A non-randomized comparison of strategies for consultation in a community-academic training program to implement an evidence-based psychotherapy. Adm Policy Mental Health Serv Mental Health Serv Res. 2017;44(1):55-66.
    1. Stirman SW, Shields N, Deloriea J, Landy MSH, Belus JM, Maslej MM, Monson CM. A randomized controlled dismantling trial of post-workshop consultation strategies to increase effectiveness and fidelity to an evidence-based psychotherapy for posttraumatic stress disorder. Implement Sci. 2013;8(82):1–8.
    1. Weathers FW, Litz BT, Keane TM, Palmieri PA, Marx BP, Schnurr PP: The PTSD checklist for DSM–5 (PCL-5). In: Scale available from the National Center for PTSD website. 2010. . Accessed 5 Dec 2017.
    1. Beck AT, Steer RA, Ball R, Ranieri WF. Comparison of Beck Depression Inventories-IA and -II in psychiatric outpatients. J Pers Assess. 1996;67(3):588–597. doi: 10.1207/s15327752jpa6703_13.
    1. Beck AT, Steer RA. Manual for the Beck anxiety inventory. 1990.
    1. Beck AT, Steer RA. Beck hopelessness scale. Psychological Corporation San Antonio; 1988.
    1. Stirman SW, Shields N, Deloriea J, Landy MS, Belus JM, Maslej MM, Lane J, Monson CM. A randomized comparison of three post-workshop consultation strategies. Philadelphia, PA: Association for Behavioral and Cognitive Therapies: 2014; 2014.
    1. Marques L, Eustis EH, Dixon L, Valentine SE, Borba C, Simon NM, Kaysen D, Wiltsey-Stirman S. Delivering cognitive processing therapy in a community health setting: the influence of Latino culture and community violence on posttraumatic cognitions. Psychol Trauma: Theory Res, Pract, Policy. 2016;9:98–106. doi: 10.1037/tra0000044.
    1. Gutner C, Barlow D, Sloan D, Wiltsey Stirman S. What do you really think? Patient, clinician and stakeholder views on transdiagnostic mental health treatment for veterans. In: Ametaj A, editor. Transdiagnostic and common element interventions: addressing multidimensional barriers to dissemination and implementation of evidence-based practices. Chicago: Association for Behavioral and Cognitive Therapies; 2015.
    1. Creed TA, Stirman SW, Evans AC, Beck AT. A model for implementation of cognitive therapy in community mental health: the Beck initiative. Behav Therapist. 2014;37(3):56-64.
    1. Gutner C, Barlow D, Sloan D, Stirman SW, Ametaj A, Chicago IL. Association for Behavioral and Cognitive Therapies. Chicago, IL: SRC - GoogleScholar; 2015. What do you really think? Patient, clinician and stakeholder views on transdiagnostic mental health treatment for veterans.
    1. Kroenke K, Spitzer RL. The PHQ-9: a new depression diagnostic and severity measure. Psychiatr Ann. 2002;32(9):509–515. doi: 10.3928/0048-5713-20020901-06.
    1. Beck AT, Epstein N, Brown GK, Steer RA. An inventory for measuring clinical anxiety: psychometric properties. J Consult Clin Psychol. 1988;56(6):893–897. doi: 10.1037/0022-006X.56.6.893.
    1. Rodriguez P, Holowka DW, Marx BP. Assessment of posttraumatic stress disorder-related functional impairment: a review. J Rehabil Res Dev. 2012;49(5):649–665. doi: 10.1682/JRRD.2011.09.0162.
    1. Shaw BF, Elkin I, Yamaguchi J, Olmsted M, Vallis TM, Dobson KS, Lowery A, Sotsky SM, Watkins JT, Imber SD. Therapist competence ratings in relation to clinical outcome in cognitive therapy of depression. J Consult Clin Psychol. 1999;67(6):837–846. doi: 10.1037/0022-006X.67.6.837.
    1. Dobson KS, Shaw BF, Vallis TM. Reliability of a measure of the quality of cognitive therapy. Br J Clin Psychol. 1985;24(4):295. doi: 10.1111/j.2044-8260.1985.tb00662.x.
    1. Beidas R, Marcus S, Aarons GA, Hoagwood KE, Schoenwald SK, Evans AC, Hurford MO, Hadley T, Barg FK, Walsh LM, et al. Predictors of community therapists’ use of therapy techniques in a large public mental health system. JAMA Pediatr. 2015.
    1. Cook JM, Thompson R, Schnurr PP. Perceived characteristics of intervention scale: development and psychometric properties. Assessment. 2015;22(6):704–714. doi: 10.1177/1073191114561254.
    1. Aarons GA, Glisson C, Hoagwood K, Kelleher K, Landsverk J, Cafri G. Health TRNoYM: psychometric properties and U.S. national norms of the evidence-based practice attitude scale (EBPAS) Psychol Assess. 2010;22(2):356–365. doi: 10.1037/a0019188.
    1. Carpenter KM, Cheng WY, Smith JL, Brooks AC, Amrhein PC, Wain RM, Nunes EV. “Old dogs” and new skills: how clinician characteristics relate to motivational interviewing skills before, during, and after training. J Consult Clin Psychol. 2012;80(4):560. doi: 10.1037/a0028362.
    1. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4(1):50. doi: 10.1186/1748-5908-4-50.
    1. Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979;86(2):420–428. doi: 10.1037/0033-2909.86.2.420.
    1. Strunk DR, Brotman MA, DeRubeis RJ, Hollon SD. Therapist competence in cognitive therapy for depression: predicting subsequent symptom change. J Consult Clin Psychol. 2010;78(3):429–437. doi: 10.1037/a0019631.
    1. Cichetti D. Guidelines, criteria, and rules of thumb for normed and standardized instruments in psychology. Psychol Assess. 1994;6(4):284–290. doi: 10.1037/1040-3590.6.4.284.
    1. Resick PA, Nishith P, Weaver TL, Astin MC, Feuer CA. A comparison of cognitive-processing therapy with prolonged exposure and a waiting condition for the treatment of chronic posttraumatic stress disorder in female rape victims. J Consult Clin Psychol. 2002;70(4):867. doi: 10.1037/0022-006X.70.4.867.
    1. Resick PA, Galovski TE, Uhlmansiek MOB, Scher CD, Clum GA, Young-Xu Y. A randomized clinical trial to dismantle components of cognitive processing therapy for posttraumatic stress disorder in female victims of interpersonal violence. J Consult Clin Psychol. 2008;76(2):243. doi: 10.1037/0022-006X.76.2.243.
    1. Zar JH. Spearman rank correlation. In: Armitage P, Colton T, editors. Encyclopedia of Biostat. Chichester: Wiley; 1998.
    1. Hastie T, Tibshirani R, Friedman J, Hastie T, Tibshirani R, Friedman J. Cross-validation. (241–249) New York. NY: Springer New York Inc; 2009.
    1. Strunk DR, Brotman MA, DeRubeis RJ. The process of change in cognitive therapy for depression: predictors of early inter-session symptom gains. Behav Res Ther. 2010;48(7):599–606. doi: 10.1016/j.brat.2010.03.011.
    1. Barber JP, Triffleman E, Marmar C. Considerations in treatment integrity: implications and recommendations for PTSD research. J Trauma Stress. 2007;20(5):793–805. doi: 10.1002/jts.20295.
    1. Hedeker D, Gibbons RD. Application of random-effects pattern-mixture models for missing data in longitudinal studies. Psychol Methods. 1997;2(1):64–78. doi: 10.1037/1082-989X.2.1.64.
    1. Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, Gøtzsche PC, Lang T. The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med. 2001;134(8):663–694. doi: 10.7326/0003-4819-134-8-200104170-00012.
    1. Zalta AK, Gillihan SJ, Fisher AJ, Mintz J, McLean CP, Yehuda R, Foa EB. Change in negative cognitions associated with PTSD predicts symptom reduction in prolonged exposure. J Consult Clin Psychol. 2014;82(1):171. doi: 10.1037/a0034735.
    1. Granger CW. Investigating causal relations by econometric models and cross-spectral methods. Econometrica: J Econometric Soc. 1969:424–38.
    1. Baldwin SA, Imel ZE, Braithwaite SR, Atkins DC. Analyzing multiple outcomes in clinical research using multivariate multilevel models. J Consult Clin Psychol. 2014;82(5):920. doi: 10.1037/a0035628.
    1. Raudenbush SW. Statistical analysis and optimal design for cluster randomized trials. Psychol Methods. 1997;2(2):173. doi: 10.1037/1082-989X.2.2.173.
    1. Snijders TA. Power and sample size in multilevel linear models. Encyclopedia Stat Behav Sci. 2005;3:1570-3.
    1. Strunk DR, Cooper AA, Ryan ET, DeRubeis RJ, Hollon SD. The process of change in cognitive therapy for depression when combined with antidepressant medication: predictors of early intersession symptom gains. J Consult Clin Psychol. 2012;80(5):730–738. doi: 10.1037/a0029281.
    1. Committe on Comparative Effectiveness Research Priorization IoM. Initial national priotities for comparative effectiveness research. Washington, DC: The National Academies Press; 2009.
    1. Glasgow RE, Lichtenstein E, Marcus AC. Why don’t we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition. Am J Public Health. 2003;93:1261–1267. doi: 10.2105/AJPH.93.8.1261.

Source: PubMed

Patrocinadores e Colaboradores

Condições médicas

Intervenções de drogas

3
Se inscrever