A randomized trial to identify accurate and cost-effective fidelity measurement methods for cognitive-behavioral therapy: project FACTS study protocol

Rinad S Beidas, Johanna Catherine Maclean, Jessica Fishman, Shannon Dorsey, Sonja K Schoenwald, David S Mandell, Judy A Shea, Bryce D McLeod, Michael T French, Aaron Hogue, Danielle R Adams, Adina Lieberman, Emily M Becker-Haimes, Steven C Marcus, Rinad S Beidas, Johanna Catherine Maclean, Jessica Fishman, Shannon Dorsey, Sonja K Schoenwald, David S Mandell, Judy A Shea, Bryce D McLeod, Michael T French, Aaron Hogue, Danielle R Adams, Adina Lieberman, Emily M Becker-Haimes, Steven C Marcus

Abstract

Background: This randomized trial will compare three methods of assessing fidelity to cognitive-behavioral therapy (CBT) for youth to identify the most accurate and cost-effective method. The three methods include self-report (i.e., therapist completes a self-report measure on the CBT interventions used in session while circumventing some of the typical barriers to self-report), chart-stimulated recall (i.e., therapist reports on the CBT interventions used in session via an interview with a trained rater, and with the chart to assist him/her) and behavioral rehearsal (i.e., therapist demonstrates the CBT interventions used in session via a role-play with a trained rater). Direct observation will be used as the gold-standard comparison for each of the three methods.

Methods/design: This trial will recruit 135 therapists in approximately 12 community agencies in the City of Philadelphia. Therapists will be randomized to one of the three conditions. Each therapist will provide data from three unique sessions, for a total of 405 sessions. All sessions will be audio-recorded and coded using the Therapy Process Observational Coding System for Child Psychotherapy-Revised Strategies scale. This will enable comparison of each measurement approach to direct observation of therapist session behavior to determine which most accurately assesses fidelity. Cost data associated with each method will be gathered. To gather stakeholder perspectives of each measurement method, we will use purposive sampling to recruit 12 therapists from each condition (total of 36 therapists) and 12 supervisors to participate in semi-structured qualitative interviews.

Discussion: Results will provide needed information on how to accurately and cost-effectively measure therapist fidelity to CBT for youth, as well as important information about stakeholder perspectives with regard to each measurement method. Findings will inform fidelity measurement practices in future implementation studies as well as in clinical practice.

Trial registration: NCT02820623 , June 3rd, 2016.

Keywords: Cost-effectiveness; Evidence-based practice; Implementation; Treatment fidelity.

Figures

Fig. 1
Fig. 1
CONSORT diagram

References

    1. IOM (Institute of Medicine). Psychosocial interventions for mental and substance use disorders: A framework for establishing evidence-based standards. Washington: The National Academies Press; 2015.
    1. Perepletchikova F, Treat TA, Kazdin AE. Treatment integrity in psychotherapy research: Analysis of the studies and examination of the associated factors. J Consult Clin Psych. 2007;75:829–841. doi: 10.1037/0022-006X.75.6.829.
    1. McLeod BD, Southam-Gerow MA, Tully CB, Rodriguez A, Smith MM. Making a case for treatment integrity as a psychosocial treatment quality indicator for youth mental health care. Clin Psychol-Sci Pr. 2013;20:14–32. doi: 10.1111/cpsp.12020.
    1. Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Health. 2011;38:4–23. doi: 10.1007/s10488-010-0327-7.
    1. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, Griffey R, Hensley M. Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011;38:65–76. doi: 10.1007/s10488-010-0319-7.
    1. Southam-Gerow MA, McLeod BD. Advances in applying treatment integrity research for dissemination and implementation science: Introduction to special issue. Clin Psychol-Sci Pr. 2013;20:1–13. doi: 10.1111/cpsp.12019.
    1. Schoenwald SK, Garland AF, Chapman JE, Frazier SL, Sheidow AJ, Southam-Gerow MA. Toward the effective and efficient measurement of implementation fidelity. Adm Policy Ment Health. 2011;38:32–43. doi: 10.1007/s10488-010-0321-0.
    1. Sherill J. Pragmatic Strategies for Assessing Psychotherapy Quality in Practice. Washington: National Institutes of Mental Health; 2015.
    1. Schoenwald SK, Garland AF. A review of treatment adherence measurement methods. Psychol Assess. 2013;25:146–156. doi: 10.1037/a0029715.
    1. Beidas RS, Cross W, Dorsey S. Show me, don’t tell me: Behavioral rehearsal as a training and analogue fidelity tool. Cogn Behav Pract. 2014;21:1–11. doi: 10.1016/j.cbpra.2013.04.002.
    1. Hurlburt MS, Garland AF, Nguyen K, Brookman-Frazee L. Child and family therapy process: Concordance of therapist and observational perspectives. Adm Policy Ment Health. 2010;37:230–244. doi: 10.1007/s10488-009-0251-x.
    1. Cunnington JP, Hanna E, Turnhbull J, Kaigas TB, Norman GR. Defensible assessment of the competency of the practicing physician. Acad Med. 1997;72:9–12. doi: 10.1097/00001888-199710001-00014.
    1. Goulet F, Jacques A, Gagnon R, Racette P, Sieber W. Assessment of family physicians’ performance using patient charts: Interrater reliability and concordance with chart-stimulated recall interview. Eval Health Prof. 2007;30:376–392. doi: 10.1177/0163278707307924.
    1. Miller PA, Nayer M, Eva KW. Psychometric properties of a peer-assessment program to assess continuing competence in physical therapy. Phys Ther. 2010;90:1026–1038. doi: 10.2522/ptj.20080137.
    1. Salvatori P, Simonavicius N, Moore J, Rimmer G, Patterson M. Meeting the challenge of assessing clinical competence of occupational therapists within a program management environment. Can J Occup Ther. 2008;75:51–60. doi: 10.1177/000841740807500109.
    1. Jennett P, Affleck L. Chart audit and chart stimulated recall as methods of needs assessment in continuing professional health education. J Contin Educ Health Prof. 1998;18:163–171. doi: 10.1002/chp.1340180306.
    1. Chambless DL, Ollendick TH. Empirically supported psychological interventions: Controversies and evidence. Annu Rev Psychol. 2001;52:685–716. doi: 10.1146/annurev.psych.52.1.685.
    1. Guerra CE, Jacobs SE, Holmes JH, Shea JA. Are physicians discussing prostate cancer screening with their patients and why or why not? A pilot study. J Gen Intern Med. 2007;22:901–907. doi: 10.1007/s11606-007-0142-3.
    1. McLeod BD, Weisz JR. The Therapy Process Observational Coding System for Child Psychotherapy-Strategies scale. J Clin Child Adolesc. 2010;39:436–443. doi: 10.1080/15374411003691750.
    1. Weersing VR, Weisz JR, Donenberg GR. Development of the Therapy Procedures Checklist: A therapist-report measure of technique use in child and adolescent treatment. J Clin Child Adolesc. 2002;31:168–180. doi: 10.1207/S15374424JCCP3102_03.
    1. Epstein RM. Assessment in medical education. N Engl J Med. 2007;356:387–396. doi: 10.1056/NEJMe078002.
    1. Carroll KM, Nich C, Sifry RL, Nuro KF, Frankforter TL, Ball SA, Fenton L, Rounsaville BJ. A general system for evaluating therapist adherence and competence in psychotherapy research in the addictions. Drug Alcohol Depend. 2000;57:225–238. doi: 10.1016/S0376-8716(99)00049-6.
    1. Stirman SW, Spokas M, Creed TA, Farabaugh DT, Bhar SS, Brown GK, Perivoliotis D, Grant PM, Beck AT. Training and consultation in evidence-based psychosocial treatments in public mental health settings: The ACCESS model. Prof Psychol Res Pr. 2010;41:48–56. doi: 10.1037/a0018099.
    1. Cohen J, Mannarino AP. Disseminating and implementing trauma-focused CBT in community settings. Trauma Violence Abuse. 2008;9:214–226. doi: 10.1177/1524838008324336.
    1. Beidas RS, Aarons G, Barg F, Evans A, Hadley T, Hoagwood K, Marcus S, Schoenwald S, Walsh L, Mandell DS. Policy to implementation: Evidence-based practice in community mental health—study protocol. Implement Sci. 2013;8:38. doi: 10.1186/1748-5908-8-38.
    1. Beidas R, Marcus S, Aarons GA, Hoagwood K, Schoenwald S, Evans A, Hurford M, Hadley T, Barg F, Walsh L, et al. Predictors of community therapists’ use of therapy techniques in a large public mental health system. JAMA Pediatr. in press.
    1. Peris T, Compton S, Kendall PC, Birmaher B, Sherill J, March J, Gosch E, Ginsburg G, Rynn M, McCracken J, et al. Trajectories of change in youth anxiety during cognitive behavior therapy. J Consult Clin Psych. in press.
    1. Rice ME, Harris GT. Comparing effect sizes in follow-up studies: ROC Area, Cohen’s d, and r. Law Human Behav. 2005;29:615–620. doi: 10.1007/s10979-005-6832-7.
    1. Drummond M, Sculpher M. Common methodological flaws in economic evaluations. Med Care. 2005;43:5–14. doi: 10.1097/01.mlr.0000170001.10393.b7.
    1. French MT, McGeary KA. Estimating the economic cost of substance abuse treatment. Health Econ. 1997;6:539–544. doi: 10.1002/(SICI)1099-1050(199709)6:5<539::AID-HEC295>;2-0.
    1. French MT, Roebuck MC, Dennis ML, Diamond G, Godley SH, Tims F, Webb C, Herrell JM. The economic cost of outpatient marijuana treatment for adolescents: Findings from a multi-site field experiment. Addiction. 2002;97:84–97. doi: 10.1046/j.1360-0443.97.s01.4.x.
    1. French MT, Roebuck MC, Dennis ML, Godley SH, Liddle HA, Tims FM. Outpatient marijuana treatment for adolescents: Economic evaluation of a multisite field experiment. Eval Rev. 2003;27:421–459. doi: 10.1177/0193841X03254349.
    1. French MT, Roebuck MC, McLellan AT. Cost estimation when time and resources are limited: The Brief DATCAP. J Subst Abuse Treat. 2004;27:187–193. doi: 10.1016/j.jsat.2004.07.001.
    1. French MT, Salome HJ, Sindelar JL, McLellan AT. Benefit-cost analysis of addiction treatment: Methodological guidelines and empirical application using the DATCAP and ASI. Health Serv Res. 2002;37:433–455. doi: 10.1111/1475-6773.031.
    1. McCollister KE, French MT, Sheidow AJ, Henggeler SW, Halliday-Boykins CA. Estimating the differential costs of criminal activity for juvenile drug court participants: Challenges and recommendations. J Behav Health Ser R. 2009;36:111–126. doi: 10.1007/s11414-007-9094-y.
    1. Roebuck MC, French MT, McLellan AT. DATStats: results from 85 studies using the Drug Abuse Treatment Cost Analysis Program. J Subst Abuse Treat. 2003;25:51–57. doi: 10.1016/S0740-5472(03)00067-9.
    1. Salome HJ, French MT. Using cost and financing instruments for economic evaluation of substance abuse treatment services. Recent Devel Alcohol. 2001;15:253–269.
    1. Salome HJ, French MT, Miller M, McLellan AT. Estimating the client costs of addiction treatment: First findings from the client drug abuse treatment cost analysis program (Client DATCAP) Drug Alcohol Depend. 2003;71:195–206. doi: 10.1016/S0376-8716(03)00133-9.
    1. Sindelar JL, Jofre-Bonet M, French MT, McLellan AT. Cost-effectiveness analysis of addiction treatment: Paradoxes of multiple outcomes. Drug Alcohol Depend. 2004;73:41–50. doi: 10.1016/j.drugalcdep.2003.09.002.
    1. DATCAP User Manual. . Accessed 3 Jan 2015.
    1. Glasgow RE, Riley WT. Pragmatic measures: What they are and why we need them. Am J Prev Med. 2013;45:237–243. doi: 10.1016/j.amepre.2013.03.010.
    1. Accurso EC, Taylor RM, Garland AF. Evidence-based practices addressed in community-based children’s mental health clinical supervision. Train Educ Prof Psychol. 2011;5:88–96.
    1. Schoenwald SK, Chapman JE, Kelleher K, Hoagwood KE, Landsverk J, Stevens J, Glisson C, Rolls-Reutz J, Health RNoYM A survey of the infrastructure for children’s mental health services: Implications for the implementation of empirically supported treatments (ESTs) Adm Policy Ment Health. 2008;35:84–97. doi: 10.1007/s10488-007-0147-6.
    1. Chambers DA, Azrin ST. Research and services partnerships: A fundamental component of dissemination and implementation research. Psychiatr Serv. 2013;64:509–511. doi: 10.1176/appi.ps.201300032.
    1. Sheeran P. Intention-behavior relations: A conceptual and empirical review. Eur Rev Soc Psychol. 2002. .
    1. Cialdini R. Influence: Science and Practice. Boston: Allyn & Bacon; 2001.
    1. Fishbein M, Cappella JN. The role of theory in developing effective health communications. J Commun. 2006;56:S1–S17. doi: 10.1111/j.1460-2466.2006.00280.x.
    1. Fishbein M, Ajzen I. Predicting and Changing Behavior: The Reasoned Action Approach. New York: Psychology Press; 2010.
    1. Guest G, Bunce A, Johnson L. How many interviews are enough? An experiment with data saturation and variability. Field Method. 2006;18:59–82. doi: 10.1177/1525822X05279903.
    1. Beatty P, Willis G. Research synthesis: The practice of cognitive interviewing. Public Opin Q. 2007;71:287–311. doi: 10.1093/poq/nfm006.
    1. Middlestadt S. Beliefs underlying eating better and moving more: Lessons learned from comparative salient belief elicitations. Ann Am Acad Polit SS. 2012;640:81–100. doi: 10.1177/0002716211425015.

Source: PubMed

Sponzoři a spolupracovníci

Zdravotní podmínky

Drogové intervence

3
Předplatit