A Comparison Between Clinical Guidelines and Real-World Treatment Data in Examining the Use of Session Summaries: Retrospective Study

Shiri Sadeh-Sharvit, Simon A Rego, Samuel Jefroykin, Gal Peretz, Tomer Kupershmidt, Shiri Sadeh-Sharvit, Simon A Rego, Samuel Jefroykin, Gal Peretz, Tomer Kupershmidt

Abstract

Background: Although behavioral interventions have been found to be efficacious and effective in randomized clinical trials for most mental illnesses, the quality and efficacy of mental health care delivery remains inadequate in real-world settings, partly owing to suboptimal treatment fidelity. This "therapist drift" is an ongoing issue that ultimately reduces the effectiveness of treatments; however, until recently, there have been limited opportunities to assess adherence beyond large randomized controlled trials.

Objective: This study explored therapists' use of a standard component that is pertinent across most behavioral treatments-prompting clients to summarize their treatment session as a means for consolidating and augmenting their understanding of the session and the treatment plan.

Methods: The data set for this study comprised 17,607 behavioral treatment sessions administered by 322 therapists to 3519 patients in 37 behavioral health care programs across the United States. Sessions were captured by a therapy-specific artificial intelligence (AI) platform, and an automatic speech recognition system transcribed the treatment meeting and separated the data to the therapist and client utterances. A search for possible session summary prompts was then conducted, with 2 psychologists validating the text that emerged.

Results: We found that despite clinical recommendations, only 54 (0.30%) sessions included a summary. Exploratory analyses indicated that session summaries mostly addressed relationships (n=27), work (n=20), change (n=6), and alcohol (n=5). Sessions with meeting summaries were also characterized by greater therapist interventions and included greater use of validation, complex reflections, and proactive problem-solving techniques.

Conclusions: To the best of our knowledge, this is the first study to assess a large, diverse data set of real-world treatment practices. Our findings provide evidence that fidelity with the core components of empirically designed psychological interventions is a challenge in real-world settings. The results of this study can inform the development of machine learning and AI algorithms and offer nuanced, timely feedback to providers, thereby improving the delivery of evidence-based practices and quality of mental health care services and facilitating better clinical outcomes in real-world settings.

Keywords: Empirically based practices; adherence; behavioral therapy; clinical training; natural language processing; psychotherapy; real-world data; real-world study; treatment fidelity.

Conflict of interest statement

Conflicts of Interest: SSS, SJ, GP, and TK are employees of Eleos Health Inc, which created the platform providing the data for this study. SAR is an unpaid advisor to Eleos Health Inc.

©Shiri Sadeh-Sharvit, Simon A Rego, Samuel Jefroykin, Gal Peretz, Tomer Kupershmidt. Originally published in JMIR Formative Research (https://formative.jmir.org), 16.08.2022.

Figures

Figure 1
Figure 1
Overview of the classification and data analysis process for real-world session summaries. ASR: automatic speech recognition; ML: machine learning; VADER: Valence Aware Dictionary and sEntiment Reasoner.

References

    1. Trautmann S, Rehm J, Wittchen H. The economic costs of mental disorders: Do our societies react appropriately to the burden of mental disorders? EMBO Rep. 2016 Sep;17(9):1245–1249. doi: 10.15252/embr.201642951. doi: 10.15252/embr.201642951.embr.201642951
    1. Chisholm D, Sweeny K, Sheehan P, Rasmussen B, Smit F, Cuijpers P, Saxena S. Scaling-up treatment of depression and anxiety: a global return on investment analysis. Lancet Psychiatry. 2016 May;3(5):415–424. doi: 10.1016/S2215-0366(16)30024-4. doi: 10.1016/s2215-0366(16)30024-4.
    1. Santomauro D, Mantilla Herrera AM, Shadid J, Zheng P, Ashbaugh C, Pigott D, Abbafati C, Adolph C, Amlag J, Aravkin AY, Bang-Jensen BL, Bertolacci GJ, Bloom SS, Castellano R, Castro E, Chakrabarti S, Chattopadhyay J, Cogen RM, Collins JK, Dai X, Dangel WJ, Dapper C, Deen A, Erickson M, Ewald SB, Flaxman AD, Frostad JJ, Fullman N, Giles JR, Giref AZ, Guo G, He J, Helak M, Hulland EN, Idrisov B, Lindstrom A, Linebarger E, Lotufo PA, Lozano R, Magistro B, Malta DC, Månsson JC, Marinho F, Mokdad AH, Monasta L, Naik P, Nomura S, O'Halloran JK, Ostroff SM, Pasovic M, Penberthy L, Reiner Jr RC, Reinke G, Ribeiro A, Sholokhov A, Sorensen RJD, Varavikova E, Vo AT, Walcott R, Watson S, Wiysonge CS, Zigler B, Hay SI, Vos T, Murray CJL, Whiteford HA, Ferrari AJ. Global prevalence and burden of depressive and anxiety disorders in 204 countries and territories in 2020 due to the COVID-19 pandemic. The Lancet. 2021 Nov;398(10312):1700–1712. doi: 10.1016/S0140-6736(21)02143-7. doi: 10.1016/s0140-6736(21)02143-7.
    1. OECD Tackling the mental health impact of the COVID-19 crisis: An integrated, whole-of-society response. OECD Policy Responses to Coronavirus (COVID-19). OECD Publishing, Paris. 2021 doi: 10.1787/0ccafa0b-en. doi: 10.1787/0ccafa0b-en.
    1. Waller G. Evidence-based treatment and therapist drift. Behav Res Ther. 2009 Feb;47(2):119–127. doi: 10.1016/j.brat.2008.10.018.S0005-7967(08)00222-2
    1. Waller G, Turner H. Therapist drift redux: Why well-meaning clinicians fail to deliver evidence-based therapy, and how to get back on track. Behav Res Ther. 2016 Feb;77:129–137. doi: 10.1016/j.brat.2015.12.005. doi: 10.1016/j.brat.2015.12.005.S0005-7967(15)30069-3
    1. Creed T, Crane M, Calloway A, Olino T, Kendall P, Stirman SW. Changes in Community Clinicians' Attitudes and Competence following a Transdiagnostic Cognitive Behavioral Therapy Training. Implement Res Pract. 2021 Jan 01;2:26334895211030220. doi: 10.1177/26334895211030220.
    1. Lau A, Lind T, Motamedi M, Lui J, Kuckertz M, Innes-Gomberg D, Pesanti K, Brookman-Frazee L. Prospective predictors of sustainment of multiple EBPs in a system-driven implementation context: Examining sustained delivery based on administrative claims. Implementation Research and Practice. 2021 Dec 23;2:263348952110578. doi: 10.1177/26334895211057884. doi: 10.1177/26334895211057884.
    1. Beidas R, Kendall P. Training Therapists in Evidence-Based Practice: A Critical Review of Studies From a Systems-Contextual Perspective. Clin Psychol (New York) 2010 Mar;17(1):1–30. doi: 10.1111/j.1468-2850.2009.01187.x.
    1. Bond G, Drake R. Assessing the Fidelity of Evidence-Based Practices: History and Current Status of a Standardized Measurement Methodology. Adm Policy Ment Health. 2020 Nov;47(6):874–884. doi: 10.1007/s10488-019-00991-6. 10.1007/s10488-019-00991-6
    1. Lewis C, Marti C, Marriott B, Scott K, Ayer D. Patterns of practice in community mental health treatment of adult depression. Psychother Res. 2019 Jan;29(1):70–77. doi: 10.1080/10503307.2017.1303210.
    1. Waller G, Stringer H, Meyer C. What cognitive behavioral techniques do therapists report using when delivering cognitive behavioral therapy for the eating disorders? J Consult Clin Psychol. 2012 Feb;80(1):171–175. doi: 10.1037/a0026559. doi: 10.1037/a0026559.2011-28275-001
    1. Kendall P, Frank H. Implementing evidence-based treatment protocols: Flexibility within fidelity. Clin Psychol (New York) 2018 Dec;25(4) doi: 10.1111/cpsp.12271.
    1. Dryden W. Single‐Session One‐At‐A‐Time Therapy: A Personal Approach. Aust N Z J Fam Ther. 2020 Oct 08;41(3):283–301. doi: 10.1002/anzf.1424. doi: 10.1002/anzf.1424.
    1. Beck JS. Cognitive Behavior Therapy (Third Edition): Basics and Beyond. New York, NY: Guilford Press; 2020.
    1. Dobson DJ, Dobson KS. In-Session Structure and Collaborative Empiricism. Cogn Behav Pract. 2013 Nov;20(4):410–418. doi: 10.1016/j.cbpra.2012.11.002.
    1. Rego SA. Treatment Plans and Interventions for Obsessive-Compulsive Disorder. New York, NY: Guilford Press; 2016.
    1. Ribeiro da Silva D, Rijo D, Castilho P, Gilbert P. The Efficacy of a Compassion-Focused Therapy–Based Intervention in Reducing Psychopathic Traits and Disruptive Behavior: A Clinical Case Study With a Juvenile Detainee. Clinical Case Studies. 2019 May 15;18(5):323–343. doi: 10.1177/1534650119849491. doi: 10.1177/1534650119849491.
    1. Sadeh-Sharvit S, Lock J. Parents with Eating Disorders: An Intervention Guide. New York, NY: Routledge; 2018.
    1. Waltman S, Hall B, McFarr L, Beck A, Creed T. In-Session Stuck Points and Pitfalls of Community Clinicians Learning CBT: Qualitative Investigation. Cogn Behav Pract. 2017 May;24(2):256–267. doi: 10.1016/j.cbpra.2016.04.002.
    1. Ginsburg G, Pella J, Pikulski P, Tein J, Drake K. School-Based Treatment for Anxiety Research Study (STARS): a Randomized Controlled Effectiveness Trial. J Abnorm Child Psychol. 2020 Mar;48(3):407–417. doi: 10.1007/s10802-019-00596-5. doi: 10.1007/s10802-019-00596-5.10.1007/s10802-019-00596-5
    1. Kavanagh D, Sitharthan G, Young R, Sitharthan T, Saunders J, Shockley N, Giannopoulos V. Addition of cue exposure to cognitive-behaviour therapy for alcohol misuse: a randomized trial with dysphoric drinkers. Addiction. 2006 Aug;101(8):1106–1116. doi: 10.1111/j.1360-0443.2006.01488.x. ADD1488
    1. Perlich A, Meinel C. Automatic Treatment Session Summaries in Psychotherapy – A Step towards Therapist-Patient Cooperation. Procedia Computer Science. 2015;63:276–283. doi: 10.1016/j.procs.2015.08.344. doi: 10.1016/j.procs.2015.08.344.
    1. Dyson M, Chlebowski C, Brookman-Frazee L. Therapists' Adaptations to an Intervention to Reduce Challenging Behaviors in Children with Autism Spectrum Disorder in Publicly Funded Mental Health Services. J Autism Dev Disord. 2019 Mar;49(3):924–934. doi: 10.1007/s10803-018-3795-3. 10.1007/s10803-018-3795-3
    1. Cummins R, Ewbank M, Martin A, Tablan V, Catarino A, Blackwell A. TIM: A Tool for Gaining Insights into Psychotherapy. WWW '19: The Web Conference; May 13-17, 2019; San Francisco, CA. 2019.
    1. Flaherty H. Teaching Note—Using Technology to Enhance Experiential Learning Through Simulated Role Plays. J Soc Work Educ. 2022 May 05;:1–7. doi: 10.1080/10437797.2022.2050869. doi: 10.1080/10437797.2022.2050869.
    1. Sadeh-Sharvit S, Hollon SD. Leveraging the Power of Nondisruptive Technologies to Optimize Mental Health Treatment: Case Study. JMIR Ment Health. 2020 Nov 26;7(11):e20646. doi: 10.2196/20646. v7i11e20646
    1. Hutto C, Gilbert E. VADER: A Parsimonious Rule-Based Model for Sentiment Analysis of Social Media Text. Eighth International AAAI Conference on Weblogs and Social Media; June 1-4, 2014; Ann Arbor, MI. 2014.
    1. Elbagir S, Yang J. Sentiment Analysis on Twitter with Python’s Natural Language Toolkit and VADER Sentiment Analyzer. International MultiConference of Engineers and Computer Scientists (IMECS 2019); March 13-15, 2019; Hong Kong. 2020.
    1. Speers A, Bhullar N, Cosh S, Wootton B. Correlates of therapist drift in psychological practice: A systematic review of therapist characteristics. Clin Psychol Rev. 2022 Apr;93:102132. doi: 10.1016/j.cpr.2022.102132. doi: 10.1016/j.cpr.2022.102132.S0272-7358(22)00017-4
    1. Janse P, De Jong K, Van Dijk MK, Hutschemaekers G, Verbraak MJPM. Improving the efficiency of cognitive-behavioural therapy by using formal client feedback. Psychother Res. 2017 Sep;27(5):525–538. doi: 10.1080/10503307.2016.1152408. doi: 10.1080/10503307.2016.1152408.
    1. Blonde L, Khunti K, Harris S, Meizinger C, Skolnik N. Interpretation and Impact of Real-World Clinical Data for the Practicing Clinician. Adv Ther. 2018 Nov;35(11):1763–1774. doi: 10.1007/s12325-018-0805-y. 10.1007/s12325-018-0805-y
    1. Gonzalez Salas Duhne P, Horan A, Ross C, Webb T, Hardy G. Assessing and promoting the use of implementation intentions in clinical practice. Soc Sci Med. 2020 Nov;265:113490. doi: 10.1016/j.socscimed.2020.113490. doi: 10.1016/j.socscimed.2020.113490.S0277-9536(20)30709-7

Source: PubMed

Sponsors en medewerkers

Medische omstandigheden

Geneesmiddelinterventies

3
Abonneren