Preliminary Evaluation of the Engagement and Effectiveness of a Mental Health Chatbot

Kate Daley, Ines Hungerbuehler, Kate Cavanagh, Heloísa Garcia Claro, Paul Alan Swinton, Michael Kapps, Kate Daley, Ines Hungerbuehler, Kate Cavanagh, Heloísa Garcia Claro, Paul Alan Swinton, Michael Kapps

Abstract

Background: Mental health difficulties are highly prevalent, yet access to support is limited by barriers of stigma, cost, and availability. These issues are even more prevalent in low- and middle-income countries, and digital technology is one potential way to overcome these barriers. Digital mental health interventions are effective but often struggle with low engagement rates, particularly in the absence of human support. Chatbots could offer a scalable solution, simulating human support at a lower cost. Objective: To complete a preliminary evaluation of engagement and effectiveness of Vitalk, a mental health chatbot, at reducing anxiety, depression and stress. Methods: Real world data was analyzed from 3,629 Vitalk users who had completed the first phase of a Vitalk program ("less anxiety," "less stress" or "better mood"). Programs were delivered through written conversation with a chatbot. Engagement was calculated from the number of responses sent to the chatbot divided by days in the program. Results: Users sent an average of 8.17 responses per day. For all three programs, target outcome scores reduced between baseline and follow up with large effect sizes for anxiety (Cohen's d = -0.85), depression (Cohen's d = -0.91) and stress (Cohen's d = -0.81). Increased engagement resulted in improved post-intervention values for anxiety and depression. Conclusion: This study highlights a chatbot's potential to reduce mental health symptoms in the general population within Brazil. While findings show promise, further research is required.

Keywords: anxiety; chatbot; conversational agent(s); depression; digital health; low-and middle-income; mental health; stress.

Conflict of interest statement

IH, KD, and MK are employees of TNH Health. TNH Health created the chatbot and paid for the cost of submitting the publication. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2020 Daley, Hungerbuehler, Cavanagh, Claro, Swinton and Kapps.

Figures

Figure 1
Figure 1
Screenshot of Vitalk check-up.

References

    1. Pan American Health Organization . The Burden of Mental Disorders in the Region of the Americas. Washington, DC: PAHO; (2018).
    1. Kohn R, Ali AA, Puac-Polanco V, Figueroa C, López-Soto V, Morgan K, et al. . Mental health in the Americas: an overview of the treatment gap. Rev Panam Salud Publica. (2018) 42:e165. 10.26633/RPSP.2018.165
    1. Andrade LH, Alonso J, Mneimneh Z, Wells JE, Al-Hamzawi A, Borges G, et al. . Barriers to mental health treatment: results from the WHO World Mental Health surveys. Psychol Med. (2014) 44:1303–17. 10.1017/S0033291713001943
    1. Fukuda CC, Penso MA, do Amparo DM, de Almeida BC, de Aquino Morais C. Mental health of young Brazilians: barriers to professional help-seeking. Estud Psicol. (2016) 33:355–65. 10.1590/1982-02752016000200017
    1. Fitzpatrick KK, Darcy A, Vierhile M. Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (Woebot): a randomized controlled trial. JMIR Mental Health. (2017) 4:e19. 10.2196/mental.7785
    1. Menezes P, Quayle J, Garcia Claro H, da Silva S, Brandt LR, Diez-Canseco F, et al. . Use of a mobile phone app to treat depression comorbid with hypertension or diabetes: a pilot study in Brazil and Peru. JMIR Ment Health. (2019) 6:e11698. 10.2196/11698
    1. Torous J, Chan SR, Tan SY-M, Behrens J, Mathew I, Conrad EJ, et al. . Patient smartphone ownership and interest in mobile apps to monitor symptoms of mental health conditions: a survey in four geographically distinct psychiatric clinics. JMIR Mental Health. (2014) 1:e5. 10.2196/mental.4004
    1. Firth J, Torous J, Nicholas J, Carney R, Rosenbaum S, Sarris J. Can smartphone mental health interventions reduce symptoms of anxiety? A meta-analysis of randomized controlled trials. J Affect Disord. (2017) 218:15–22. 10.1016/j.jad.2017.04.046
    1. Davies EB, Bethan Davies E, Morriss R, Glazebrook C. Computer-delivered and web-based interventions to improve depression, anxiety, and psychological well-being of university students: a systematic review and meta-analysis. J Med Int Res. (2014) 16:e130. 10.2196/jmir.3142
    1. Torous J, Nicholas J, Larsen ME, Firth J, Christensen H. Clinical review of user engagement with mental health smartphone apps: evidence, theory and improvements. Evid Based Ment Health. (2018) 21:116–9. 10.1136/eb-2018-102891
    1. Ly KH, Ly A-M, Andersson G. A fully automated conversational agent for promoting mental well-being: a pilot RCT using mixed methods. Internet Interv. (2017) 10:39–46. 10.1016/j.invent.2017.10.002
    1. Johansson R, Andersson G. Internet-based psychological treatments for depression. Expert Rev Neurother. (2012) 12:861–9; quiz 870. 10.1586/ern.12.63
    1. Baumeister H, Reichler L, Munzinger M, Lin J. The impact of guidance on Internet-based mental health interventions — A systematic review. Internet Interv. (2014) 1:205–15. 10.1016/j.invent.2014.08.003
    1. Suganuma S, Sakamoto D, Shimoyama H. An embodied conversational agent for unguided internet-based cognitive behavior therapy in preventative mental health: feasibility and acceptability pilot trial. JMIR Mental Health. (2018) 5:e10454. 10.2196/10454
    1. Vaidyam AN, Wisniewski H, Halamka JD, Kashavan MS, Torous JB. Chatbots and conversational agents in mental health: a review of the psychiatric landscape. Can J Psychiatry. (2019) 64:456–64. 10.1177/0706743719828977
    1. Gaffney H, Mansell W, Tai S. Conversational agents in the treatment of mental health problems: mixed-method systematic review. JMIR Ment Health. (2019) 6:e14166. 10.2196/14166
    1. Fulmer R, Joerin A, Gentile B, Lakerink L, Rauws M. Using psychological artificial intelligence (Tess) to relieve symptoms of depression and anxiety: randomized controlled trial. JMIR Mental Health. (2018) 5:e64. 10.2196/mental.9782
    1. Seligman MEP. Building human strength: psychology's forgotten mission. APA Monitor. (1998) 29. 10.1037/e529932010-003
    1. Beck JS. Cognitive Behavior Therapy, Second Edition: Basics and Beyond. New York, NY: Guilford Press; (2011). p. 391.
    1. Spitzer RL, Kroenke K, Williams JBW, Löwe B. A brief measure for assessing generalized anxiety disorder. Arch Int Med. (2006) 166:1092. 10.1001/archinte.166.10.1092
    1. Moreno AL, DeSousa DA, Ana Maria Frota Lisbôa, Manfro GG, Salum GA, Koller SH, et al. . Factor structure, reliability, and item parameters of the brazilian-portuguese version of the GAD-7 questionnaire. Temas Psicol. (2016) 24:367–76. 10.9788/TP2016.1-25
    1. Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. (2001) 16:606–13. 10.1046/j.1525-1497.2001.016009606.x
    1. Santos IS Tavares BF Munhoz TN Almeida LSP de Silva NTB da Tams BD . [Sensitivity and specificity of the Patient Health Questionnaire-9 (PHQ-9) among adults from the general population]. Cad Saude Publ. (2013) 29:1533–43. 10.1590/S0102-311X2013001200006
    1. Lovibond PF, Lovibond SH. The structure of negative emotional states: comparison of the Depression Anxiety Stress Scales (DASS) with the beck depression and anxiety inventories. Behav Res Ther. (1995) 33:335–43. 10.1016/0005-7967(94)00075-U
    1. Vignola RCB, Tucci AM. Adaptation and validation of the depression, anxiety and stress scale (DASS) to Brazilian Portuguese. J Affect Disord. (2014) 155:104–9. 10.1016/j.jad.2013.10.031
    1. Cohen J. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. Hillsdale, NJ: Lawrence Earlbaum Associates; (1988).
    1. Evans C, Margison F, Barkham M. The contribution of reliable and clinically significant change methods to evidence-based mental health. Evid Based Mental Health. (1998) 1:70–2. 10.1136/ebmh.1.3.70
    1. Jacobson NS, Truax P. Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. J Consult Clin Psychol. (1991) 59:12–9. 10.1037/0022-006X.59.1.12
    1. Roberts J, Lenton P, Keetharuth AD, Brazier J. Quality of life impact of mental health conditions in England: results from the adult psychiatric morbidity surveys. Health Qual Life Outcomes. (2014) 12:6. 10.1186/1477-7525-12-6
    1. R Core Team . R: A Language and Environment for Statistical Computing. Vienna: R Foundation for Statistical Computing; (2019). Available from: (accessed March 9, 2020).
    1. Cuponation . Facebook 2018/2019 Confira Levantamento Sobre Perfil e Quantidade de Usuarios. Available from: (accessed February 26, 2020).
    1. Andrade LH, Wang Y-P, Andreoni S, Silveira CM, Alexandrino-Silva C, Siu ER, et al. . Mental disorders in megacities: findings from the são paulo megacity mental health survey, Brazil. PLoS ONE. (2012) 7:e31879. 10.1371/journal.pone.0031879
    1. Oliveira MLC de, de Oliveira MLC, de Rosalmeida Dantas C, de Azevedo RCS, Banzato CEM. Demographics and complaints of university students who sought help at a campus mental health service between 1987 and 2004. São Paulo Med J. (2008) 126:58–62. 10.1590/S1516-31802008000100011
    1. Munhoz TN Santos IS Matijasevich A . Major depressive episode among Brazilian adults: a cross-sectional population-based study. J Affect Disord. (2013) 150:401–7. 10.1016/j.jad.2013.04.031
    1. Silva MT, Roa MC, Martins SS, da Silva ATC, Galvao TF. Generalized anxiety disorder and associated factors in adults in the Amazon, Brazil: a population-based study. J Affect Disord. (2018) 236:180–6. 10.1016/j.jad.2018.04.079
    1. Lindner P, Nyström MBT, Hassmén P, Andersson G, Carlbring P. Who seeks ICBT for depression and how do they get there? Effects of recruitment source on patient demographics and clinical characteristics. Internet Interv. (2015) 2:221–5. 10.1016/j.invent.2015.04.002
    1. Inkster B, Sarda S, Subramanian V. An empathy-driven, conversational artificial Intelligence Agent (Wysa) for digital mental well-being: real-world data evaluation mixed-methods study. JMIR Mhealth Uhealth. (2018) 6:e12106. 10.2196/12106
    1. Beintner I, Görlich D, Berger T, Ebert DD, Zeiler M, Herrero Camarano R, et al. . Interrelations between participant and intervention characteristics, process variables and outcomes in online interventions: a protocol for overarching analyses within and across seven clinical trials in ICare. Internet Interv. (2019) 16:86–97. 10.1016/j.invent.2018.05.001

Source: PubMed

Szponzorok és közreműködők

Egészségi állapot

Kábítószer-beavatkozások

3
Iratkozz fel