An Empathy-Driven, Conversational Artificial Intelligence Agent (Wysa) for Digital Mental Well-Being: Real-World Data Evaluation Mixed-Methods Study

Becky Inkster, Shubhankar Sarda, Vinod Subramanian, Becky Inkster, Shubhankar Sarda, Vinod Subramanian

Abstract

Background: A World Health Organization 2017 report stated that major depression affects almost 5% of the human population. Major depression is associated with impaired psychosocial functioning and reduced quality of life. Challenges such as shortage of mental health personnel, long waiting times, perceived stigma, and lower government spends pose barriers to the alleviation of mental health problems. Face-to-face psychotherapy alone provides only point-in-time support and cannot scale quickly enough to address this growing global public health challenge. Artificial intelligence (AI)-enabled, empathetic, and evidence-driven conversational mobile app technologies could play an active role in filling this gap by increasing adoption and enabling reach. Although such a technology can help manage these barriers, they should never replace time with a health care professional for more severe mental health problems. However, app technologies could act as a supplementary or intermediate support system. Mobile mental well-being apps need to uphold privacy and foster both short- and long-term positive outcomes.

Objective: This study aimed to present a preliminary real-world data evaluation of the effectiveness and engagement levels of an AI-enabled, empathetic, text-based conversational mobile mental well-being app, Wysa, on users with self-reported symptoms of depression.

Methods: In the study, a group of anonymous global users were observed who voluntarily installed the Wysa app, engaged in text-based messaging, and self-reported symptoms of depression using the Patient Health Questionnaire-9. On the basis of the extent of app usage on and between 2 consecutive screening time points, 2 distinct groups of users (high users and low users) emerged. The study used mixed-methods approach to evaluate the impact and engagement levels among these users. The quantitative analysis measured the app impact by comparing the average improvement in symptoms of depression between high and low users. The qualitative analysis measured the app engagement and experience by analyzing in-app user feedback and evaluated the performance of a machine learning classifier to detect user objections during conversations.

Results: The average mood improvement (ie, difference in pre- and post-self-reported depression scores) between the groups (ie, high vs low users; n=108 and n=21, respectively) revealed that the high users group had significantly higher average improvement (mean 5.84 [SD 6.66]) compared with the low users group (mean 3.52 [SD 6.15]); Mann-Whitney P=.03 and with a moderate effect size of 0.63. Moreover, 67.7% of user-provided feedback responses found the app experience helpful and encouraging.

Conclusions: The real-world data evaluation findings on the effectiveness and engagement levels of Wysa app on users with self-reported symptoms of depression show promise. However, further work is required to validate these initial findings in much larger samples and across longer periods.

Keywords: artificial intelligence; chatbots; conversational agents; coping skills; depression; emotions; empathy; mHealth; mental health; resilience, psychological.

Conflict of interest statement

Conflicts of Interest: BI is a scientific advisor to Wysa with no fiduciary associations. VS is an independent research consultant at Wysa and draws a consulting fee. SS is a technical lead and a paid employee at Wysa.

©Becky Inkster, Shubhankar Sarda, Vinod Subramanian. Originally published in JMIR Mhealth and Uhealth (http://mhealth.jmir.org), 23.11.2018.

Figures

Figure 1
Figure 1
The study inclusion criteria. PHQ: Patient Health Questionnaire.
Figure 2
Figure 2
Thematic map with prevalence.
Figure 3
Figure 3
Coping experience–based feedback response distribution.
Figure 4
Figure 4
Coping major event-app experience-based feedback response distribution.
Figure 5
Figure 5
Confusion matrix of the objection handling machine learning model.

References

    1. World Health Organization. Geneva: World Health Organization; 2017. Depression and Other Common Mental Disorders: Global Health Estimates
    1. Ishak WW, Balayan K, Bresee C, Greenberg JM, Fakhry H, Christensen S, Rapaport MH. A descriptive analysis of quality of life using patient-reported measures in major depressive disorder in a naturalistic outpatient setting. Qual Life Res. 2013 Apr;22(3):585–96. doi: 10.1007/s11136-012-0187-6.
    1. Fried EI, Nesse RM. The impact of individual depressive symptoms on impairment of psychosocial functioning. PLoS One. 2014;9(2):e90311. doi: 10.1371/journal.pone.0090311.
    1. Pulcu E, Elliott R. Neural origins of psychosocial functioning impairments in major depression. Lancet Psychiatry. 2015 Sep;2(9):835–43. doi: 10.1016/S2215-0366(15)00237-0.
    1. Greenberg PE, Fournier A, Sisitsky T, Pike CT, Kessler RC. The economic burden of adults with major depressive disorder in the United States (2005 and 2010) J Clin Psychiatry. 2015 Feb;76(2):155–62. doi: 10.4088/JCP.14m09298.
    1. Centre for Mental Health 2010. [2018-08-23]. The economic and social costs of mental health problems in 2009/10 .
    1. Farmer P, Stevenson D. . 2017. Oct 26, Thriving at Work: a review of mental health and employers .
    1. World Health Organization. Geneva: World Health Organization; 2018. Mental Health Atlas 2017
    1. Cuijpers P, Sijbrandij M, Koole SL, Andersson G, Beekman AT, Reynolds CF. The efficacy of psychotherapy and pharmacotherapy in treating depressive and anxiety disorders: a meta-analysis of direct comparisons. World Psychiatry. 2013 Jun;12(2):137–48. doi: 10.1002/wps.20038. doi: 10.1002/wps.20038.
    1. Patel V. SUNDAR: mental health for all by all. BJPsych Int. 2015 Feb;12(1):21–3. doi: 10.1192/S2056474000000118.
    1. Department of Health, UK British Library. 2014. Sep 25, [2018-08-23]. Access and waiting times standards for 2015-16 in mental health services: Impact Assessment .
    1. British Medical Association. 2018. Feb 26, New BMA research unveils blindspot in mental healthcare .
    1. Pedersen ER, Paves AP. Comparing perceived public stigma and personal stigma of mental health treatment seeking in a young adult sample. Psychiatry Res. 2014 Sep 30;219(1):143–50. doi: 10.1016/j.psychres.2014.05.017.
    1. Andrade LH, Alonso J, Mneimneh Z, Wells JE, Al-Hamzawi A, Borges G, Bromet E, Bruffaerts R, de Girolamo G, de Graaf R, Florescu S, Gureje O, Hinkov HR, Hu C, Huang Y, Hwang I, Jin R, Karam EG, Kovess-Masfety V, Levinson D, Matschinger H, O'Neill S, Posada-Villa J, Sagar R, Sampson NA, Sasu C, Stein DJ, Takeshima T, Viana MC, Xavier M, Kessler RC. Barriers to mental health treatment: results from the WHO World Mental Health surveys. Psychol Med. 2014 Apr;44(6):1303–17. doi: 10.1017/S0033291713001943.
    1. Tindall L, Mikocka-Walus A, McMillan D, Wright B, Hewitt C, Gascoyne S. Is behavioural activation effective in the treatment of depression in young people? A systematic review and meta-analysis. Psychol Psychother. 2017 Dec;90(4):770–96. doi: 10.1111/papt.12121.
    1. Cuijpers P, Donker T, van Straten A, Li J, Andersson G. Is guided self-help as effective as face-to-face psychotherapy for depression and anxiety disorders? A systematic review and meta-analysis of comparative outcome studies. Psychol Med. 2010 Dec;40(12):1943–57. doi: 10.1017/S0033291710000772.
    1. Lucas GM, Rizzo A, Gratch J, Scherer S, Stratou G, Boberg J, Morency LP. Reporting mental health symptoms: breaking down barriers to care with virtual human interviewers. Front Robot AI. 2017 Oct 12;4 doi: 10.3389/frobt.2017.00051.
    1. Callan JA, Wright J, Siegle GJ, Howland RH, Kepler BB. Use of computer and mobile technologies in the treatment of depression. Arch Psychiatr Nurs. 2017 Dec;31(3):311–18. doi: 10.1016/j.apnu.2016.10.002.
    1. Carlbring P, Andersson G, Cuijpers P, Riper H, Hedman-Lagerlöf E. Internet-based vs face-to-face cognitive behavior therapy for psychiatric and somatic disorders: an updated systematic review and meta-analysis. Cogn Behav Ther. 2018 Jan;47(1):1–18. doi: 10.1080/16506073.2017.1401115.
    1. Ebert DD, Donkin L, Andersson G, Andrews G, Berger T, Carlbring P, Rozenthal A, Choi I, Laferton JA, Johansson R, Kleiboer A, Lange A, Lehr D, Reins JA, Funk B, Newby J, Perini S, Riper H, Ruwaard J, Sheeber L, Snoek FJ, Titov N, Ünlü IB, van Bastelaar K, Vernmark K, van Straten SA, Warmerdam L, Salsman N, Cuijpers P. Does Internet-based guided-self-help for depression cause harm? An individual participant data meta-analysis on deterioration rates and its moderators in randomized controlled trials. Psychol Med. 2016 Oct;46(13):2679–93. doi: 10.1017/S0033291716001562.
    1. Karyotaki E, Riper H, Twisk J, Hoogendoorn A, Kleiboer A, Mira A, Mackinnon A, Meyer B, Botella C, Littlewood E, Andersson G, Christensen H, Klein JP, Schröder J, Bretón-López J, Scheider J, Griffiths K, Farrer L, Huibers MJH, Phillips R, Gilbody S, Moritz S, Berger T, Pop V, Spek V, Cuijpers P. Efficacy of self-guided internet-based cognitive behavioral therapy in the treatment of depressive symptoms: a meta-analysis of individual participant data. J Am Med Assoc Psychiatry. 2017 Apr 01;74(4):351–59. doi: 10.1001/jamapsychiatry.2017.0044.
    1. Andrews G, Basu A, Cuijpers P, Craske MG, McEvoy P, English CL, Newby JM. Computer therapy for the anxiety and depression disorders is effective, acceptable and practical health care: an updated meta-analysis. J Anxiety Disord. 2018 Apr;55:70–8. doi: 10.1016/j.janxdis.2018.01.001.
    1. Rathbone AL, Clarry L, Prescott J. Assessing the efficacy of mobile health apps using the basic principles of cognitive behavioral therapy: systematic review. J Med Internet Res. 2017 Nov 28;19(11):e399. doi: 10.2196/jmir.8598.
    1. Firth J, Torous J, Nicholas J, Carney R, Pratap A, Rosenbaum S, Sarris J. The efficacy of smartphone-based mental health interventions for depressive symptoms: a meta-analysis of randomized controlled trials. World Psychiatry. 2017 Oct;16(3):287–98. doi: 10.1002/wps.20472. doi: 10.1002/wps.20472.
    1. Ly KH, Topooco N, Cederlund H, Wallin A, Bergström J, Molander O, Carlbring P, Andersson G. Smartphone-supported versus full behavioural activation for depression: a randomised controlled trial. PLoS One. 2015;10(5):e0126559. doi: 10.1371/journal.pone.0126559.
    1. Ly KH, Trüschel A, Jarl L, Magnusson S, Windahl T, Johansson R, Carlbring P, Andersson G. Behavioural activation versus mindfulness-based guided self-help treatment administered through a smartphone application: a randomised controlled trial. Br Med J Open. 2014;4(1):e003440. doi: 10.1136/bmjopen-2013-003440.
    1. Ly KH, Janni E, Wrede R, Sedem M, Donker T, Carlbring P, Andersson G. Experiences of a guided smartphone-based behavioral activation therapy for depression: a qualitative study. Internet Interv. 2015 Mar;2(1):60–8. doi: 10.1016/j.invent.2014.12.002.
    1. Ben-Zeev D, Brian RM, Jonathan G, Razzano L, Pashka N, Carpenter-Song E, Drake RE, Scherer EA. Mobile health (mHealth) versus clinic-based group intervention for people with serious mental illness: a randomized controlled trial. Psychiatr Serv. 2018 May 25;:appips201800063. doi: 10.1176/appi.ps.201800063.
    1. Noone C, Hogan MJ. A randomised active-controlled trial to examine the effects of an online mindfulness intervention on executive control, critical thinking and key thinking dispositions in a university student sample. BMC Psychol. 2018 Apr 05;6(1):13. doi: 10.1186/s40359-018-0226-3.
    1. Hoermann S, McCabe KL, Milne DN, Calvo RA. Application of synchronous text-based dialogue systems in mental health interventions: systematic review. J Med Internet Res. 2017 Jul 21;19(8):e267. doi: 10.2196/jmir.7023.
    1. Fitzpatrick KK, Darcy A, Vierhile M. Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (Woebot): a randomized controlled trial. JMIR Ment Health. 2017 Jun 06;4(2):e19. doi: 10.2196/mental.7785.
    1. Ly KH, Ly A, Andersson G. A fully automated conversational agent for promoting mental well-being: a pilot RCT using mixed methods. Internet Interv. 2017 Dec;10:39–46. doi: 10.1016/j.invent.2017.10.002.
    1. BBC Tomorrow's world BBC. 2017. [2018-09-04]. Would you trust a chatbot therapist? .
    1. Emma S. Wysa. Wysa Case Studies: 2018. Apr 18, [2018-09-04]. NHS Children Services: How the North East London NHS Foundation Trust uses Wysa for children's mental health .
    1. Braun A, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006 Jan;3(2):77–101. doi: 10.1191/1478088706qp063oa.
    1. Braun V, Clarke V. What can “thematic analysis” offer health and wellbeing researchers? Int J Qual Stud Health Well-being. 2014;9:26152. doi: 10.3402/qhw.v9.26152.
    1. Mitchell AJ, Yadegarfar M, Gill J, Stubbs B. Case finding and screening clinical utility of the Patient Health Questionnaire (PHQ-9 and PHQ-2) for depression in primary care: a diagnostic meta-analysis of 40 studies. BJPsych Open. 2016 Mar;2(2):127–38. doi: 10.1192/bjpo.bp.115.001685.
    1. Ruscio J. A probability-based measure of effect size: robustness to base rates and other factors. Psychol Methods. 2008 Mar;13(1):19–30. doi: 10.1037/1082-989X.13.1.19.
    1. Rice ME, Harris GT. Comparing effect sizes in follow-up studies: ROC Area, Cohen's d, and r. Law Hum Behav. 2005 Oct;29(5):615–20. doi: 10.1007/s10979-005-6832-7.
    1. Barnett AG, van der Pols JC, Dobson AJ. Regression to the mean: what it is and how to deal with it. Int J Epidemiol. 2005 Feb;34(1):215–20. doi: 10.1093/ije/dyh299.
    1. Zimmerman DW. Invalidation of parametric and nonparametric statistical tests by concurrent violation of two assumptions. J Exp Educ. 1998 Jan;67(1):55–68. doi: 10.1080/00220979809598344.
    1. Wagnild GM, Young HM. Development and psychometric evaluation of the Resilience Scale. J Nurs Meas. 1993;1(2):165–78.
    1. McAlpine DD, McCreedy E, Alang S. The meaning and predictive value of self-rated mental health among persons with a mental health problem. J Health Soc Behav. 2018 Jun;59(2):200–14. doi: 10.1177/0022146518755485.
    1. Kramer J, Conijn B, Oijevaar P, Riper H. Effectiveness of a web-based solution-focused brief chat treatment for depressed adolescents and young adults: randomized controlled trial. J Med Internet Res. 2014;16(5):e141. doi: 10.2196/jmir.3261.
    1. Kessler D, Lewis G, Kaur S, Wiles N, King M, Weich S, Sharp DJ, Araya R, Hollinghurst S, Peters TJ. Therapist-delivered internet psychotherapy for depression in primary care: a randomised controlled trial. Lancet. 2009 Aug 22;374(9690):628–34. doi: 10.1016/S0140-6736(09)61257-5.

Source: PubMed

Sponsoren und Mitarbeiter

Krankheiten

Drogeninterventionen

3
Abonnieren