Evaluation of Chatbot for Mental Well-being

Evaluation of Chatbot for Mental Well-being: A Randomized Controlled Trial

The present study consists of two two-armed randomized controlled trials between experimental and waitlist control groups. It aims to evaluate the effectiveness of conversational chatbot in improving mental health literacy, uptake of self-care behaviors, and mental well-being, compared to the waitlist control, and the effectiveness of daily notification on adherence. This study will provide important findings for the future development and implementation of chatbots in mental health, which may increase public access to immediate mental health support. It is hypothesized that participants in the experimental condition will show (H1) better mental health literacy (H2) better improvement in self-care and self-efficacy in mental well-being, and (H3) better mental well-being, compared with participants in the control condition. Also, it is hypothesized that participants with daily reminders will show (H4) a better adherence rate in using chatbot compared with participants without daily reminders

Study Overview

• Status: Completed
• Conditions: Mental Well-being; Mental Health Literacy
• Intervention / Treatment: Other: Mental Well-being Chatbot

Detailed Description

Background

In Hong Kong, insufficient resources in the current public health system cause a long waiting time. Mental health services provided by the public health system mainly rely on traditional one-to-one face-to-face sessions. In the past 12 months, there were 48,520 new bookings in public psychiatry outpatient clinics and the longest waiting time was 94 weeks. Priority is always given to people with more severe mental health issues, which causes long waiting time for people with mild mental health symptoms. Untreated mental health issues can be escalated to more severe symptoms. Thus, in addition to treating mental illness, preventing common mental health issues and fostering mental health self-care in the general population are crucial to promote public mental health and reduce illness burden in the society.

The Hong Kong Mental Morbidity Study found 1 in 7 individuals in Hong Kong has either depression, anxiety, or a mix of the two disorders; however, only a quarter of them sought professional help. Rather than resorting to mental health professionals for face-to-face service to treat common mental health concerns, digital technology provides a highly scalable and accessible means through which individuals can access mental health resources for self-care. Among these tools, conversational agent is one of the viable options. It has been applied in health care industries to cater to different health needs, including providing timely information and supporting mental health disorders. Healthcare conversational agents were found to be effective in reducing depression and anxiety symptoms and had higher engagement rate compared with standard industry metrics. Chatbot is a type of conversational agent. It is a rule-based computer algorithm that conducts an automatic conversation with people based on predefined instructions. Based on a self-guided approach, users can search for topics that they are interested in and engage with pre-designed computer algorithms at the convenience of their own space and time without the constraints of specialized care. Applying chatbots in mental health self-care provides an opportunity for individuals to directly learn about relevant mental health-related knowledge and tips as well as practice self-care exercises at anytime, anywhere.

The Present Study

The present study aims to evaluate the effectiveness of conversational chatbot in improving mental health literacy, uptake of self-care behaviors, and mental well-being, compared to the waitlist control, and the effectiveness of daily notification on adherence. This study will provide important findings for the future development and implementation of chatbots in mental health, which may increase public access to immediate mental health support. It is hypothesized that participants in the experimental condition will show (H1) better mental health literacy (H2) better improvement in self-care and self-efficacy in mental well-being, and (H3) better mental well-being, compared with participants in the control condition. Also, it is hypothesized that participants with daily reminders will show (H4) a better adherence rate in using chatbot compared with participants without daily reminders

Participants

Participants will be recruited through (1) advertising at popular online networking platforms (e.g., Facebook and Instagram), mass mailing at investigator's institutions, and snowball sampling.

Procedure Upon completing the screening and pre-evaluation questionnaire, participants will be randomly assigned to the experimental group or waitlist control group, with and without notifications based on computer-generated random digits. They will complete 2 more sets of questionnaires, including a post-evaluation 11 days after group allocation, and a follow-up questionnaire 21 days after group allocation. In the experimental group, participants will go through one assigned chatbot each day for 10 days, with the sequence of the assigned chatbot randomized. The chatbot contents are developed by clinical psychologists and well-being promotion officers. Content includes relationships, stress, value, emotion, and positive psychology. Each chatbot can only be assessed on the day of distribution. The access link will expire upon completion and the day after distribution to prevent repeat and delay in completion. Experimental group participants can freely access all chatbots after the completion of post-evaluation and before the follow-up questionnaire is sent to them. In the waitlist control group, participants are to refrain from using the chatbot until they finished the follow-up questionnaire. All participants will be able to access the chatbot materials in an online platform after they have completed the research.

• Study Type: Interventional
• Enrollment (Actual): 293
• Phase: Not Applicable

Contacts and Locations

Study Locations

• Hong Kong
  • N.t.
    • Sha Tin, N.t., Hong Kong
      • Diversity and Well-being Lab, CUHK

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: 18 years and older (Adult, Older Adult)
• Accepts Healthy Volunteers: No

Description

Inclusion Criteria:

• Adults aged 18 years old or above
• Able to read and understand Chinese and spoken Cantonese
• Have access to the Internet

Exclusion Criteria:

• Under 18 years old
• Unable to read and understand Chinese and spoken Cantonese
• Unable to access the internet
• Existing users of the Jockey Club TourHeart+ Project and people who have participated in related research projects will be excluded from this study, as chatbots in this study were extracted from this online platform.

Study Plan

How is the study designed?

Design Details

• Primary Purpose: Other
• Allocation: Randomized
• Interventional Model: Parallel Assignment
• Masking: None (Open Label)

Number of Arms

2

Arms and Interventions

Participant Group / Arm	Intervention / Treatment
Experimental: Experimental group; In the experimental group, participants will go through one assigned chatbot each day for 10 days, with the sequence of the assigned chatbot randomized. One group of participants will receive chatbot notifications for 10 days, another group of participants will not receive chatbot notifications. They can freely access all chatbots after the completion of post-evaluation. After completing the pre-evaluation questionnaire, they will complete 2 more sets of questionnaires, including a post-evaluation 11 days after group allocation, and a follow-up questionnaire 21 days after group allocation. All participants will be able to access the chatbot materials in an online platform after they have completed the research.	Other: Mental Well-being Chatbot; The chatbot contents are developed by Clinical psychologists and well-being promotion officers. Content includes relationships, stress, value, emotion, and positive psychology.
No Intervention: Waitlist control group; In the waitlist control group, participants are to refrain from using the chatbot until they finished the follow-up questionnaire. One group of participants will receive study notifications for 10 days, another group of participants will not receive study notifications. After completing the pre-evaluation questionnaire, they will complete 2 more sets of questionnaires, including a post-evaluation 11 days after group allocation, and a follow-up questionnaire 21 days after group allocation. All participants will be able to access the chatbot materials in an online platform after they have completed the research.

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Self-Care Self-efficacy - Strategies Used by People to Promote Health (SUPPH)	The scale is reliable, with a Cronbach's alpha of internal consistency of 0.93 . It includes 29 items to measure self-care self-efficacy using a 6-point scale from 1 (very little confidence) to 5 (quite a lot of confidence). High scores reflect better self-care self-efficacy. The SUPPH includes three subscales, stress reduction, decision-making, and positive attitudes. This study only includes items on stress reduction and positive attitudes. Cronbach's alpha of stress reduction and positive attitudes were 0.92 and 0.92 respectively.	Day 10
Self-Care Self-efficacy - Strategies Used by People to Promote Health (SUPPH)	The scale is reliable, with a Cronbach's alpha of internal consistency of 0.93 . It includes 29 items to measure self-care self-efficacy using a 6-point scale from 1 (very little confidence) to 5 (quite a lot of confidence). High scores reflect better self-care self-efficacy. The SUPPH includes three subscales, stress reduction, decision-making, and positive attitudes. This study only includes items on stress reduction and positive attitudes. Cronbach's alpha of stress reduction and positive attitudes were 0.92 and 0.92 respectively.	Day 20
Self-Care - Self-Care Behavior Inventory	It includes 19 items to measure self-care behavior using a 5-point scale from 1 (very little) to 5 (quite a lot). Internal consistency if the scale was 0.83. One item regarding medication is removed, while two items related to time spent on things that respondents enjoy and feel interested in and time spent alone were added.	Day 10
Self-Care - Self-Care Behavior Inventory	It includes 19 items to measure self-care behavior using a 5-point scale from 1 (very little) to 5 (quite a lot). Internal consistency if the scale was 0.83. One item regarding medication is removed, while two items related to time spent on things that respondents enjoy and feel interested in and time spent alone were added.	Day 20
Mental Health Literacy	Sixteen items were developed to measure the knowledge of various aspects to do with mental health. Items are rated on a 7-points scale from 1 (strongly disagree) to 7 (strongly agree). Items are designed according to the content of chatbots and a well-developed Mental Health Literacy Scale.	Day 10
Mental Health Literacy	Sixteen items were developed to measure the knowledge of various aspects to do with mental health. Items are rated on a 7-points scale from 1 (strongly disagree) to 7 (strongly agree). Items are designed according to the content of chatbots and a well-developed Mental Health Literacy Scale.	Day 20

Secondary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Depressive symptoms - Patient Health Questionnaire (PHQ-9)	Patient Health Questionnaire. It includes 9 items to assess the extent to which respondents are bothered by depression-related symptoms using a 4-point scale from 0 (not at all) to 3 (nearly every day). PHQ-9 has been validated and used widely in the general population for screening and measuring depression severity. Scores of 5, 10, 15, and 20 denote mild, moderate, moderately severe, and severe levels of depression respectively (range: 0-27). PHQ-9 has a sensitivity of 0.88 and a specificity of 0.88 in detecting major depressive disorder (MDD) at a cut-off of 10. The internal consistency reliability of the Chinese version of the scale was 0.86.	Day 10 and 20
Anxiety symptoms - Generalized Anxiety Disorder Assessment (GAD-7)	It is a 7-item scale to assess the extent to which respondents are bothered by anxiety-related symptoms using a 4-point scale from 0 (not at all) to 3 (nearly every day). Scores of 5, 10, and 15 denote the mild, moderate, and severe levels of anxiety respectively (range: 0-21). At a cut-off of 10, GAD-7 has sensitivity of 0.89 and specificity of 0.82 in detecting generalized anxiety disorder (GAD). The internal consistency of the Chinese version was 0.93.	Day 10 and 20
Mindfulness - Mindful Awareness Attention Scale (MASS)	It is a 15-item scale to measure individual differences in mindful states during the previous week using a 6-point scale from 1 (almost always) to 6 (almost never). Higher scores reflect higher levels of dispositional mindfulness. The internal consistency among general adults was 0.87.	Day 10 and 20
Well-being - PERMA-Profiler (PERMA)	It includes 23 items to assess respondents' well-being using an 11-point scale from 0 (never/terrible/not at all) to 10 (always/excellent/completely). It includes domains in positive emotion, engagement, relationship, meaning, accomplishment, negative emotion, physical health, loneliness, and overall well-being. PERMA is reliable (Cronbach's alpha = 0.94) and has been validated in different populations such as Australian, Italian, Malaysian, and Turkish.	Day 10 and 20
Behavioral Intention - a subscale in the E-therapy Attitude and Process Questionnaire (eTAP)	It includes 3 items to measure participants' intention in using online psychological intervention on a 7-point scale from 1 (strongly disagree) to 7 (strongly agree). The subscale showed excellent reliability (Cronbach's alpha = 0.94).	Day 10 and 20
Attitude towards chatbot	Nine items were developed to measure attitudes toward the ability of chatbots in improving mental well-being. It is measured by a 7-points scale from 1 (strongly disagree) to 7 (strongly agree).	Day 10 and 20
Credibility and Expectancy. The Credibility and Expectancy Questionnaire (CEQ)	It measures treatment expectancy and rationale credibility for use in clinical outcomes. Six items are rated from 0% to 100%. The first section of the questionnaire is about thoughts toward online self-care intervention and the second part is about feelings toward online self-care intervention. The internal consistency of the scale was 0.81.	Day 10 and 20
Usability	Twenty-four items were developed to measure the usability of the chatbot, including satisfaction, engagement, functions, favorite elements, and useful topics of chatbots.	Day 10 and 20
Usage	Chatbot usage, including time spent on chatbots, number of conversations engaged with the chatbot, completion of exercises, behavioral change, attitudinal shifts, condition improvements, and the level of knowledge transfer, will be extracted from the back-end system.	Day 10 and 20

Collaborators and Investigators

• Sponsor: Chinese University of Hong Kong
• Investigators: Principal Investigator: Winnie WS Mak, PROFESSOR

Publications and helpful links

General Publications

• Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. 2001 Sep;16(9):606-13. doi: 10.1046/j.1525-1497.2001.016009606.x.
• Spitzer RL, Kroenke K, Williams JB, Lowe B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med. 2006 May 22;166(10):1092-7. doi: 10.1001/archinte.166.10.1092.
• Devilly GJ, Borkovec TD. Psychometric properties of the credibility/expectancy questionnaire. J Behav Ther Exp Psychiatry. 2000 Jun;31(2):73-86. doi: 10.1016/s0005-7916(00)00012-4.
• Brown KW, Ryan RM. The benefits of being present: mindfulness and its role in psychological well-being. J Pers Soc Psychol. 2003 Apr;84(4):822-48. doi: 10.1037/0022-3514.84.4.822.
• Fitzpatrick KK, Darcy A, Vierhile M. Delivering Cognitive Behavior Therapy to Young Adults With Symptoms of Depression and Anxiety Using a Fully Automated Conversational Agent (Woebot): A Randomized Controlled Trial. JMIR Ment Health. 2017 Jun 6;4(2):e19. doi: 10.2196/mental.7785.
• Wang W, Bian Q, Zhao Y, Li X, Wang W, Du J, Zhang G, Zhou Q, Zhao M. Reliability and validity of the Chinese version of the Patient Health Questionnaire (PHQ-9) in the general population. Gen Hosp Psychiatry. 2014 Sep-Oct;36(5):539-44. doi: 10.1016/j.genhosppsych.2014.05.021. Epub 2014 Jun 6.
• Lev EL, Owen SV. A measure of self-care self-efficacy. Res Nurs Health. 1996 Oct;19(5):421-9. doi: 10.1002/(SICI)1098-240X(199610)19:53.0.CO;2-S.
• Ayse, E. B. (2018). Adaptation of the PERMA Well-Being Scale into Turkish: Validity and reliability Studies. Educational Research and Reviews, 13(4), 129-35.
• Butler, J., & Kern, M. L. (2016). The PERMA-Profiler: A brief multidimensional measure of flourishing. International Journal of Wellbeing, 6(3), 1-48.
• Clough BA, Eigeland JA, Madden IR, Rowland D, Casey LM. Development of the eTAP: A brief measure of attitudes and process in e-interventions for mental health. Internet Interv. 2019 Jun 18;18:100256. doi: 10.1016/j.invent.2019.100256. eCollection 2019 Dec.
• Giangrasso, B. (2018). Psychometric properties of the PERMA-Profiler as hedonic and eudaimonic well-being measure in an Italian context. Current Psychology, 40, 1-10.
• Hospital Authority. (n.d.). Waiting time for new case booking at Psychiatry specialist out-patient clinics. Retrieved December 20, 2022, from https://www.ha.org.hk/visitor/sopc_waiting_time.asp?id=7&lang=ENG
• Iasiello, M., Bartholomaeus, J., Jarden, A., & Kelly, G. (2017). Measuring PERMA+ in South Australia, the State of Wellbeing: A comparison with national and international norms. Journal of Positive Psychology and Wellbeing, 1(2), 53-72.
• Khaw, D., & Kern, M. (2014). A cross-cultural comparison of the PERMA model of well-being. Undergraduate Journal of Psychology at Berkeley, University of California, 8, 10-23.
• Martinengo L, Jabir AI, Goh WWT, Lo NYW, Ho MR, Kowatsch T, Atun R, Michie S, Tudor Car L. Conversational Agents in Health Care: Scoping Review of Their Behavior Change Techniques and Underpinning Theory. J Med Internet Res. 2022 Oct 3;24(10):e39243. doi: 10.2196/39243.
• Meheli S, Sinha C, Kadaba M. Understanding People With Chronic Pain Who Use a Cognitive Behavioral Therapy-Based Artificial Intelligence Mental Health App (Wysa): Mixed Methods Retrospective Observational Study. JMIR Hum Factors. 2022 Apr 27;9(2):e35671. doi: 10.2196/35671.
• O'Connor M, Casey L. The Mental Health Literacy Scale (MHLS): A new scale-based measure of mental health literacy. Psychiatry Res. 2015 Sep 30;229(1-2):511-6. doi: 10.1016/j.psychres.2015.05.064. Epub 2015 Jul 16.
• Ryan J, Curtis R, Olds T, Edney S, Vandelanotte C, Plotnikoff R, Maher C. Psychometric properties of the PERMA Profiler for measuring wellbeing in Australian adults. PLoS One. 2019 Dec 23;14(12):e0225932. doi: 10.1371/journal.pone.0225932. eCollection 2019.
• Santana, M. C., & Fouad, N. A. (2017). Development and validation of a Self-Care Behavior Inventory. Training and Education in Professional Psychology, 11(3), 140-145. https://doi.org/10.1037/tep0000142
• Shih YC, Chou CC, Lu YJ, Yu HY. Reliability and validity of the traditional Chinese version of the GAD-7 in Taiwanese patients with epilepsy. J Formos Med Assoc. 2022 Nov;121(11):2324-2330. doi: 10.1016/j.jfma.2022.04.018. Epub 2022 May 16.
• Sinha C, Cheng AL, Kadaba M. Adherence and Engagement With a Cognitive Behavioral Therapy-Based Conversational Agent (Wysa for Chronic Pain) Among Adults With Chronic Pain: Survival Analysis. JMIR Form Res. 2022 May 23;6(5):e37302. doi: 10.2196/37302.
• Lam LC, Wong CS, Wang MJ, Chan WC, Chen EY, Ng RM, Hung SF, Cheung EF, Sham PC, Chiu HF, Lam M, Chang WC, Lee EH, Chiang TP, Lau JT, van Os J, Lewis G, Bebbington P. Prevalence, psychosocial correlates and service utilization of depressive and anxiety disorders in Hong Kong: the Hong Kong Mental Morbidity Survey (HKMMS). Soc Psychiatry Psychiatr Epidemiol. 2015 Sep;50(9):1379-88. doi: 10.1007/s00127-015-1014-5. Epub 2015 Feb 8.

Study record dates

Study Major Dates

• Study Start (Actual): February 1, 2023
• Primary Completion (Actual): June 30, 2023
• Study Completion (Actual): June 30, 2023

Study Registration Dates

• First Submitted: January 5, 2023
• First Submitted That Met QC Criteria: January 12, 2023
• First Posted (Actual): January 23, 2023

Study Record Updates

• Last Update Posted (Actual): September 1, 2023
• Last Update Submitted That Met QC Criteria: August 31, 2023
• Last Verified: August 1, 2023

More Information

Terms related to this study

• Other Study ID Numbers: SBRE-22-0383

Plan for Individual participant data (IPD)

• Plan to Share Individual Participant Data (IPD)?: NO

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No