Efficacy, Usability, and Acceptability of a Chatbot for Promoting COVID-19 Vaccination in Unvaccinated or Booster-Hesitant Young Adults: Pre-Post Pilot Study

Tzu Tsun Luk, Judy Hiu Tung Lui, Man Ping Wang, Tzu Tsun Luk, Judy Hiu Tung Lui, Man Ping Wang

Abstract

Background: COVID-19 vaccines are highly effective in preventing severe disease and death but are underused. Interventions to address COVID-19 vaccine hesitancy are paramount to reducing the burden of COVID-19.

Objective: We aimed to evaluate the preliminary efficacy, usability, and acceptability of a chatbot for promoting COVID-19 vaccination and examine the factors associated with COVID-19 vaccine hesitancy.

Methods: In November 2021, we conducted a pre-post pilot study to evaluate "Vac Chat, Fact Check," a web-based chatbot for promoting COVID-19 vaccination. We conducted a web-based survey (N=290) on COVID-19 vaccination at a university in Hong Kong. A subset of 46 participants who were either unvaccinated (n=22) or were vaccinated but hesitant to receive boosters (n=24) were selected and given access to the chatbot for a 7-day trial period. The chatbot provided information about COVID-19 vaccination (eg, efficacy and common side effects), debunked common myths about the vaccine, and included a decision aid for selecting vaccine platforms (inactivated and mRNA vaccines). The main efficacy outcome was changes in the COVID-19 Vaccine Hesitancy Scale (VHS) score (range 9-45) from preintervention (web-based survey) to postintervention (immediately posttrial). Other efficacy outcomes included changes in intention to vaccinate or receive boosters and willingness to encourage others to vaccinate on a scale from 1 (not at all) to 5 (very). Usability was assessed by the System Usability Scale (range 0-100). Linear regression was used to examine the factors associated with COVID-19 VHS scores in all survey respondents.

Results: The mean (SD) age of all survey respondents was 21.4 (6.3) years, and 61% (177/290) of respondents were female. Higher eHealth literacy (B=-0.26; P<.001) and perceived danger of COVID-19 (B=-0.17; P=.009) were associated with lower COVID-19 vaccine hesitancy, adjusting for age, sex, chronic disease status, previous flu vaccination, and perceived susceptibility to COVID-19. The main efficacy outcome of COVID-19 VHS score significantly decreased from 28.6 (preintervention) to 24.5 (postintervention), with a mean difference of -4.2 (P<.001) and an effect size (Cohen d) of 0.94. The intention to vaccinate increased from 3.0 to 3.9 (P<.001) in unvaccinated participants, whereas the intention to receive boosters increased from 1.9 to 2.8 (P<.001) in booster-hesitant participants. Willingness to encourage others to vaccinate increased from 2.7 to 3.0 (P=.04). At postintervention, the median (IQR) System Usability Scale score was 72.5 (65-77.5), whereas the median (IQR) recommendation score was 7 (6-8) on a scale from 0 to 10. In a post hoc 4-month follow-up, 82% (18/22) of initially unvaccinated participants reported having received the COVID-19 vaccine, whereas 29% (7/24) of booster-hesitant participants received boosters.

Conclusions: This pilot study provided initial evidence to support the efficacy, usability, and acceptability of a chatbot for promoting COVID-19 vaccination in young adults who were unvaccinated or booster-hesitant.

Keywords: COVID-19; Chinese; booster; booster hesitancy; chatbot; chatbot usability; conversational agent; coronavirus; health intervention; health promotion; immunization; pandemic; students; university students; vaccine; vaccine hesitancy; virtual assistant; web-based survey; young adult; youth.

Conflict of interest statement

Conflicts of Interest: None declared.

©Tzu Tsun Luk, Judy Hiu Tung Lui, Man Ping Wang. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 04.10.2022.

Figures

Figure 1
Figure 1
Study flow diagram.
Figure 2
Figure 2
Screenshot of “Vac Chat, Fact Check” showing chatbot navigation by menu options.
Figure 3
Figure 3
Screenshot of “Vac Chat, Fact Check” showing chatbot navigation by keyword.

References

    1. Ritchie H, Mathieu E, Rodés-Guirao L, Appel C, Giattino C, Ortiz-Ospina E. Coronavirus (COVID-19) vaccinations. Our World in Data. [2022-06-30]. .
    1. Rzymski P, Poniedziałek B, Fal A. Willingness to receive the booster COVID-19 vaccine dose in Poland. Vaccines (Basel) 2021 Nov 05;9(11):1286. doi: 10.3390/vaccines9111286. vaccines9111286
    1. Pal S, Shekhar R, Kottewar S, Upadhyay S, Singh M, Pathak D, Kapuria D, Barrett E, Sheikh AB. COVID-19 vaccine hesitancy and attitude toward booster doses among US healthcare workers. Vaccines (Basel) 2021 Nov 19;9(11):1358. doi: 10.3390/vaccines9111358. vaccines9111358
    1. Paul E, Fancourt D. Predictors of uncertainty and unwillingness to receive the COVID-19 booster vaccine: an observational study of 22,139 fully vaccinated adults in the UK. Lancet Reg Health Eur. 2022 Mar;14:100317. doi: 10.1016/j.lanepe.2022.100317. S2666-7762(22)00010-2
    1. Ten threats to global health in 2019. World Health Organization. 2019. [2022-03-11]. .
    1. Robinson E, Jones A, Lesser I, Daly M. International estimates of intended uptake and refusal of COVID-19 vaccines: a rapid systematic review and meta-analysis of large nationally representative samples. Vaccine. 2021 Apr 08;39(15):2024–2034. doi: 10.1016/j.vaccine.2021.02.005. S0264-410X(21)00140-7
    1. Cascini F, Pantovic A, Al-Ajlouni Y, Failla G, Ricciardi W. Attitudes, acceptance and hesitancy among the general population worldwide to receive the COVID-19 vaccines and their contributing factors: a systematic review. EClinicalMedicine. 2021 Oct;40:101113. doi: 10.1016/j.eclinm.2021.101113. S2589-5370(21)00393-X
    1. Loomba S, de Figueiredo A, Piatek SJ, de Graaf K, Larson HJ. Measuring the impact of COVID-19 vaccine misinformation on vaccination intent in the UK and USA. Nat Hum Behav. 2021 Mar;5(3):337–348. doi: 10.1038/s41562-021-01056-1.10.1038/s41562-021-01056-1
    1. Abbasi J. Widespread misinformation about infertility continues to create COVID-19 vaccine hesitancy. JAMA. 2022 Mar 15;327(11):1013–1015. doi: 10.1001/jama.2022.2404.2789477
    1. Roozenbeek J, Schneider CR, Dryhurst S, Kerr J, Freeman ALJ, Recchia G, van der Bles AM, van der Linden S. Susceptibility to misinformation about COVID-19 around the world. R Soc Open Sci. 2020 Oct;7(10):201199. doi: 10.1098/rsos.201199. rsos201199
    1. Almalki M, Azeez F. Health chatbots for fighting COVID-19: a scoping review. Acta Inform Med. 2020 Dec;28(4):241–247. doi: 10.5455/aim.2020.28.241-247. AIM-28-241
    1. Amiri P, Karahanna E. Chatbot use cases in the COVID-19 public health response. J Am Med Inform Assoc. 2022 Apr 13;29(5):1000–1010. doi: 10.1093/jamia/ocac014. 6523926
    1. Herriman M, Meer E, Rosin R, Lee V, Washington V, Volpp KG. Asked and answered: building a chatbot to address COVID-19-related concerns. NEJM Catal Innov Care Deliv. 2020 Jun 18; doi: 10.1056/CAT.20.0230.
    1. Siedlikowski S, Noël LP, Moynihan SA, Robin M. Chloe for COVID-19: evolution of an intelligent conversational agent to address infodemic management needs during the COVID-19 pandemic. J Med Internet Res. 2021 Sep 21;23(9):e27283. doi: 10.2196/27283. v23i9e27283
    1. WHO Health Alert brings COVID-19 facts to billions via WhatsApp. World Health Organization. 2021. Apr 26, [2022-03-11]. .
    1. Luk TT, Zhao S, Wu Y, Wong JY, Wang MP, Lam TH. Prevalence and determinants of SARS-CoV-2 vaccine hesitancy in Hong Kong: a population-based survey. Vaccine. 2021 Jun 16;39(27):3602–3607. doi: 10.1016/j.vaccine.2021.05.036. S0264-410X(21)00608-3
    1. Guo Z, Zhao SZ, Guo N, Wu Y, Weng X, Wong JY, Lam TH, Wang MP. Socioeconomic disparities in eHealth literacy and preventive behaviors during the COVID-19 pandemic in Hong Kong: cross-sectional study. J Med Internet Res. 2021 Apr 14;23(4):e24577. doi: 10.2196/24577. v23i4e24577
    1. Amith M, Zhu A, Cunningham R, Lin R, Savas L, Shay L, Chen Y, Gong Y, Boom J, Roberts K, Tao C. Early usability assessment of a conversational agent for HPV vaccination. Stud Health Technol Inform. 2019;257:17–23.
    1. MacDonald NE, SAGE Working Group on Vaccine Hesitancy Vaccine hesitancy: definition, scope and determinants. Vaccine. 2015 Aug 14;33(34):4161–4. doi: 10.1016/j.vaccine.2015.04.036. S0264-410X(15)00500-9
    1. Lazarus JV, Ratzan SC, Palayew A, Gostin LO, Larson HJ, Rabin K, Kimball S, El-Mohandes A. A global survey of potential acceptance of a COVID-19 vaccine. Nat Med. 2021 Feb;27(2):225–228. doi: 10.1038/s41591-020-1124-9. 10.1038/s41591-020-1124-9
    1. Freeman D, Loe BS, Yu LM, Freeman J, Chadwick A, Vaccari C, Shanyinde M, Harris V, Waite F, Rosebrock L, Petit A, Vanderslott S, Lewandowsky S, Larkin M, Innocenti S, Pollard AJ, McShane H, Lambe S. Effects of different types of written vaccination information on COVID-19 vaccine hesitancy in the UK (OCEANS-III): a single-blind, parallel-group, randomised controlled trial. Lancet Public Health. 2021 Jun;6(6):e416–e427. doi: 10.1016/S2468-2667(21)00096-7. S2468-2667(21)00096-7
    1. Shapiro GK, Tatar O, Dube E, Amsel R, Knauper B, Naz A, Perez S, Rosberger Z. The vaccine hesitancy scale: psychometric properties and validation. Vaccine. 2018 Jan 29;36(5):660–667. doi: 10.1016/j.vaccine.2017.12.043.S0264-410X(17)31796-6
    1. Larson Heidi J, Jarrett Caitlin, Schulz William S, Chaudhuri Mohuya, Zhou Yuqing, Dube Eve, Schuster Melanie, MacDonald Noni E, Wilson Rose, SAGE Working Group on Vaccine Hesitancy Measuring vaccine hesitancy: the development of a survey tool. Vaccine. 2015 Aug 14;33(34):4165–75. doi: 10.1016/j.vaccine.2015.04.037. S0264-410X(15)00501-0
    1. Bland JM, Altman DG. Cronbach's alpha. BMJ. 1997 Mar 22;314(7080):572. doi: 10.1136/bmj.314.7080.572.
    1. Sedgwick P. Spearman's rank correlation coefficient. BMJ. 2014 Nov 28;349:g7327. doi: 10.1136/bmj.g7327.
    1. Brooke J. SUS: a retrospective. J Usability Stud. 2013 Feb;8(2):29–40.
    1. Norman CD, Skinner HA. eHEALS: the eHealth Literacy Scale. J Med Internet Res. 2006 Nov 14;8(4):e27. doi: 10.2196/jmir.8.4.e27. v8i4e27
    1. Wong DK, Cheung M. Online health information seeking and eHealth literacy among patients attending a primary care clinic in Hong Kong: a cross-sectional survey. J Med Internet Res. 2019 Mar 27;21(3):e10831. doi: 10.2196/10831. v21i3e10831
    1. Cohen J. Statistical Power Analysis for the Behavioral Sciences. New York, NY: Routledge Academic; 1988.
    1. Bangor A, Kortum P, Miller J. Determining what individual SUS scores mean: adding an adjective rating scale. J Usability Stud. 2009 May;4(3):114–123.
    1. Krol MW, de Boer Dolf, Delnoij DM, Rademakers JJDJM. The Net Promoter Score--an asset to patient experience surveys? Health Expect. 2015 Dec;18(6):3099–109. doi: 10.1111/hex.12297. doi: 10.1111/hex.12297.
    1. Altay S, Hacquin AS, Chevallier C, Mercier H. Information delivered by a chatbot has a positive impact on COVID-19 vaccines attitudes and intentions. J Exp Psychol Appl. 2021 Oct 28; doi: 10.1037/xap0000400.2021-99618-001
    1. Kobayashi T, Nishina Y, Tomoi H, Harada K, Tanaka K, Matsumoto E. Corowa-kun: impact of a COVID-19 vaccine information chatbot on vaccine hesitancy, Japan 2021. medRxiv. Preprint posted online on May 30, 2021. 2022 doi: 10.1101/2021.05.26.21257854.
    1. Statistics on 5th wave of COVID-19 (from 31 Dec 2021 up till 30 Mar 2022 00:00) The Government of Hong Kong Special Administrative Region. [2022-04-15]. .
    1. An L, Bacon E, Hawley S, Yang P, Russell D, Huffman S, Resnicow K. Relationship between coronavirus-related eHealth literacy and COVID-19 knowledge, attitudes, and practices among US adults: web-based survey study. J Med Internet Res. 2021 Mar 29;23(3):e25042. doi: 10.2196/25042. v23i3e25042

Source: PubMed

Sponsorer och medarbetare

Medicinska tillstånd

Läkemedelsinterventioner

3
Prenumerera