Chatbot-Delivered COVID-19 Vaccine Communication Message Preferences of Young Adults and Public Health Workers in Urban American Communities: Qualitative Study

Rose Weeks, Lyra Cooper, Pooja Sangha, João Sedoc, Sydney White, Assaf Toledo, Shai Gretz, Dan Lahav, Nina Martin, Alexandra Michel, Jae Hyoung Lee, Noam Slonim, Naor Bar-Zeev, Rose Weeks, Lyra Cooper, Pooja Sangha, João Sedoc, Sydney White, Assaf Toledo, Shai Gretz, Dan Lahav, Nina Martin, Alexandra Michel, Jae Hyoung Lee, Noam Slonim, Naor Bar-Zeev

Abstract

Background: Automated conversational agents, or chatbots, have a role in reinforcing evidence-based guidance delivered through other media and offer an accessible, individually tailored channel for public engagement. In early-to-mid 2021, young adults and minority populations disproportionately affected by COVID-19 in the United States were more likely to be hesitant toward COVID-19 vaccines, citing concerns regarding vaccine safety and effectiveness. Successful chatbot communication requires purposive understanding of user needs.

Objective: We aimed to review the acceptability of messages to be delivered by a chatbot named VIRA from Johns Hopkins University. The study investigated which message styles were preferred by young, urban-dwelling Americans as well as public health workers, since we anticipated that the chatbot would be used by the latter as a job aid.

Methods: We conducted 4 web-based focus groups with 20 racially and ethnically diverse young adults aged 18-28 years and public health workers aged 25-61 years living in or near eastern-US cities. We tested 6 message styles, asking participants to select a preferred response style for a chatbot answering common questions about COVID-19 vaccines. We transcribed, coded, and categorized emerging themes within the discussions of message content, style, and framing.

Results: Participants preferred messages that began with an empathetic reflection of a user concern and concluded with a straightforward, fact-supported response. Most participants disapproved of moralistic or reasoning-based appeals to get vaccinated, although public health workers felt that such strong statements appealing to communal responsibility were warranted. Responses tested with humor and testimonials did not appeal to the participants.

Conclusions: To foster credibility, chatbots targeting young people with vaccine-related messaging should aim to build rapport with users by deploying empathic, reflective statements, followed by direct and comprehensive responses to user queries. Further studies are needed to inform the appropriate use of user-customized testimonials and humor in the context of chatbot communication.

Keywords: AI; COVID-19; artificial intelligence; chatbots; conversational agent; digital health; health communication; infodemic; infodemiology; misinformation; natural language processing; online health information; public health; social media; user need; vaccination; vaccine communication; vaccine hesitancy.

Conflict of interest statement

Conflicts of Interest: NB-Z received research grants from Johnson & Johnson and Merck for unrelated work outside the scope of this paper. All authors declare no other conflicts of interest.

©Rose Weeks, Lyra Cooper, Pooja Sangha, João Sedoc, Sydney White, Assaf Toledo, Shai Gretz, Dan Lahav, Nina Martin, Alexandra Michel, Jae Hyoung Lee, Noam Slonim, Naor Bar-Zeev. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 06.07.2022.

Figures

Figure 1
Figure 1
Message attributes supporting and hindering credibility with young focus group participants. Textual excerpts coded with both directness- and rapport-related variables (eg, each cell shows a textual passage double coded with a directness-related variable and rapport-related variable). P: participant.

References

    1. Ozawa S, Stack ML. Public trust and vaccine acceptance--international perspectives. Hum Vaccin Immunother. 2013 Aug;9(8):1774–8. doi: 10.4161/hv.24961. 24961
    1. Gross L. A broken trust: lessons from the vaccine--autism wars. PLoS Biol. 2009 May 26;7(5):e1000114. doi: 10.1371/journal.pbio.1000114.
    1. Worldwide measles deaths climb 50% from 2016 to 2019 claiming over 207 500 lives in 2019. World Health Organization. 2020. Nov 12, [2022-05-16]. .
    1. Ten threats to global health in 2019. World Health Organization. [2022-05-23]. .
    1. Mortality analyses. Johns Hopkins Coronavirus Resource Center. [2022-05-16]. .
    1. King WC, Rubinstein M, Reinhart A, Mejia R. Time trends, factors associated with, and reasons for COVID-19 vaccine hesitancy: a massive online survey of US adults from January-May 2021. PLoS One. 2021 Dec 21;16(12):e0260731. doi: 10.1371/journal.pone.0260731. PONE-D-21-27471
    1. Siegel M, Critchfield-Jain I, Boykin M, Owens A. Actual racial/ethnic disparities in COVID-19 mortality for the non-Hispanic Black compared to non-Hispanic White population in 35 US states and their association with structural racism. J Racial Ethn Health Disparities. 2022 Jun;9(3):886–898. doi: 10.1007/s40615-021-01028-1. 10.1007/s40615-021-01028-1
    1. Callaghan T, Moghtaderi A, Lueck JA, Hotez P, Strych U, Dor A, Fowler EF, Motta M. Correlates and disparities of intention to vaccinate against COVID-19. Soc Sci Med. 2021 Mar;272:113638. doi: 10.1016/j.socscimed.2020.113638. S0277-9536(20)30857-1
    1. Fisher KA, Bloomstone SJ, Walder J, Crawford S, Fouayzi H, Mazor KM. Attitudes toward a potential SARS-CoV-2 vaccine : a survey of U.S. adults. Ann Intern Med. 2020 Dec 15;173(12):964–973. doi: 10.7326/M20-3569.
    1. Ruiz JB, Bell RA. Predictors of intention to vaccinate against COVID-19: results of a nationwide survey. Vaccine. 2021 Feb 12;39(7):1080–1086. doi: 10.1016/j.vaccine.2021.01.010. S0264-410X(21)00014-1
    1. Khubchandani J, Macias Y. COVID-19 vaccination hesitancy in Hispanics and African-Americans: a review and recommendations for practice. Brain Behav Immun Health. 2021 Aug;15:100277. doi: 10.1016/j.bbih.2021.100277. S2666-3546(21)00080-6
    1. COVID data tracker. Centers for Disease Control and Prevention. [2022-02-27]. .
    1. Goldstein S, MacDonald NE, Guirguis S, SAGE Working Group on Vaccine Hesitancy Health communication and vaccine hesitancy. Vaccine. 2015 Aug 14;33(34):4212–4. doi: 10.1016/j.vaccine.2015.04.042. S0264-410X(15)00506-X
    1. Nowak GJ, Gellin BG, MacDonald NE, Butler R, SAGE Working Group on Vaccine Hesitancy Addressing vaccine hesitancy: the potential value of commercial and social marketing principles and practices. Vaccine. 2015 Aug 14;33(34):4204–11. doi: 10.1016/j.vaccine.2015.04.039. S0264-410X(15)00503-4
    1. Evans WD, French J. Demand creation for COVID-19 vaccination: overcoming vaccine hesitancy through social marketing. Vaccines (Basel) 2021 Apr 01;9(4):319. doi: 10.3390/vaccines9040319. vaccines9040319
    1. Butler R, MacDonald NE, SAGE Working Group on Vaccine Hesitancy Diagnosing the determinants of vaccine hesitancy in specific subgroups: the guide to Tailoring Immunization Programmes (TIP) Vaccine. 2015 Aug 14;33(34):4176–9. doi: 10.1016/j.vaccine.2015.04.038. S0264-410X(15)00502-2
    1. Vaccine misinformation management field guide. United Nations Children's Fund (UNICEF) 2020. Dec, [2021-02-02]. .
    1. Strully KW, Harrison TM, Pardo TA, Carleo-Evangelist J. Strategies to address COVID-19 vaccine hesitancy and mitigate health disparities in minority populations. Front Public Health. 2021 Apr 23;9:645268. doi: 10.3389/fpubh.2021.645268. doi: 10.3389/fpubh.2021.645268.
    1. Miner AS, Laranjo L, Kocaballi AB. Chatbots in the fight against the COVID-19 pandemic. NPJ Digit Med. 2020 May 04;3:65. doi: 10.1038/s41746-020-0280-0. doi: 10.1038/s41746-020-0280-0.280
    1. Amiri P, Karahanna E. Chatbot use cases in the Covid-19 public health response. J Am Med Inform Assoc. 2022 Apr 13;29(5):1000–1010. doi: 10.1093/jamia/ocac014. 6523926
    1. Bonnevie E, Lloyd TD, Rosenberg SD, Williams K, Goldbarg J, Smyser J. Layla’s Got You: developing a tailored contraception chatbot for Black and Hispanic young women. Health Educ J. 2021;80(4):413–424. doi: 10.1177/0017896920981122.
    1. Kowatsch T, Schachner T, Harperink S, Barata F, Dittler U, Xiao G, Stanger C, V Wangenheim Florian, Fleisch E, Oswald H, Möller Alexander. Conversational agents as mediating social actors in chronic disease management involving health care professionals, patients, and family members: multisite single-arm feasibility study. J Med Internet Res. 2021 Feb 17;23(2):e25060. doi: 10.2196/25060. v23i2e25060
    1. Pereira J, Díaz Óscar. Using health chatbots for behavior change: a mapping study. J Med Syst. 2019 Apr 04;43(5):135. doi: 10.1007/s10916-019-1237-1.10.1007/s10916-019-1237-1
    1. Wysa. [2021-02-16].
    1. Copper-Ind C. Aetna partners with well-being app to provide mental healthcare during pandemic. International Investment. 2020. May 19, [2021-02-16]. .
    1. Alkire (née Nasr) L, O'Connor GE, Myrden S, Köcher S. Patient experience in the digital age: an investigation into the effect of generational cohorts. Journal of Retailing and Consumer Services. 2020 Nov;57:102221. doi: 10.1016/j.jretconser.2020.102221.
    1. Yasgur BS. Millennials flock to telehealth, online research. WebMD. 2021. Apr 02, [2022-03-30]. .
    1. Altay S, Hacquin AS, Chevallier C, Mercier H. Information delivered by a chatbot has a positive impact on COVID-19 vaccines attitudes and intentions. J Exp Psychol Appl. doi: 10.1037/xap0000400. Preprint posted on October 28, 2021.2021-99618-001
    1. WHO Health Alert brings COVID-19 facts to billions via WhatsApp. World Health Organization. 2021. Apr 26, [2021-07-26]. .
    1. Almalki M, Azeez F. Health chatbots for fighting COVID-19: a scoping review. Acta Inform Med. 2020 Dec;28(4):241–247. doi: 10.5455/aim.2020.28.241-247. AIM-28-241
    1. Limaye RJ, Holroyd TA, Blunt M, Jamison AF, Sauer M, Weeks R, Wahl B, Christenson K, Smith C, Minchin J, Gellin B. Social media strategies to affect vaccine acceptance: a systematic literature review. Expert Rev Vaccines. 2021 Aug;20(8):959–973. doi: 10.1080/14760584.2021.1949292.
    1. Noar SM. A 10-year retrospective of research in health mass media campaigns: where do we go from here? J Health Commun. 2006;11(1):21–42. doi: 10.1080/10810730500461059.H6Q277205W1540U0
    1. National Academies of Sciences, Engineering, and Medicine . Communicating Science Effectively: A Research Agenda. Washington, DC: The National Academies Press; 2017.
    1. Renn O, Levine D. Credibility and trust in risk communication. In: Kasperson RE, Stallen PJM, editors. Communicating Risks to the Public. Dordrecht, Netherlands: Springer; 2010. pp. 175–217.
    1. Cole JW, Chen AMH, McGuire K, Berman S, Gardner J, Teegala Y. Motivational interviewing and vaccine acceptance in children: The MOTIVE study. Vaccine. 2022 Mar 15;40(12):1846–1854. doi: 10.1016/j.vaccine.2022.01.058.S0264-410X(22)00114-1
    1. Gagneur A, Lemaître Thomas, Gosselin V, Farrands A, Carrier N, Petit G, Valiquette L, De Wals P. A postpartum vaccination promotion intervention using motivational interviewing techniques improves short-term vaccine coverage: PromoVac study. BMC Public Health. 2018 Jun 28;18(1):811. doi: 10.1186/s12889-018-5724-y. 10.1186/s12889-018-5724-y
    1. Vira, the Vax Chatbot. [2022-03-29].
    1. About COVID-19 materials for tribes. Center for American Indian Health. [2022-03-05]. .
    1. Answers to all your questions about getting vaccinated for Covid-19. The New York Times. 2021. Oct 18, [2022-05-10]. .
    1. Twitter API. Twitter. [2022-05-10]. .
    1. Chen X, Hay JL, Waters EA, Kiviniemi MT, Biddle C, Schofield E, Li Y, Kaphingst K, Orom H. Health literacy and use and trust in health information. J Health Commun. 2018;23(8):724–734. doi: 10.1080/10810730.2018.1511658.
    1. Peters RG, Covello VT, McCallum DB. The determinants of trust and credibility in environmental risk communication: an empirical study. Risk Anal. 1997 Feb;17(1):43–54. doi: 10.1111/j.1539-6924.1997.tb00842.x.
    1. Bilu Y, Gera A, Hershcovich D, Sznajder B, Lahav D, Moshkowich G, Malet A, Gavron A, Slonim N. Argument invention from first principles. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics; July 28 to August 2, 2019; Florence, Italy. Proc 57th Annu Meet Assoc Comput Linguist Florence, Italy: Association for Computational Linguistics; 2019. Jul, pp. 1013–1026.
    1. Mack N, Woodsong C, MacQueen K, Guest G, Namey E. Qualitative Research Methods: A Data Collector's Field Guide. Research Triangle Park, NC: Family Health International; 2005.
    1. Report of the SAGE Working Group on vaccine hesitancy. World Health Organization. 2014. Nov 12, [2022-06-24]. .
    1. MacDonald NE, SAGE Working Group on Vaccine Hesitancy Vaccine hesitancy: definition, scope and determinants. Vaccine. 2015 Aug 14;33(34):4161–4. doi: 10.1016/j.vaccine.2015.04.036. S0264-410X(15)00500-9
    1. Video conferencing, cloud phone, webinars, chat, virtual events. Zoom. [2022-03-29].
    1. Temi. [2022-03-27].
    1. User guide. Dedoose. [2022-02-28]. .
    1. Corbin J, Strauss A. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. 4th ed. Thousand Oaks, CA: SAGE Publications; 2008.
    1. Charmaz K. Constructing Grounded Theory. 2nd ed. Thousand Oaks, CA: SAGE Publications; 2014.
    1. Hennink MM, Kaiser BN, Weber MB. What influences saturation? estimating sample sizes in focus group research. Qual Health Res. 2019 Aug;29(10):1483–1496. doi: 10.1177/1049732318821692.
    1. Vraga EK, Kim SC, Cook J. Testing logic-based and humor-based corrections for science, health, and political misinformation on social media. J Broadcast Electron Media. 2019 Sep 20;63(3):393–414. doi: 10.1080/08838151.2019.1653102.
    1. Kim J, Muhic J, Park S, Robert L. Trustworthy conversational agent design for African Americans with chronic conditions during COVID-19. Realizing AI in Healthcare: Challenges Appearing in the Wild; 2021 CHI Conference on Human Factors in Computing Systems (CHI 2021); May 8-13, 2021; Online Virtual Conference (originally Yokohama, Japan). 2020.
    1. Ta V, Griffith C, Boatfield C, Wang X, Civitello M, Bader H, DeCero E, Loggarakis A. User experiences of social support from companion chatbots in everyday contexts: thematic analysis. J Med Internet Res. 2020 Mar 06;22(3):e16235. doi: 10.2196/16235. v22i3e16235
    1. Inkster B, Sarda S, Subramanian V. An empathy-driven, conversational artificial intelligence agent (Wysa) for digital mental well-being: real-world data evaluation mixed-methods study. JMIR mHealth uHealth. 2018 Nov 23;6(11):e12106. doi: 10.2196/12106. v6i11e12106
    1. Fitzpatrick KK, Darcy A, Vierhile M. Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (Woebot): a randomized controlled trial. JMIR Ment Health. 2017 Jun 06;4(2):e19. doi: 10.2196/mental.7785. v4i2e19
    1. Czeisler MÉ, Lane RI, Petrosky E, Wiley JF, Christensen A, Njai R, Weaver MD, Robbins R, Facer-Childs ER, Barger LK, Czeisler CA, Howard ME, Rajaratnam SM. Mental health, substance use, and suicidal ideation during the COVID-19 pandemic - United States, June 24-30, 2020. MMWR Morb Mortal Wkly Rep. 2020 Aug 14;69(32):1049–1057. doi: 10.15585/mmwr.mm6932a1. doi: 10.15585/mmwr.mm6932a1.
    1. Nass C, Steuer J, Siminoff ER. Computers are social actors. CHI '94: Conference Companion on Human Factors in Computing Systems; April 24-28, 1994; Boston, MA. 1994. Apr 28, pp. 72–78.
    1. Nass C, Moon Y. Machines and mindlessness: social responses to computers. J Social Isssues. 2000 Jan;56(1):81–103. doi: 10.1111/0022-4537.00153.
    1. Christoforakos Lara, Feicht Nina, Hinkofer Simone, Löscher Annalena, Schlegl Sonja F, Diefenbach Sarah. Connect With Me. Exploring Influencing Factors in a Human-Technology Relationship Based on Regular Chatbot Use. Front Digit Health. 2021;3:689999. doi: 10.3389/fdgth.2021.689999.
    1. Keller PA, Lehmann DR. Designing effective health communications: a meta-analysis. Journal of Public Policy & Marketing. 2008 Sep 01;27(2):117–130. doi: 10.1509/jppm.27.2.117.
    1. Bode L, Vraga EK. In related news, that was wrong: the correction of misinformation through related stories functionality in social media. J Commun. 2015 Jun 23;65(4):619–638. doi: 10.1111/jcom.12166.
    1. Cook J, Lewandowsky S, Ecker UKH. Neutralizing misinformation through inoculation: exposing misleading argumentation techniques reduces their influence. PLoS One. 2017 May 5;12(5):e0175799. doi: 10.1371/journal.pone.0175799. PONE-D-16-15763
    1. Ramirez A, Despres C, Chalela P, Weis J, Sukumaran P, Munoz E, McAlister A. Pilot study of peer modeling with psychological inoculation to promote coronavirus vaccination. Health Educ Res. 2022 Mar 23;37(1):1–6. doi: 10.1093/her/cyab042.6511803
    1. Public Health Messaging Can Be Funny Too! American College Health Association. [2022-06-28]. .
    1. Chen S, Forster S, Yang J, Yu F, Jiao L, Gates J, Wang Z, Liu H, Chen Q, Geldsetzer P, Wu P, Wang C, McMahon S, Bärnighausen Till, Adam M. Animated, video entertainment-education to improve vaccine confidence globally during the COVID-19 pandemic: an online randomized controlled experiment with 24,000 participants. Trials. 2022 Feb 19;23(1):161. doi: 10.1186/s13063-022-06067-5. 10.1186/s13063-022-06067-5
    1. Vaala S, Ritter MB, Palakshappa D. Experimental effects of tweets encouraging social distancing: effects of source, emotional appeal, and political ideology on emotion, threat, and efficacy. J Public Health Manag Pract. 2022 Apr;28(2):E586–E594. doi: 10.1097/PHH.0000000000001427.00124784-900000000-99134
    1. The AI companion who cares. Replika. [2021-07-26]. .
    1. Wojcik S, Hughes A. Sizing up Twitter users. Pew Research Center. 2019. Apr 24, [2022-05-10].
    1. KFF COVID-19 vaccine monitor. Kaiser Family Foundation. [2022-05-22].
    1. Loft LH, Pedersen EA, Jacobsen SU, Søborg Bolette, Bigaard J. Using Facebook to increase coverage of HPV vaccination among Danish girls: an assessment of a Danish social media campaign. Vaccine. 2020 Jun 26;38(31):4901–4908. doi: 10.1016/j.vaccine.2020.04.032.S0264-410X(20)30512-0
    1. Nan X. Communicating to young adults about HPV vaccination: consideration of message framing, motivation, and gender. Health Commun. 2012;27(1):10–8. doi: 10.1080/10410236.2011.567447.

Source: PubMed

Sponsors and Collaborators

Medical Conditions

Drug Interventions

3
Subscribe