Using Explainable AI Risk Predictions to Nudge Influenza Vaccine Uptake

The study team previously demonstrated that patients are more likely to receive flu vaccine after learning that they are at high risk for flu complications. Building on this past work, the present study will explore whether providing reasons that patients are considered high risk for flu complications (a) further increases the likelihood they will receive flu vaccine and (b) decreases the likelihood that they receive diagnoses of flu and/or flu-like symptoms in the ensuing flu season. It will also examine whether informing patients that their high-risk status was determined by analyzing their medical records or by an artificial intelligence (AI) / machine-learning (ML) algorithm analyzing their medical records will affect the likelihood of receiving the flu vaccine or diagnoses of flu and/or flu-like symptoms.

Study Overview

• Status: Completed
• Conditions: Influenza; Health Promotion; Health Behavior; Risk Reduction; Vaccination
• Intervention / Treatment: Behavioral: Reminder; Behavioral: Risk reduction; Behavioral: Medical records-based recommendation; Behavioral: Algorithm-based recommendation

Detailed Description

Geisinger has partnered with Medial EarlySign and developed an ML algorithm to identify patients at risk for serious (moderate to severe) flu-associated complications on the basis of their existing electronic health record (EHR) data. Geisinger will apply this algorithm to current patients during the 2021-22 flu season.

This study will evaluate the effect of contacting patients identified as high risk with special messages to encourage vaccination. These communications will inform patients they are at high risk with either (a) no additional explanation, (b) an explanation that this determination comes from an analysis of their medical records, along with a short list of the top factors from their medical record that explain their risk, and (c) the additional explanation that an AI or ML algorithm made this determination, along with a short list of the top factors from their medical record that explain their risk.

Included in the study will be current Geisinger patients 18+ years of age with no contraindications for flu vaccine and who have been assessed by the Medial algorithm and assigned a risk score. The primary study outcomes will be the rates of flu vaccination and flu diagnosis during the 2020-21 season by targeted patients.

• Study Type: Interventional
• Enrollment (Actual): 45061
• Phase: Not Applicable

Contacts and Locations

Study Locations

• United States
  • Pennsylvania
    • Danville, Pennsylvania, United States, 17822
      • Geisinger Clinic

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: 18 years and older (Adult, Older Adult)
• Accepts Healthy Volunteers: No

Description

Inclusion Criteria:

• Aged 18 or older
• Current Geisinger patient at the time of study
• Falls in the top 10% of patients at highest risk, as identified by the flu-complication risk scores of machine learning algorithm (which operates on coded EHR data)

Exclusion Criteria:

• Has contraindications for flu vaccination
• Has opted out of receiving communications from Geisinger via all of the modalities being tested

Study Plan

How is the study designed?

Design Details

• Primary Purpose: Prevention
• Allocation: Randomized
• Interventional Model: Parallel Assignment
• Masking: Single

Number of Arms

5

Arms and Interventions

Participant Group / Arm	Intervention / Treatment
No Intervention: No-Contact Control; Subjects in the no-contact control arm will receive no additional pro-vaccination intervention beyond the health system's normal efforts. Although some patients are currently targeted for flu vaccination encouragement due to a conventional non-ML assessment that they are at high risk for complications, these patients are not told that they are at high risk or that they have been targeted.
Experimental: Reminder Control; Subjects in the reminder control arm will receive messages reminding them to get the flu shot without being advised of their risk status.	Behavioral: Reminder; Mailed letter, short message service (SMS) text, and/or patient portal message
Experimental: High Risk Only; Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications, without specifying how or why the health system believes this to be the case.	Behavioral: Reminder; Mailed letter, short message service (SMS) text, and/or patient portal message; Behavioral: Risk reduction; Mailed letter, SMS, and/or patient portal message
Experimental: High Risk with Explanation Based on Medical Records; Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via review of their medical records and will be provided a human-understandable short list of the top factors from their medical record that explain their risk.	Behavioral: Reminder; Mailed letter, short message service (SMS) text, and/or patient portal message; Behavioral: Risk reduction; Mailed letter, SMS, and/or patient portal message; Behavioral: Medical records-based recommendation; Mailed letter, SMS, and/or patient portal message; Other Names: Credibility
Experimental: High Risk with Explanation Based on Algorithm; Subjects in this treatment arm will receive messages telling them they have been identified to be at high risk for flu complications via analysis of their medical records by a computer algorithm and will be provided a human-understandable short list of the top factors from their medical record that explain their risk.	Behavioral: Reminder; Mailed letter, short message service (SMS) text, and/or patient portal message; Behavioral: Risk reduction; Mailed letter, SMS, and/or patient portal message; Behavioral: Medical records-based recommendation; Mailed letter, SMS, and/or patient portal message; Other Names: Credibility; Behavioral: Algorithm-based recommendation; Mailed letter, SMS, and/or patient portal message; Other Names: Credibility

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Flu Vaccination at 2 Weeks After Final Outreach Date	Received flu vaccination	Within 2 weeks of the final outreach date, 57 days (8.14 weeks) after the study start

Secondary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Flu Vaccination at 9 Weeks After Final Outreach Date	Received a flu vaccination	Within 9 weeks of the final outreach date, 106 days (15.14 weeks) after the study start
Flu Diagnosis	Received a "high confidence flu" diagnosis (with positive polymerase chain reaction [PCR]/antigen/molecular test) and/or "likely flu" diagnosis (as assessed via International Classification of Disease [ICD] codes or Tamiflu administration or positive PCR/antigen/molecular test) Note that "likely flu" is a superset of the "high confidence flu" diagnoses.	8 months (between September 9, 2021 and April 30, 2022)
Flu Complications	Diagnosed with flu-related complications	11 months (between September 9, 2021 and July 31, 2022)
Healthcare Utilization	Visited ER or was hospitalized NOTE: Our prespecified outcome was description was "Visited ER, was hospitalized, or had flu-related insurance claims." We did not receive claims data, so this outcomes includes only ER visits and hospitalizations.	11 months (between September 9, 2021 and July 31, 2022)

Other Outcome Measures

Outcome Measure	Measure Description	Time Frame
Flu Vaccination at 2 Weeks After Final Outreach Date by Gender	Received a flu vaccination	Within 2 weeks of the final outreach date, 57 days (8.14 weeks) after the study start
Flu Vaccination at 2 Weeks After Final Outreach Date by Race	Received Flu Vaccination	Within 2 weeks of the final outreach date, 57 days (8.14 weeks) after the study start
Flu Vaccination at 2 Weeks After Final Outreach Date by Ethnicity	Received flu vaccination	Within 2 weeks of the final outreach date, 57 days (8.14 weeks) after the study start

Collaborators and Investigators

• Sponsor: National Bureau of Economic Research, Inc.
• Collaborators: National Institute on Aging (NIA); Massachusetts Institute of Technology; Geisinger Clinic
• Investigators: Principal Investigator: Michelle N Meyer, PhD JD, Geisinger Clinic; Principal Investigator: Christopher F Chabris, PhD, Geisinger Clinic

Study record dates

Study Major Dates

• Study Start (Actual): September 9, 2021
• Primary Completion (Actual): November 5, 2021
• Study Completion (Actual): July 31, 2022

Study Registration Dates

• First Submitted: August 4, 2021
• First Submitted That Met QC Criteria: August 14, 2021
• First Posted (Actual): August 17, 2021

Study Record Updates

• Last Update Posted (Actual): March 25, 2025
• Last Update Submitted That Met QC Criteria: December 16, 2024
• Last Verified: December 1, 2024

More Information

Terms related to this study

• Keywords: Choice Architecture; Flu Vaccine; Machine Learning; Perceived Credibility; Personalized Risk Factors; Flu Complications
• Additional Relevant MeSH Terms: Respiratory Tract Infections; Infections; Orthomyxoviridae Infections; RNA Virus Infections; Virus Diseases; Respiratory Tract Diseases; Influenza, Human
• Other Study ID Numbers: 2021-0483; P30AG034532 (U.S. NIH Grant/Contract)

Plan for Individual participant data (IPD)

• Plan to Share Individual Participant Data (IPD)?: YES
• IPD Plan Description: Data with no personally identifiable information will be made available to other researchers on the Open Science Framework for transparency. This will include the essential data and code needed to replicate the analysis that yielded reported findings.
• IPD Sharing Time Frame: By the paper's online publication date. Data will remain available for as long as the Open Science Framework hosts it.
• IPD Sharing Access Criteria: The data and code are available in the below repository in the folder ClinicalTrials.gov data/Study 2.
• IPD Sharing Supporting Information Type: STUDY_PROTOCOL; SAP; ANALYTIC_CODE; CSR

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No