DHL Survey on Generative AI for MyChart Messaging

Duke Health Listens Survey on Generative Artificial Intelligence (AI) for MyChart Messaging

The purpose of this study is to understand how patients feel about the use of computer programs to create responses when they send electronic messages to their doctors.

Study Overview

• Status: Completed
• Conditions: Communication
• Intervention / Treatment: Behavioral: Generative AI for electronic communication and disclosure

Detailed Description

• The investigators will create short surveys online to ask patients how they feel about using computer programs that create messages in their medical records.
• The surveys will show fictional situations where patients ask questions and get answers from either real people or computer programs, with or without a disclosure about how the response was written.
• The investigators will ask the people taking the survey to share what they think about these situations using tools like rating scales, comparison scales, or written responses.
• If patients want to, they can provide their contact information to be part of future discussion groups. Participants do not have to give any personal information to complete the survey.

• Study Type: Interventional
• Enrollment (Actual): 1454
• Phase: Not Applicable

Contacts and Locations

Study Locations

• United States
  • North Carolina
    • Durham, North Carolina, United States, 27710
      • Duke University Health System

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: Adult; Older Adult
• Accepts Healthy Volunteers: Yes

Description

Inclusion Criteria:

• Member of the Duke Health Listens patient advocacy community

Exclusion Criteria:

• Age < 18

Study Plan

How is the study designed?

Design Details

• Primary Purpose: Other
• Allocation: Randomized
• Interventional Model: Parallel Assignment
• Masking: Single

Number of Arms

6

Arms and Interventions

Participant Group / Arm	Intervention / Treatment
Other: Arm A; Each arm will receive 3 clinical scenarios spaced over time across 3 Sends. The 6 groups (Arms A-F) will be arranged with naming conventions as such: First letter = A, B, or C where A = scenario 1, B = scenario 2, and C = scenario 3.; Second letter(s) = H or AI, where H = human response and AI = AI-written response to the patient question posed.; Third letter(s) = N, C, or H, where N = no disclosure, C = computer disclosure, and H = human disclosure. This refers to the disclosure at the bottom of the response message whereby the author is or is not disclosed. Arm A receives AHN in Send 1, BAIC in Send 2, and CHH in Send 3	Behavioral: Generative AI for electronic communication and disclosure; We will use a large language model such as GPT 3.5 to automatically generate responses to fictional messages to a physician. We will disclose whether the message was generated using this technology or not. There are 3 clinical scenarios and 6 pairs of human/AI response and human disclosure/AI disclosure/not disclosed that will test patient attitudes toward this technology.
Other: Arm B; Each arm will receive 3 clinical scenarios spaced over time across 3 Sends. The 6 groups (Arms A-F) will be arranged with naming conventions as such: First letter = A, B, or C where A = scenario 1, B = scenario 2, and C = scenario 3.; Second letter(s) = H or AI, where H = human response and AI = AI-written response to the patient question posed.; Third letter(s) = N, C, or H, where N = no disclosure, C = computer disclosure, and H = human disclosure. This refers to the disclosure at the bottom of the response message whereby the author is or is not disclosed. Arm B receives BHC in Send 1, CAIH in Send 2, and AAIN in Send 3	Behavioral: Generative AI for electronic communication and disclosure; We will use a large language model such as GPT 3.5 to automatically generate responses to fictional messages to a physician. We will disclose whether the message was generated using this technology or not. There are 3 clinical scenarios and 6 pairs of human/AI response and human disclosure/AI disclosure/not disclosed that will test patient attitudes toward this technology.
Other: Arm C; Each arm will receive 3 clinical scenarios spaced over time across 3 Sends. The 6 groups (Arms A-F) will be arranged with naming conventions as such: First letter = A, B, or C where A = scenario 1, B = scenario 2, and C = scenario 3.; Second letter(s) = H or AI, where H = human response and AI = AI-written response to the patient question posed.; Third letter(s) = N, C, or H, where N = no disclosure, C = computer disclosure, and H = human disclosure. This refers to the disclosure at the bottom of the response message whereby the author is or is not disclosed. Arm C receives CHC in Send 1, AHH in Send 2, and BAIN in Send 3	Behavioral: Generative AI for electronic communication and disclosure; We will use a large language model such as GPT 3.5 to automatically generate responses to fictional messages to a physician. We will disclose whether the message was generated using this technology or not. There are 3 clinical scenarios and 6 pairs of human/AI response and human disclosure/AI disclosure/not disclosed that will test patient attitudes toward this technology.
Other: Arm D; Each arm will receive 3 clinical scenarios spaced over time across 3 Sends. The 6 groups (Arms A-F) will be arranged with naming conventions as such: First letter = A, B, or C where A = scenario 1, B = scenario 2, and C = scenario 3.; Second letter(s) = H or AI, where H = human response and AI = AI-written response to the patient question posed.; Third letter(s) = N, C, or H, where N = no disclosure, C = computer disclosure, and H = human disclosure. This refers to the disclosure at the bottom of the response message whereby the author is or is not disclosed. Arm D receives AAIH in Send 1, BHN in Send 2, and CAIC in Send 3	Behavioral: Generative AI for electronic communication and disclosure; We will use a large language model such as GPT 3.5 to automatically generate responses to fictional messages to a physician. We will disclose whether the message was generated using this technology or not. There are 3 clinical scenarios and 6 pairs of human/AI response and human disclosure/AI disclosure/not disclosed that will test patient attitudes toward this technology.
Other: Arm E; Each arm will receive 3 clinical scenarios spaced over time across 3 Sends. The 6 groups (Arms A-F) will be arranged with naming conventions as such: First letter = A, B, or C where A = scenario 1, B = scenario 2, and C = scenario 3.; Second letter(s) = H or AI, where H = human response and AI = AI-written response to the patient question posed.; Third letter(s) = N, C, or H, where N = no disclosure, C = computer disclosure, and H = human disclosure. This refers to the disclosure at the bottom of the response message whereby the author is or is not disclosed. Arm E receives BAIH in Send 1, CHN in Send 2, and AHC in Send 3	Behavioral: Generative AI for electronic communication and disclosure; We will use a large language model such as GPT 3.5 to automatically generate responses to fictional messages to a physician. We will disclose whether the message was generated using this technology or not. There are 3 clinical scenarios and 6 pairs of human/AI response and human disclosure/AI disclosure/not disclosed that will test patient attitudes toward this technology.
Other: Arm F; Each arm will receive 3 clinical scenarios spaced over time across 3 Sends. The 6 groups (Arms A-F) will be arranged with naming conventions as such: First letter = A, B, or C where A = scenario 1, B = scenario 2, and C = scenario 3.; Second letter(s) = H or AI, where H = human response and AI = AI-written response to the patient question posed.; Third letter(s) = N, C, or H, where N = no disclosure, C = computer disclosure, and H = human disclosure. This refers to the disclosure at the bottom of the response message whereby the author is or is not disclosed. Arm F receives CAIN in Send 1, AAIC in Send 2, and BHH in Send 3	Behavioral: Generative AI for electronic communication and disclosure; We will use a large language model such as GPT 3.5 to automatically generate responses to fictional messages to a physician. We will disclose whether the message was generated using this technology or not. There are 3 clinical scenarios and 6 pairs of human/AI response and human disclosure/AI disclosure/not disclosed that will test patient attitudes toward this technology.

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Patient satisfaction, as measured by survey	Likert-scale responses to satisfaction question: "I am satisfied with this interaction", on a scale from 1-5 with answer options of Strongly Disagree (1), Disagree (2), Neither agree nor disagree (3), Agree (4), and Strongly agree (5).	Up to 2 weeks
Patient attitudes towards utility, as measured by survey	Likert-scale responses to utility question: "The information is useful", on a scale from 1-5 with answer options of Strongly Disagree (1), Disagree (2), Neither agree nor disagree (3), Agree (4), and Strongly agree (5).	Up to 2 weeks
Patient empathy, as measured by survey	Likert-scale responses to empathy question: "I feel cared for during this interaction", on a scale from 1-5 with answer options of Strongly Disagree (1), Disagree (2), Neither agree nor disagree (3), Agree (4), and Strongly agree (5).	Up to 2 weeks

Collaborators and Investigators

• Sponsor: Duke University
• Investigators: Principal Investigator: Anand Chowdhury, MD, MMCi, Duke University

Publications and helpful links

General Publications

• Ayers JW, Poliak A, Dredze M, Leas EC, Zhu Z, Kelley JB, Faix DJ, Goodman AM, Longhurst CA, Hogarth M, Smith DM. Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum. JAMA Intern Med. 2023 Jun 1;183(6):589-596. doi: 10.1001/jamainternmed.2023.1838.

Study record dates

Study Major Dates

• Study Start (Actual): October 31, 2023
• Primary Completion (Actual): December 11, 2023
• Study Completion (Actual): December 11, 2023

Study Registration Dates

• First Submitted: October 23, 2023
• First Submitted That Met QC Criteria: October 26, 2023
• First Posted (Actual): October 30, 2023

Study Record Updates

• Last Update Posted (Estimated): December 13, 2023
• Last Update Submitted That Met QC Criteria: December 12, 2023
• Last Verified: October 1, 2023

More Information

Terms related to this study

• Keywords: Artificial Intelligence; Generative Artificial Intelligence
• Other Study ID Numbers: Pro00113587

Plan for Individual participant data (IPD)

• Plan to Share Individual Participant Data (IPD)?: NO
• IPD Plan Description: The investigators will not collect individual patient identifiers, and aggregate data will be reported

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No