Enhancing Interdisciplinary Understanding of Ophthalmology Notes Through a Local Large Language Model

Enhancing Interdisciplinary Understanding of Ophthalmology Notes Through Local Large Language Model

This prospective, randomized controlled trial evaluated the efficacy of adding Large Language model (LLM)-generated Plain Language Summaries (PLSs) to Standard Ophthalmology Notes (SONs) in enhancing comprehension among non-ophthalmology providers. The study utilized surveys to assess non-ophthalmology providers' comprehension and satisfaction with the notes and ophthalmologists' evaluation of PLS accuracy, safety, and time burden. An objective semantic and linguistic analysis of the PLSs was also conducted.

Study Overview

• Status: Completed
• Conditions: Communication; Interdisciplinary Communication; Artificial Intelligence (AI); Artificial Intelligence Technology
• Intervention / Treatment: Other: Large Language Model-generated Plain Language Summary of Ophthalmology notes

• Study Type: Interventional
• Enrollment (Actual): 851
• Phase: Not Applicable

Contacts and Locations

Study Locations

• United States
  • Minnesota
    • Rochester, Minnesota, United States, 55905
      • Mayo Clinic

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: Adult; Older Adult
• Accepts Healthy Volunteers: No

Description

Inclusion Criteria:

• Evaluation by an Ophthalmologist in the Mayo Clinic Department of Ophthalmology in Rochester, MN in either the inpatient or outpatient setting between Feb 1, 2024 - May 31, 2024.

Exclusion Criteria:

• No in-network non-Ophthalmologist care providers in the medical record
• Procedure-only visits containing only a procedure note

Study Plan

How is the study designed?

Design Details

• Primary Purpose: Health Services Research
• Allocation: Randomized
• Interventional Model: Parallel Assignment
• Masking: None (Open Label)

Number of Arms

2

Arms and Interventions

Participant Group / Arm	Intervention / Treatment
No Intervention: Control; Participants receive standard ophthalmology notes without any additions.
Experimental: Intervention; Participants receive standard ophthalmology notes with an appended LLM-generated plain language summary, reviewed and edited by the ophthalmologist. The plain language summary is clearly marked as generated as part of a Quality Improvement initiative.	Other: Large Language Model-generated Plain Language Summary of Ophthalmology notes; Prospective, randomized Quality Improvement study with real-world implementation of Large Language Model-generated Plain Language Summaries of Ophthalmology notes.

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Non-Ophthalmologist Note Comprehension and Satisfaction	Assessed via survey responses evaluating understanding of the patient's ophthalmology diagnosis, satisfaction with the note, and overall preference between Standard Ophthalmology Note and Standard Ophthalmology Note + Plain Language Summary. Degree of understanding was assessed on the following Likert scale: Not at all Neutral Moderately A great deal Satisfaction was graded on the following Likert scale: Not satisfied at all Somewhat unsatisfied Neutral Somewhat satisfied Satisfied Preference between note types was assessed on the following Likert scale: Standard Note, a great deal Standard Note, somewhat Neutral Plain Language Summary, somewhat Plain Language Summary, a great deal	From enrollment to 8 weeks after enrollment

Secondary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Ophthalmologist Evaluation of Summary Accuracy and Time burden	Assessed via survey responses regarding Plain Language Summary accuracy in reflecting the Standard Ophthalmology Note findings, time spent reviewing/editing, and perceived burden. Accuracy in reflecting findings was graded on the following Likert scale: Not at all Neutral A little A great deal Time spent reviewing/editing was reported using the following options: <1 minute; minute; minutes; minutes; minutes; or more minutes Perceived burden was graded on the following Likert scale: Not at all Neutral A little A great deal	Assessed at a single time point <24 hr after enrollment
Semantic and Linguistic Quality of Plain Language Summaries	Flesch Reading Ease Scale: 0 (poor) to 100 (good)	Assessed at a single time point <24 hr following enrollment
Semantic and Linguistic Quality of Plain Language Summaries	Flesch-Kincaid Grade Level Scale: 0 (most difficult to read, highest grade level) to 80 (easiest to read, lowest grade level)	Assessed at a single time point <24 hr following enrollment
Semantic and Linguistic Quality of Plain Language Summaries	Semantic Measure of Gobbledygook (SMOG) Index Scale: 0 (easiest to read) to 11 (most difficult to read)	Assessed at a single time point <24 hr following enrollment
Semantic and Linguistic Quality of Plain Language Summaries	BERTScore Measure of Linguistic Similarity Scale: -1 (least similar) to +1 (most similar)	Assessed at a single time point <24 hr following enrollment
Semantic and Linguistic Quality of Plain Language Summaries	SBERT Cosine Similarity Measure of Linguistic Similarity Scale: -1 (least similar) to +1 (most similar)	Assessed at a single time point <24 hr following enrollment

Other Outcome Measures

Outcome Measure	Measure Description	Time Frame
Presence of Errors, Type of Errors, and Potential Harm of Errors	Presence of any incorrect or missing information in the Plain Language Summary, error rates, types of errors, and potential harm rating. Types of errors were reported as: HPI Exam findings Diagnosis Plan Explanation of the patient's overall eye health Other N/A Likelihood of harm was graded on the following scale: Low risk Medium risk High risk N/A Potential harm rating was graded on the following scale: Mild or moderate harm Death or severe harm	Assessed at a single time point <24 hr following enrollment and again at a single time point 8 weeks after enrollment

Collaborators and Investigators

• Sponsor: John J Chen
• Investigators: Study Director: John J Chen, MD, PhD, Mayo Clinic

Study record dates

Study Major Dates

• Study Start (Actual): February 1, 2024
• Primary Completion (Actual): May 31, 2024
• Study Completion (Actual): May 31, 2024

Study Registration Dates

• First Submitted: September 30, 2024
• First Submitted That Met QC Criteria: October 1, 2024
• First Posted (Actual): October 3, 2024

Study Record Updates

• Last Update Posted (Actual): October 3, 2024
• Last Update Submitted That Met QC Criteria: October 1, 2024
• Last Verified: October 1, 2024

More Information

Terms related to this study

• Other Study ID Numbers: 24-000550

Plan for Individual participant data (IPD)

• Plan to Share Individual Participant Data (IPD)?: YES
• IPD Plan Description: All collected IPD.
• IPD Sharing Time Frame: Beginning after publication with no end date
• IPD Sharing Access Criteria: The underlying data supporting the findings of this study can be made available to clinical investigators and researchers upon request. Written requests for data sharing including an analysis plan will be required before approval. These requests will be individually assessed in consultation with the study team leads and co-investigators as appropriate. If other investigators are interested in performing additional analyses, these requests can be made to the corresponding author (PDT) and analyses will be performed in collaboration with the Mayo Clinic. In all cases, any data and materials to be shared will be released via a Material Transfer Agreement. Individual-level data will be available and data sharing will ensure that the rights and privacy of individuals participating in the research always remains protected. The anticipated time frame to respond to initial data requests is 1 month.
• IPD Sharing Supporting Information Type: STUDY_PROTOCOL; ANALYTIC_CODE; CSR

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No
• product manufactured in and exported from the U.S.: No