Assessing the Performance of Artificial Intelligence (AI)-Augmented Electronic Health Record (EHR) Data Abstraction for Clinical Trial Patient Screening

Identifying eligible patients is a key process in the clinical trial enterprise. Currently, this process relies on time-intensive manual chart review, creating a rate-limiting step for trial participation. The integration of AI technology into the trial screening process has potential to improve participation rates. This study aims to assess the performance (accuracy, efficiency) of AI-augmented patient identification and inform optimal integration into clinical research screening processes.

Study Overview

• Status: Completed
• Conditions: Cancer
• Intervention / Treatment: Other: Chart review

Detailed Description

The objective of this study is to assess and compare the accuracy and efficiency of three different approaches to abstracting clinical data used to identify oncology patients who meet the inclusion criteria for participation in clinical trials. The three approaches under evaluation include: (1) an autonomous AI algorithm (Mendel AI; developed by artificial intelligence startup company Mendel) which analyzes patient medical records to extract relevant clinical facts ("AI-alone"); (2) a human researcher who manually reviews patient charts as per the current norm/practice ("Human-alone"); and (3) a human researcher utilizing AI augmentation ("Human+AI"), where Mendel AI serves as a supportive tool in the decision-making process by providing the researcher a list of elements abstracted by the AI algorithm and a rank-order list of patients most likely to meet inclusion criteria for a trial.

The study primarily aims to compare (1) the chart-level accuracy of the Human+AI collaboration relative to Human-alone given the relevance of this comparison for real-world clinical workflows, defined by the percentage of pre-identified chart elements classified correctly compared against a predetermined "gold standard"; and (2) the efficiency of the Human+AI vs. Human-alone arms, defined by the time per chart review in minutes, measured for each chart.

Our hypotheses are (1) the Human+AI arm will be non-inferior in accuracy when compared to the Human-alone arm, in relation to a predetermined "gold standard", and (2) that a Human+AI arm will be superior in efficiency of abstraction when compared to Human-alone screening.

The identification of eligible patients for clinical trials is a critical component of clinical research, as it directly impacts patient recruitment, study enrollment, and the generalizability of research findings. Currently, the process of identifying eligible patients often relies on manual chart review by clinical research staff, which can be time-consuming, labor-intensive, and prone to human error. Consequently, eligible patients may be overlooked, and opportunities for trial participation may be missed. The integration of AI technology into the patient identification process has the potential to enhance the accuracy and efficiency of this critical task, leading to improved clinical trial recruitment and outcomes.

This study holds important implications for the field of clinical research by evaluating the effectiveness of AI-augmented patient identification compared to traditional manual methods and autonomous AI algorithms. By examining the strengths and limitations of each approach, the study will provide valuable insights into the optimal integration of AI technology in clinical research processes. Furthermore, the results of this study have the potential to benefit patients by improving their access to clinical trials and increasing awareness of available treatment options. For clinical research institutions, enhancing the efficiency of patient identification can lead to more effective use of research resources and the potential for accelerated clinical trial timelines. Ultimately, the findings of this study may contribute to advancements in clinical research practices, promoting more equitable access to trials and facilitating the development of innovative treatments for patients with cancer.

• Study Type: Observational
• Enrollment (Actual): 355

Contacts and Locations

Study Locations

• United States
  • Georgia
    • Atlanta, Georgia, United States, 30307
      • Emory University
  • Pennsylvania
    • Philadelphia, Pennsylvania, United States, 19104
      • University of Pennsylvania

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: Child; Adult; Older Adult
• Accepts Healthy Volunteers: No

• Sampling Method: Probability Sample

Study Population

De-identified patient charts from community oncology practices, with a diagnosis of non-small cell lung cancer (NSCLC) or colorectal cancer (CRC).

Description

Inclusion Criteria:

• Diagnosis of colorectal or non-small cell lung cancer.
• A minimum of 5 patient documents in the Mendel database.
• Most recent document was within 5 years from the time of data extraction.

Exclusion Criteria:

• None.

Study Plan

How is the study designed?

Number of groups / cohorts

3

Cohorts and Interventions

Group / Cohort	Intervention / Treatment
Human + AI	Other: Chart review; Chart review
AI-alone	Other: Chart review; Chart review
Human-alone	Other: Chart review; Chart review

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Abstracted Chart-level Accuracy	The primary outcome measured was mean chart-level accuracy, defined as the percentage of elements identified by clinical research coordinators among all elements in the gold-standard set, measured for each chart, and averaged across all charts. Research coordinator-abstracted responses were identified as being accurate when they exactly matched with the gold-standard set. The gold-standard set was determined by 2-3 clinicians blinded to experimental arms.	1 year

Secondary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Efficiency of Chart-level Abstraction (in Minutes)	Efficiency was calculated as the number of minutes spent on each chart abstraction.	1 year

Collaborators and Investigators

• Sponsor: University of Pennsylvania
• Collaborators: Mendel AI

Study record dates

Study Major Dates

• Study Start (Actual): August 18, 2023
• Primary Completion (Actual): July 12, 2024
• Study Completion (Actual): July 12, 2024

Study Registration Dates

• First Submitted: August 16, 2024
• First Submitted That Met QC Criteria: August 16, 2024
• First Posted (Actual): August 19, 2024

Study Record Updates

• Last Update Posted (Actual): July 25, 2025
• Last Update Submitted That Met QC Criteria: July 7, 2025
• Last Verified: July 1, 2025

More Information

Terms related to this study

• Other Study ID Numbers: 854016

Plan for Individual participant data (IPD)

• Plan to Share Individual Participant Data (IPD)?: NO
• IPD Plan Description: No individual participant data will be shared.

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No