Safe and Explainable AI

SAFE AND EXPLAINABLE AI-ENABLED DECISION MAKING FOR PERSONALIZED CLINICAL DECISION SUPPORT

While current AI technology is suitable for automating some repetitive clinical tasks, technical challenges remain in solving critical and gainful problems in the domains of patient and disease management. The proposed research seeks to address issues in medical AI, such as integrating medical knowledge effectively, making AI recommendations explainable to clinicians, and establishing safety guarantees.

Study Overview

• Status: Recruiting
• Conditions: Sepsis; Breast Cancer; Cardiology; Artifical Inteligence
• Intervention / Treatment: Other: AI-PERSONALIZED CLINICAL DECISION SUPPORT

• Study Type: Observational
• Enrollment (Estimated): 300000

Contacts and Locations

• Study Contact: Name: Haideliza Soto Calderon; Phone Number: 215-220-9425; Email: Haideliza.Soto-Calderon@pennmedicine.upenn.edu
• Study Contact Backup: Name: Nicholas Bishop; Email: nicholas.bishop@pennmedicine.upenn.edu

Study Locations

• United States
  • Pennsylvania
    • Philadelphia, Pennsylvania, United States, 19104
      • Recruiting
      • Hospital of the University of Pennsylvania
      • Contact: Haideliza Soto Calderon; Phone Number: 215-237-4509; Email: haideliza.soto-calderon@pennmedicine.upenn.edu

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: Adult; Older Adult
• Accepts Healthy Volunteers: No

• Sampling Method: Non-Probability Sample

Study Population

The study will look at EMR data from cardiac, oncology, and patients at risk for acquiring sepsis.

Each group has clearly defined inclusion-exclusion criteria. For the machine learning model to predict ventricular arrhythmias and in-hospital cardiac arrest, we will construct patient cohorts consisting of adult (i.e., 18 years of age or older) patients admitted to Sickbay-accessible bed within a Penn Medicine hospital from 2020 to the present as well as patients identified as having experienced cardiac arrest using EHR records.

Description

Inclusion Criteria:

Cardiology 18 years of age and older, admitted to any of the Penn Medicine hospitals from 2017 to the present. Sepsis 18 years of age at the time of presentation to an emergency department or admission to any Penn Medicine hospital from July 1, 2017, onward will be eligible as this represents the population at risk for acquiring sepsis Oncology 18 years of age and older with a diagnosis of invasive breast cancer (Stage 1-4) in the Penn Cancer registry

Exclusion Criteria All prediction models will exclude patients under the age of 18 from their patient data sets.

Cardiology Patients whose primary admission diagnosis was cardiac arrest Sepsis Those with pre-existing limitations on life-sustaining therapy will be excluded because their eligibility for sepsis definitions, care received, and outcomes, may be significantly and variably affected by pre-existing limitations on care. Oncology There are no other exclusions.

Study Plan

How is the study designed?

Number of groups / cohorts

3

Cohorts and Interventions

Group / Cohort	Intervention / Treatment
Cardiology; The primary objective in this clinical case scenario is to evaluate an ML model utilizing real-time cardiac telemetry, as well as other clinical, demographic, and imaging structured data sources, among hospitalized, intensive care unit (ICU) patients to predict impending inhospital cardiac arrest, identify potentially reversible causes of cardiac arrest, and predict which patients may have impending cardiac arrest due to shockable rhythms i.e. ventricular tachycardia (VT) or ventricular fibrillation (VF).	Other: AI-PERSONALIZED CLINICAL DECISION SUPPORT; AI-ENABLED DECISION MAKING FOR PERSONALIZED CLINICAL DECISION SUPPORT
Oncology - Breast Cancer; The primary objective in this clinical case scenario is to evaluate an ML model utilizing structured and unstructured data from clinical, demographic, and tumor molecular and germline sequencing, among outpatients with cancer, to predict short-term mortality and/or symptom decline. The model for prediction to treatment response in breast cancer patients will be compared with two prognostic tools: 1) Conversation Connect, a previously validated machine learning mortality prediction tool that has been used at the University of Pennsylvania for routine clinical decision support, and 2) the Elixhauser Comorbidity Index, a comorbidity-based prognostic index used commonly in research and risk-adjustment.	Other: AI-PERSONALIZED CLINICAL DECISION SUPPORT; AI-ENABLED DECISION MAKING FOR PERSONALIZED CLINICAL DECISION SUPPORT
Sepsis; The primary objective in this clinical case scenario is to develop and evaluate an ML model that utilizes multidmodal clinical data (e.g., structured EHR data such as demographics, laboratory test results, and vital signs; unstructured EHR data including the text of clinical encounter notes and, where available, waveforms from real-time cardiac, hemodynamic, and respiratory monitoring devices) to predict the need for initiation of broad-spectrum antimicrobial therapy for hospitalized patients with sepsis. With a focus on implementable and explainable AI, we will produce well calibrated predictions that are also clinically meaningful at the bedside to aid real-time decision-making about diagnosis and treatment initiation. The model for timely diagnosis and intervention in sepsis will be compared with widely used commercial and open-source sepsis prediction models.	Other: AI-PERSONALIZED CLINICAL DECISION SUPPORT; AI-ENABLED DECISION MAKING FOR PERSONALIZED CLINICAL DECISION SUPPORT

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Neurosymbolic Learning Algorithms	Develop and evaluate novel algorithms for training neurosymbolic models. We will develop data- and compute-efficient algorithms for end-to-end training of neurosymbolic models. This task will reduce the burden on clinician experts to provide fine-grained labels on voluminous EHR data.	Prototype and develop new learning algorithms; 18 months. Benchmark and evaluate the learning algorithms; 24 months. Publish research results; 24 months
Explanation Methods	We will develop new explainable AI techniques that come with verifiable guarantees. These guarantees will enable trust and transparency in AI at a fundamental level.	Prototype and develop new explanation algorithms; 18 months. Derive certified guarantees for explanations; 18 months. Benchmark and evaluate the explanation algorithms; 24 months. Extend certificates to new properties and tasks; 30 months. Publ
Methods for Safety Guarantees	We will develop new algorithms that can scalably extract complex logical rules governing safety within the data that have statistical guarantees. These techniques will be rooted in statistical analysis and assist users in identifying out of distribution data and detecting anomalies.	Prototype and develop new rule learning algorithms; 30 months. Scale rule learning algorithms to larger data settings; 36 months. Incorporate new primitives to express complex rules; 36 months. Implement rule learning algorithms on baseline tasks

Collaborators and Investigators

• Sponsor: Abramson Cancer Center at Penn Medicine

Study record dates

Study Major Dates

• Study Start (Actual): November 29, 2025
• Primary Completion (Estimated): November 1, 2028
• Study Completion (Estimated): November 1, 2028

Study Registration Dates

• First Submitted: November 15, 2024
• First Submitted That Met QC Criteria: November 15, 2024
• First Posted (Actual): November 19, 2024

Study Record Updates

• Last Update Posted (Actual): February 25, 2026
• Last Update Submitted That Met QC Criteria: February 23, 2026
• Last Verified: February 1, 2026

More Information

Terms related to this study

• Additional Relevant MeSH Terms: Pathologic Processes; Neoplasms by Site; Neoplasms; Infections; Systemic Inflammatory Response Syndrome; Inflammation; Skin Diseases; Breast Diseases; Pathological Conditions, Signs and Symptoms; Skin and Connective Tissue Diseases; Breast Neoplasms; Sepsis
• Other Study ID Numbers: 34124

Plan for Individual participant data (IPD)

• Plan to Share Individual Participant Data (IPD)?: NO

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No