Head-to-Head Evaluation of the Cancer Ontology Supervised Multimodal Orchestration (COSMO) AI System Versus Pathologist-Only Review (COSMO)

December 13, 2025 updated by: Kun-Hsing Yu, Harvard Medical School (HMS and HSDM)

This study evaluates the diagnostic performance of the Cancer Ontology Supervised Multimodal Orchestration (COSMO) AI system for cancer subtype classification and compares it head-to-head with pathologist-only review. Pathologists will independently review de-identified whole-slide images derived from up to 300 patients across three anatomical sites (brain, lung, kidney) and provide diagnostic assessments. In parallel, COSMO will process the same cases offline to generate independent predictions, enabling direct comparison of diagnostic accuracy between human experts and the AI system.

The study will characterize the diagnostic accuracy of COSMO and pathologists, inter-observer agreement, and variations in performance across anatomical sites and cancer types with different incidence rates. Results will establish how COSMO compares to pathologists on identical cases and will inform the development of AI-assisted diagnostic systems in clinical practice.

Study Overview

Status

Status

Indicates the current recruitment status or the expanded access status.
Enrolling by invitation

Conditions

Conditions

The disease, disorder, syndrome, illness, or injury that is being studied. On ClinicalTrials.gov, conditions may also include other health-related issues, such as lifespan, quality of life, and health risks.

Intervention / Treatment

Intervention / Treatment

A process or action that is the focus of a clinical study. Interventions include drugs, medical devices, procedures, vaccines, and other products that are either investigational or already available. Interventions can also include noninvasive approaches, such as education or modifying diet and exercise.

Detailed Description

Study Rationale and Background Diagnostic accuracy in cancer subtype classification varies significantly among pathologists due to differences in expertise, experience, and access to diagnostic resources. The emergence of AI systems in pathology offers the potential to enhance diagnostic performance and consistency in cancer classification. However, direct empirical comparisons of AI-based predictions and pathologists' diagnostic performance on identical cases remain limited in the literature.

Study Aims This head-to-head comparative study aims to: (1) evaluate the diagnostic performance of the COSMO AI system in cancer subtype classification across multiple anatomical sites; (2) characterize the diagnostic accuracy of experienced pathologists on the same cases; (3) directly compare diagnostic performance metrics between COSMO and pathologists; and (4) examine concordance patterns and performance variation by anatomical site, cancer incidence category, pathologist experience, and case complexity.

Study Setting and Participants The study will involve up to 25 board-certified pathologists with 3 to 10+ years of diagnostic experience, recruited from institutions across North America, Europe, and the Asia-Pacific region. Participating pathologists will have domain expertise in neuropathology, pulmonary pathology, urologic pathology, or general anatomical pathology.

Cases and Stratification The study will employ de-identified archival whole-slide images representing up to 300 patients with confirmed reference diagnoses, including 100 brain cancers, 100 lung cancers, and 100 kidney cancers. Cases will be stratified by cancer type and incidence category (common vs. rare or uncommon), consistent with World Health Organization (WHO) guidelines.

Data Collection Pathologists will independently review each case and provide diagnostic classifications along with confidence assessments using a 5-point scale. The digital pathology interface will automatically record time-to-diagnosis metrics. COSMO will process the same cases offline to generate independent diagnostic predictions and confidence scores. Both pathologist and AI predictions will be evaluated against established reference standard diagnoses.

Analysis Framework The primary analysis will characterize diagnostic performance metrics (including accuracy, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and area under the receiver operating characteristic curve (AUROC)) for both pathologists (at the individual and aggregated levels) and the COSMO system. Secondary analyses will assess performance stratified by anatomical site, cancer incidence category, and pathologist experience level.

Study Type

Study Type

Describes the nature of a clinical study. Study types include interventional studies (also called clinical trials), observational studies (including patient registries), and expanded access.
Observational

Enrollment (Estimated)

Enrollment

The number of participants in a clinical study. The "anticipated" enrollment is the target number of participants that the researchers need for the study.
30

Contacts and Locations

This section provides the contact details for those conducting the study, and information on where this study is being conducted.

Study Locations

  • United States
    • Massachusetts
      • Boston, Massachusetts, United States, 02115
        • Harvard Medical School

Participation Criteria

Researchers look for people who fit a certain description, called eligibility criteria. Some examples of these criteria are a person's general health condition or prior treatments.

Eligibility Criteria

Eligibility Criteria

The key requirements that people who want to participate in a clinical study must meet or the characteristics they must have. Eligibility criteria consist of both inclusion criteria (which are required for a person to participate in the study) and exclusion criteria (which prevent a person from participating). Types of eligibility criteria include whether a study accepts healthy volunteers, has age or age group requirements, or is limited by sex.

Ages Eligible for Study

  • Child
  • Adult
  • Older Adult

Accepts Healthy Volunteers

No

Sampling Method

Non-Probability Sample

Study Population

We will recruit pathologists from international academic medical centers, hospital systems, and diagnostic pathology practices across North America (United States), Europe (Austria, Hungary), and the Asia-Pacific (Taiwan, Hong Kong, South Korea, China, India) region. Participating sites will include major academic institutions with established pathology departments, with recruitment targeting expertise in neuropathology, pulmonary pathology, and urologic pathology.

Description

Inclusion Criteria:

  • Board-certified pathologist with expertise in neuropathology, pulmonary pathology, urologic pathology, or general anatomical pathology
  • Minimum of 3 years of clinical diagnostic experience
  • Active clinical practice involving diagnostic pathology slide review
  • Willingness to independently review and diagnose up to 300 de-identified whole-slide images
  • Ability to access the study platform and complete case reviews within the specified study timeline
  • Provision of informed consent for study participation

Exclusion Criteria:

  • Prior involvement in the design or validation of the COSMO AI system
  • Inability to commit sufficient time to complete assigned case reviews
  • Presence of significant financial conflicts of interest related to the study outcomes

Study Plan

This section provides details of the study plan, including how the study is designed and what the study is measuring.

How is the study designed?

Design Details

Cohorts and Interventions

Group / Cohort

Group / Cohort

A group or subgroup of participants in an observational study that is assessed for biomedical or health outcomes.
Intervention / Treatment

Intervention / Treatment

A process or action that is the focus of a clinical study. Interventions include drugs, medical devices, procedures, vaccines, and other products that are either investigational or already available. Interventions can also include noninvasive approaches, such as education or modifying diet and exercise.
AI-Based Evaluation using COSMO
Pathologist-Based Evaluation
Diagnostic test: Digital Pathology Evaluation
Digital Pathology Evaluation

What is the study measuring?

Primary Outcome Measures

Primary Outcome Measures

In a clinical study's protocol, the planned outcome measure that is the most important for evaluating the effect of an intervention/treatment. Most clinical studies have one primary outcome measure, but some have more than one.
Outcome Measure
Measure Description
Time Frame
Diagnostic performance
Time Frame: Periprocedural (at the time of slide review)
Diagnostic performance of the COSMO AI system and pathologists in identifying cancer subtypes across brain, lung, and kidney tumors, as assessed by accuracy, balanced accuracy, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and area under the receiver operating characteristic curve (AUROC). We will include both overall comparisons and stratified evaluations by anatomical site and cancer incidence category (common vs. rare or uncommon).
Periprocedural (at the time of slide review)

Secondary Outcome Measures

Secondary Outcome Measures

In a clinical study's protocol, a planned outcome measure that is not as important as the primary outcome measure for evaluating the effect of an intervention but is still of interest. Most clinical studies have more than one secondary outcome measure.
Outcome Measure
Measure Description
Time Frame
Inter-Observer Agreement Among Pathologists
Time Frame: Periprocedural (at the time of slide review)
Diagnostic concordance among participating pathologists, measured by Fleiss' kappa, intraclass correlation coefficient (ICC), and pairwise concordance rates.
Periprocedural (at the time of slide review)
Pathologist-COSMO AI Concordance
Time Frame: Periprocedural (at the time of slide review)
Agreement patterns between pathologist diagnoses and COSMO AI predictions, including proportion of concordant cases overall and stratified by anatomical site, cancer incidence category, and pathologist experience level.
Periprocedural (at the time of slide review)
Diagnostic Confidence
Time Frame: Periprocedural (at the time of slide review)
Mean confidence scores (5-point scale) reported by pathologists during diagnostic assessment, stratified by anatomical site, cancer incidence category, and diagnostic correctness (correct vs. incorrect).
Periprocedural (at the time of slide review)
Time-to-Diagnosis
Time Frame: Periprocedural (at the time of slide review)
Mean diagnostic time (in seconds) required by pathologists to provide cancer subtype classification, stratified by anatomical site, cancer incidence category, and pathologist experience level.
Periprocedural (at the time of slide review)
Diagnostic Performance Stratified by Pathologist Experience
Time Frame: Periprocedural (at the time of slide review)
Diagnostic accuracy of pathologists stratified by years of clinical experience (3-5 years, 6-10 years, >10 years) to assess the relationship between experience level and diagnostic performance in cancer subtype classification.
Periprocedural (at the time of slide review)

Collaborators and Investigators

This is where you will find people and organizations involved with this study.

Sponsor

Sponsor

The organization or person who initiates the study and who has authority and control over the study.
Harvard Medical School (HMS and HSDM)

Investigators

Investigators

A researcher involved in a clinical study. Related terms include site principal investigator, site sub-investigator, study chair, study director, and study principal investigator.
  • Principal Investigator: Kun-Hsing Yu, MD, PhD, Harvard Medical School (HMS and HSDM)

Study record dates

These dates track the progress of study record and summary results submissions to ClinicalTrials.gov. Study records and reported results are reviewed by the National Library of Medicine (NLM) to make sure they meet specific quality control standards before being posted on the public website.

Study Major Dates

Study Start (Actual)

Study Start

The actual date on which the first participant was enrolled in a clinical study. The "anticipated" study start date is the date that the researchers think will be the study start date.
June 12, 2025

Primary Completion (Estimated)

Primary Completion

The date on which the last participant in a clinical study was examined or received an intervention to collect final data for the primary outcome measure. Whether the clinical study ended according to the protocol or was terminated does not affect this date. For clinical studies with more than one primary outcome measure with different completion dates, this term refers to the date on which data collection is completed for all the primary outcome measures. The "anticipated" primary completion date is the date that the researchers think will be the primary completion date for the study.
January 31, 2026

Study Completion (Estimated)

Study Completion

The date on which the last participant in a clinical study was examined or received an intervention/treatment to collect final data for the primary outcome measures, secondary outcome measures, and adverse events (that is, the last participant's last visit). The "anticipated" study completion date is the date that the researchers think will be the study completion date.
January 31, 2026

Study Registration Dates

First Submitted

First Submitted

The date on which the study sponsor or investigator first submitted a study record to ClinicalTrials.gov. There is typically a delay of a few days between the first submitted date and the record's availability on ClinicalTrials.gov (the first posted date).
December 13, 2025

First Submitted That Met QC Criteria

First Submitted That Met QC Criteria

The date on which the study sponsor or investigator first submits a study record that is consistent with National Library of Medicine (NLM) quality control (QC) review criteria. The sponsor or investigator may need to revise and submit a study record one or more times before NLM's QC review criteria are met. It is the responsibility of the sponsor or investigator to ensure that the study record is consistent with the NLM QC review criteria.
December 13, 2025

First Posted (Actual)

First Posted

The date on which the study record was first available on ClinicalTrials.gov after National Library of Medicine (NLM) quality control (QC) review has concluded. There is typically a delay of a few days between the date the study sponsor or investigator submitted the study record and the first posted date.
December 29, 2025

Study Record Updates

Last Update Posted (Actual)

Last Update Posted

The most recent date on which changes to a study record were made available on ClinicalTrials.gov. There may be a delay between when the changes were submitted to ClinicalTrials.gov by the study's sponsor or investigator (the last update submitted date) and the last update posted date.
December 29, 2025

Last Update Submitted That Met QC Criteria

Last Update Submitted That Met QC Criteria

The most recent date on which the study sponsor or investigator submitted changes to a study record that are consistent with National Library of Medicine (NLM) quality control (QC) review criteria. It is the responsibility of the sponsor or investigator to ensure that the study record is consistent with the NLM QC review criteria.
December 13, 2025

Last Verified

Last Verified

The most recent date on which the study sponsor or investigator confirmed the information about a clinical study on ClinicalTrials.gov as accurate and current. If a study with a recruitment status of recruiting; not yet recruiting; or active, not recruiting has not been confirmed within the past 2 years, the study's recruitment status is shown as unknown.
December 1, 2025

More Information

Terms related to this study

Keywords

Additional Relevant MeSH Terms

Other Study ID Numbers

Other Study ID Numbers

Identifiers or ID numbers other than the NCT number that are assigned to a clinical study by the study's sponsor, funders, or others. These numbers may include unique identifiers from other trial registries and National Institutes of Health grant numbers.
  • Yu Lab COSMO Study

Plan for Individual participant data (IPD)

Plan to Share Individual Participant Data (IPD)?

NO

IPD Plan Description

Individual pathologist diagnostic assessments will not be shared to protect evaluator anonymity and privacy. De-identified case data and aggregated performance metrics will be made available through published results and supplementary materials. The protocol document will be uploaded to enable full methodological transparency.

Drug and device information, study documents

Studies a U.S. FDA-regulated drug product

No

Studies a U.S. FDA-regulated device product

No

This information was retrieved directly from the website clinicaltrials.gov without any changes. If you have any requests to change, remove or update your study details, please contact register@clinicaltrials.gov. As soon as a change is implemented on clinicaltrials.gov, this will be updated automatically on our website as well.

Clinical Trials on Brain Cancer

Clinical Trials on Digital Pathology Evaluation

Search Similar Trials

Sponsors and Collaborators

Medical Conditions

Drug Interventions

Conditions

Rare Diseases

Drug Interventions

Dietary Supplements

Sponsor/Collaborators

Locations

Cookie Settings

We use cookies to ensure that we give you the best experience on our website. For more details carefully read our Privacy and Cookies Policy. ...>>>You can change your preferences or withdraw your consent at any time by deleting the cookies from your website or computer as described in the policy. Please accept it and choose your preferences by ticking the corresponding boxes in the "Manage Settings" section below. Please carefully read our Terms of Service before you proceed with using any part of this website.
«Manage Settings