AI Assisted Reader Evaluation in Acute Computed Tomography (CT) Head Interpretation (AI-REACT)

November 6, 2023 updated by: Alex Novak, Oxford University Hospitals NHS Trust

AI Assisted Reader Evaluation in Acute CT Head Interpretation

This study has been added as a sub study to the Simulation Training for Emergency Department Imaging 2 study (ClinicalTrials.gov ID NCT05427838).

The purpose of the study is to assess the impact of an Artificial Intelligence (AI) tool called qER 2.0 EU on the performance of readers, including general radiologists, emergency medicine clinicians, and radiographers, in interpreting non-contrast CT head scans. The study aims to evaluate the changes in accuracy, review time, and diagnostic confidence when using the AI tool. It also seeks to provide evidence on the diagnostic performance of the AI tool and its potential to improve efficiency and patient care in the context of the National Health Service (NHS). The study will use a dataset of 150 CT head scans, including both control cases and abnormal cases with specific abnormalities. The results of this study will inform larger follow-up studies in real-life Emergency Department (ED) settings.

Study Overview

Status

Active, not recruiting

Conditions

Intervention / Treatment

Study Type

Observational

Enrollment (Actual)

33

Contacts and Locations

This section provides the contact details for those conducting the study, and information on where this study is being conducted.

Study Contact

Study Contact Backup

Study Locations

  • United Kingdom
      • Glasgow, United Kingdom, G12 0XH
        • NHS Greater Glasgow and Clyde
      • London, United Kingdom, SE1 7EH
        • Guy's & St Thomas NHS Foundation Trust
      • Newcastle Upon Tyne, United Kingdom, NE27 0QJ
        • Northumbria Healthcare NHS Foundation Trust
    • Oxfordshire
      • Oxford, Oxfordshire, United Kingdom, OX3 9DU
        • Oxford University Hospitals NHS Foundation Trust

Participation Criteria

Researchers look for people who fit a certain description, called eligibility criteria. Some examples of these criteria are a person's general health condition or prior treatments.

Eligibility Criteria

Ages Eligible for Study

  • Child
  • Adult
  • Older Adult

Accepts Healthy Volunteers

Yes

Sampling Method

Non-Probability Sample

Study Population

Setting:

Readers will be recruited from the following four hospital Trusts (secondary and tertiary level:

  • Guy's & St Thomas NHS Foundation Trust
  • Northumbria Healthcare NHS Foundation Trust
  • NHS Greater Glasgow and Clyde
  • Oxford University Hospitals NHS Foundation Trust

Participants:

30 volunteer participant readers will be selected from the following groups:

  • Emergency Medicine Consultants and Registrars (5 Consultant, 5 Registrar (ST3-6), 5 junior (F1-ST2)
  • General Radiologist Consultants and Registrars (5 Consultant, 5 Registrar)
  • 5 CT Radiographers

Description

Inclusion Criteria:

  • Radiologists/Radiographers/ED clinicians who review CT head scans as part of their clinical practice

Exclusion Criteria:

  • Neuroradiologists.
  • Non-radiologist groups: Clinicians with previous formal postgraduate CT reporting training
  • Emergency Medicine group: Clinicians with previous career in radiology/neurosurgery to registrar level

Study Plan

This section provides details of the study plan, including how the study is designed and what the study is measuring.

How is the study designed?

Design Details

Cohorts and Interventions

Group / Cohort
Intervention / Treatment
Readers
30 readers will be recruited across four NHS trusts including ten general radiologists, fifteen emergency medicine clinicians, and five CT radiographers of varying seniority. Readers will interpret each scan first without, then with, the assistance of the AI tool, with an intervening 4-week washout period. Using a panel of neuroradiologists as ground truth, the stand-alone performance of qER will be assessed, and its impact on the readers' performance will be analysed as change in accuracy, mean review time per scan, and self-reported diagnostic confidence. Subgroup analyses will be performed by reader professional group, reader seniority, pathological finding, and neuroradiologist-rated difficulty.
Other: Reading
All 30 readers will review all 150 cases, in each of two study phases. The readers will provide their opinion on the presence or absence of some acute abnormalities, including intracranial haemorrhage, infarct, midline shift and fracture. They will provide a confidence in their diagnosis (10-point visual analogue scale), and a single click point to mark the location of each abnormality that they consider as being present. The time taken for each scan will be automatically recorded.
Ground truthers
Two Consultant neuroradiologists will independently review the images to establish the 'ground truth' findings on the CT scans which will be used as the reference standard. In the case of disagreement, a third senior neuroradiologist's opinion will be sought for arbitration. A difficulty score will be assigned to each scan by the ground truthers using a 5-point Likert scale.
Other: Ground truthing
Two Consultant neuroradiologists will independently review the images to establish the 'ground truth' findings on the CT scans which will be used as the reference standard. In the case of disagreement, a third senior neuroradiologist's opinion will be sought for arbitration.

What is the study measuring?

Primary Outcome Measures

Outcome Measure
Measure Description
Time Frame
Reader performance: Sensitivity, specificity, comparative between with and without AI assistance.
Time Frame: During 6 weeks, which is the period for reading or reviewing the cases/scans.
Reader performance will be evaluated as sensitivity, specificity, with and without AI assistance.
During 6 weeks, which is the period for reading or reviewing the cases/scans.
Reader performance: Positive and negative predictive value, comparative between with and without AI assistance.
Time Frame: During 6 weeks, which is the period for reading or reviewing the cases/scans.
Reader performance will be evaluated as Positive Predictive Value (PPV) and negative predictive value (NPV), with and without AI assistance.
During 6 weeks, which is the period for reading or reviewing the cases/scans.
Reader performance: Area Under Receiver Operating Characteristic Curve (AUROC), comparative between with and without AI assistance.
Time Frame: During 6 weeks, which is the period for reading or reviewing the cases/scans.
Reader performance will be evaluated as Area Under Receiver Operating Characteristic Curve (AUROC), with and without AI assistance.
During 6 weeks, which is the period for reading or reviewing the cases/scans.
Reader speed: Mean time taken to review a scan, with versus without AI assistance.
Time Frame: During 6 weeks, which is the period for reading or reviewing the cases/scans.
Reader speed will be evaluated as the man time taken to review a scan, using time unite of seconds.
During 6 weeks, which is the period for reading or reviewing the cases/scans.
Reader confidence: Self-reported diagnostic confidence on a 10 point visual analogue scale, with vs without AI assistance.
Time Frame: During 6 weeks, which is the period for reading or reviewing the cases/scans.
On the reading platform (RAIQC), one of the questions asks the level of confidence that the participant has in their diagnostic opinion. The question offers a scale of 1 to 10, where 1 is not confident, and 10 is highly confident.
During 6 weeks, which is the period for reading or reviewing the cases/scans.
qER (AI algorithm) performance: Sensitivity and specificity
Time Frame: During 6 weeks, which is the period for reading or reviewing the cases/scans.
qER performance will be evaluated as sensitivity, specificity.
During 6 weeks, which is the period for reading or reviewing the cases/scans.
qER (AI algorithm) performance: Positive and negative predictive value.
Time Frame: During 6 weeks, which is the period for reading or reviewing the cases/scans.
qER performance will be evaluated as Positive Predictive Value (PPV) and negative predictive value (NPV).
During 6 weeks, which is the period for reading or reviewing the cases/scans.
qER (AI algorithm) performance: Area Under Receiver Operating Characteristic Curve (AUROC).
Time Frame: During 6 weeks, which is the period for reading or reviewing the cases/scans.
qER performance will be evaluated as Area Under Receiver Operating Characteristic Curve (AUROC)
During 6 weeks, which is the period for reading or reviewing the cases/scans.

Collaborators and Investigators

This is where you will find people and organizations involved with this study.

Sponsor

Oxford University Hospitals NHS Trust

Investigators

  • Principal Investigator: Alex Novak, MSc, National Health Services in the United Kingdom (NHS UK)
  • Principal Investigator: Sarim Ather, PhD, National Health Services in the United Kingdom (NHS UK)

Publications and helpful links

The person responsible for entering information about the study voluntarily provides these publications. These may be about anything related to the study.

General Publications

Helpful Links

Study record dates

These dates track the progress of study record and summary results submissions to ClinicalTrials.gov. Study records and reported results are reviewed by the National Library of Medicine (NLM) to make sure they meet specific quality control standards before being posted on the public website.

Study Major Dates

Study Start (Actual)

June 1, 2023

Primary Completion (Actual)

September 1, 2023

Study Completion (Estimated)

September 1, 2024

Study Registration Dates

First Submitted

July 25, 2023

First Submitted That Met QC Criteria

August 25, 2023

First Posted (Actual)

August 30, 2023

Study Record Updates

Last Update Posted (Actual)

November 8, 2023

Last Update Submitted That Met QC Criteria

November 6, 2023

Last Verified

August 1, 2023

More Information

Terms related to this study

Keywords

Additional Relevant MeSH Terms

Other Study ID Numbers

  • 310995 - A

Drug and device information, study documents

Studies a U.S. FDA-regulated drug product

No

Studies a U.S. FDA-regulated device product

No

This information was retrieved directly from the website clinicaltrials.gov without any changes. If you have any requests to change, remove or update your study details, please contact register@clinicaltrials.gov. As soon as a change is implemented on clinicaltrials.gov, this will be updated automatically on our website as well.

Clinical Trials on Cerebral Infarction

Clinical Trials on Reading

Search Similar Trials

Sponsors and Collaborators

Medical Conditions

Drug Interventions

CROs by country

CROs in Jamaica

Conditions

Rare Diseases

Drug Interventions

Dietary Supplements

Sponsor/Collaborators

Locations

3
Subscribe