Assess the Clinical Effectiveness in AI Prioritising CT Heads (ACCEPT)

A Mixed Methods Study to Assess the Clinical Effectiveness and Acceptability of qER Artificial Intelligence Software to Prioritise CT Head Interpretation.

Non-Contrast Computed Tomography (NCCT) of the head is the most common imaging method used to assess patients attending the Emergency Department (ED) with a wide range of significant neurological presentations including trauma, stroke, seizure and reduced consciousness. Rapid review of the images supports clinical decision-making including treatment and onward referral.

Radiologists, those reporting scans, often have significant backlogs and are unable to prioritise abnormal images of patients with time critical abnormalities. Similarly, identification of normal scans would support patient turnover in ED with significant waits and pressure on resources.

To address this problem, Qure.AI has worked to develop the market approved qER algorithm, which is a software program that can analyse CT head to identify presence of abnormalities supporting workflow prioritisation.

This study will trial the software in 4 NHS hospitals across the UK to evaluate the ability of the software to reduce the turnaround time of reporting scans with abnormalities that need to be prioritised.

Study Overview

• Status: Recruiting
• Conditions: Stroke, Ischemic; Skull Fractures; Trauma, Brain; Encephalomalacia; Mass Lesion; Hemorrhage, Intracranial; Midline Deviation
• Intervention / Treatment: Device: qER (qER EU 2.0)

Detailed Description

Study background:

Emergency Departments (ED) across the UK are overburdened with increasing patient demand, radiology staff shortages and rising patient wait times. Head injuries are a frequent cause of emergency attendance in the UK with computed tomography(CT) scans usually the first imaging tests to diagnose head injuries and strokes.

A report issued by National Institute of Health and Care Excellence (NICE), confirms that each year 1.4 million people attend emergency departments in England and Wales with head injury. Among the 200,000 patients admitted annually, one-fifth of them suffer from a Traumatic Brain Injury with skull fracture or evidence of brain damage. Head injury is the most common cause of death and disability in people up to the age of 40. Early detection and prompt treatment is vital to save lives and minimise risk of disability, according to the NICE guidelines of Head injury: assessment and early management. Head CT scans are the gold standard for diagnosing these and it is critical that these are performed and reported by Radiologists in line with NICE guidelines.

The potential applications of AI in radiology go well beyond image analysis for diagnostic and prognostic opportunities. It is becoming increasingly clear that AI algorithms have the potential to improve productivity, operational efficiency, and accuracy in diagnostic radiology. AI tools are being developed to aide diagnosis and enhance processes at multiple point in the radiology workflow including:

(a) protocolling the prioritised scan,(b) clinical decision support systems for detection of critical findings, (c) worklist priority adjustment via AI results, and (d) reducing turnaround time through worklist prioritisation and semiautomated structures reporting. The adoption of AI tools is dependent on the demonstration of a tangible effect on patient care and improvement in radiologist workflow.

Thus, in this study, we aim to assess whether real-world implementation of an AI tool which augments (b), (c) and (d) of the imaging life cycle would affect turnaround times.

qER medical device:

qER, a CE Class II approved medical software device, detects, and localizes the presence of six target abnormalities - intracranial haemorrhage, cranial fracture, midline shift, mass effect, atrophy and hypodensities suggestive of infarcts in non-contrast Head-CT scans. A priority status is assigned if any one of the target abnormalities (intracranial haemorrhage, cranial fracture, midline shift or mass effect) is detected by the software, and the user will be able to view a single summary slice listing all the target abnormalities found by qER on the CT scan followed by all slices in scan with the overlay of above abnormalities localization. Alternately, if none of the target abnormalities are detected, the output will indicate that the software has analysed the image and identified no critical findings. qER reports are intended to support certified radiologists and/or licensed medical practitioners for clinical decision making. It is a support tool and, when used with original scans, can assist the clinician to improve efficiency, accuracy, and turnaround time in reading head CTs. It is not to be used to provide medical advice, determine treatment plan, or recommend a course of action to the patient.

Study design:

A multi-centre stepped wedged cluster randomised study will be conducted in 4 NHS hospitals over a 13-month period. Hospitals will be identified and initiated into the qER solution with a 30-day implementation period. The order in which sites will receive the qER intervention will be determined by computer-based randomisation. The stepped wedge design allows delivery of the intervention at an organisational level with evaluation of outcome measures at a patient level. Structuring the implementation through a staged activation in a random order provides important methodological advantages for both qualitative and quantitative elements of the study. The design allows control of adoption bias and adjust for time-based changes in the background patient characteristics at a patient level.

All patients under this pathway would receive an AI reading, and no additional or different tests will be performed as a result of the AI findings. The turnaround time will be the interval between the time the scan was taken to the time when the final scan report becomes available and will be measured in minutes. When qER assistance is used for reporting Head-CT scans and if there is a difference between the output of the qER and the radiologist, the latter will be considered as final for further patient management.

Primary objective:

The primary objective is to assess if qER based reporting and triage significantly reduce turnaround time (TAT) of critical NCCT head reporting for patients attending the emergency department.

Secondary objective(s):

• To assess utility of qER to support emergency department pathways for patients requiring NCCT head and radiology reporting workflow.
• To assess the safety of qER at identifying patients with critical findings on NCCT heads.
• To evaluate the technical performance of qER.
• To conduct a Heath Economic, cost utility analysis of qER.

• Study Type: Observational
• Enrollment (Estimated): 16800

Contacts and Locations

• Study Contact: Name: Haris Shuaib, MSc; Phone Number: 020 7188 7188; Email: haris.shuaib@gstt.nhs.uk
• Study Contact Backup: Name: Kavitha Vimalesvaran, MBBS MSc; Phone Number: 020 7188 7188; Email: kavitha.vimalesvaran@gstt.nhs.uk

Study Locations

• United Kingdom
  • Glasgow, United Kingdom
    • Recruiting
    • NHS Greater Glasgow and Clyde
    • Contact: David J Lowe; Phone Number: 0141 201 4444; Email: david.lowe@ggc.scot.nhs.uk
    • Contact: Mark Hall; Phone Number: 0141 201 4444; Email: mark.hall2@ggc.scot.nhs.uk
    • Principal Investigator: David J Lowe
  • London, United Kingdom, SE1 7EH
    • Recruiting
    • Guy's and St.Thomas Trusts
    • Contact: Mariapaola Narbone; Phone Number: 020 7188 8888; Email: mariapaola.narbone@gstt.nhs.uk
    • Contact: Kavitha Vimalesvaran; Phone Number: 020 7188 8888; Email: kavitha.vimalesvaran@gstt.nhs.uk
    • Principal Investigator: Mariapaola Narbone
  • Northumberland, United Kingdom, NE27 0QJ
    • Recruiting
    • Northumbria Healthcare NHS Foundation Trust
    • Contact: Rahul Dharmadhikari; Phone Number: 0344 811 8118; Email: rahul.dharmadhikari@northumbria-heatlthcare.nhs.uk
    • Principal Investigator: Rahul Dharmadhikari
  • Oxford, United Kingdom, OX3 9DU
    • Recruiting
    • Oxford University Hospitals
    • Contact: Sarim Ather; Phone Number: 0300 304 7777; Email: sarim.ather@ouh.nhs.uk
    • Contact: Alex Novak; Phone Number: 0300 304 7777; Email: alex.novak@ouh.nhs.uk
    • Principal Investigator: Sarim Ather

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: Adult; Older Adult
• Accepts Healthy Volunteers: N/A

• Sampling Method: Non-Probability Sample

Study Population

At each of the four participating sites, we will identify all patients referred through the Emergency Department NCCT requests.

Description

Inclusion Criteria:

• Individuals undergoing Head CT scan at the ED / A&E (Accident and Emergency Services).
• Non-contrast axial CT scan series with consistently spaced axial slices.
• Soft reconstruction kernel covering the complete Brain.
• Maximum slice thickness of 6mm.

Exclusion Criteria:

There are no explicit exclusion criteria for qER as all scans in inclusion criteria will be processed by qER. Exclusion criteria are implicit within the inclusion criteria listed above.

Study Plan

How is the study designed?

Number of groups / cohorts

2

Cohorts and Interventions

Group / Cohort	Intervention / Treatment
Pre-implementation of qER; Baseline data: During the pre-implementation phase, we will be gathering data around the technical requirements for integrating qER into the radiology workflow. A random sample of 500 scans per site will be sent for the ground-truthing process for the purpose of technical evaluation. We will also be collecting data on the baseline status of all the endpoints including TAT. The reporting of NCCT scans will follow the same workflow as the current standard of care (i.e., the images/cases will appear in the RIS chronologically and the radiologist either follows this order or prioritises some cases based on communication from ED).
Post-implementation of qER; Post-implementation (Trial Intervention) In the post-implementation phase, there will be a notification (prioritised flag) in RIS. The order of the cases in RIS will not be altered. When the radiologist clicks a case in RIS, a secondary capture of qER along with the original images will be available in PACS. This secondary capture will have a contour showing the algorithm's attention point for a specific abnormality. The radiologist can then choose to agree with qER findings as it is or modify or ignore it according to their clinical judgement, writing and finally signing off the report. For scans which were not processed by qER the radiologist can prioritise and report as per the standard of care.	Device: qER (qER EU 2.0); Qure.ai's emergency room software solution qER (qER EU 2.0) is an AI medical device, developed by training a deep-learning algorithm using over 300,000 scans labelled by expert radiologists. qER has been shown to be accurate in identifying a range of abnormalities in NCCT head scans as well as prioritising them for urgent review and radiologist reporting. It is designated as a clinical support tool and, when used with original scans, can assist the clinician to improve efficiency, accuracy, and turnaround time in reading head CTs.

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Reporting turnaround time with qER prioritisation	Time taken to report NCCT head from acquisition for patients with prioritised findings in Emergency Department compared to standard of care. Measured as time in minutes from the scan acquisition to the final radiology report of prioritised scans.	1 year

Secondary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Assess the safety of qER	Mortality within 30 days of Emergency Department discharge. Measured by the number of study participants who upon discharge from the impatient setting are dead within 30 days.	1 year
Reporting turnaround time with qER prioritisation for scans without prioritised findings in Emergency Department compared to standard of care.	Measured as time in minutes from the scan acquisition to the final radiology report of scans without prioritised findings.	1 year
Reporting turnaround time with qER prioritisation for scans with an absence of findings in Emergency Department compared to standard of care.	Measured as time in minutes from the scan acquisition to the final radiology report of scans with an absence of findings.	1 year
Assess the impact of qER on radiology reporting workflow on other requests for CT scans.	To assess how this impacts the types of CT scans requested from the Emergency Department.	1 year
Impact of qER supported reporting on teleradiology.	Time taken to report NCCT head from acquisition for patients in Emergency Department compared to standard of care where teleradiology is used.	1 year
Assess utility of qER to support clinical decision making of the patients from the emergency department requiring an NCCT.	Time to diagnosis from NCCT acquisition. Measured as time in hours for the electronic record of diagnosis assigned in the emergency department.	1 year
Assess utility of qER to support referral or discharge of the patients from the emergency department requiring an NCCT.	Time to referral or discharge designation from NCCT acquisition. Measured as time in hours to the electronic record of referral or discharge after diagnosis in the emergency department.	1 year
Time to initiation of treatment from NCCT acquisition for prioritised scans.	Time to initiation of treatment from NCCT acquisition. Measured as time in hours to the electronic record of the initiation of treatment in the emergency department.	1 year
Death within 28 days of NCCT head acquisition.	Number of patients presenting via the emergency department who then had a NCCT and died 28 days after their scan.	1 year
Percentage of NCCT heads that qER classifies as prioritised, non-prioritised and absence of findings.	Number of scans that qER identifies as prioritised, non-prioritised and absence of findings in accordance with the target abnormality definitions.	1 year
Percentage of qER non-prioritised scans but identified by the radiologist as prioritised.	Accuracy of qER in classifying non-prioritised scans in comparison to a radiologist. Rate of accurate classification of non-prioritised scans compared with the final radiology report.	1 year
Percentage of qER non-prioritised scans but identified by the radiologist as absence of finding.	Accuracy of qER in classifying scans with an absence of findings. Rate of accurate classification of scans with an absence of findings compared with the final radiology report.	1 year
Technical evaluation of product performance.	Sensitivity, specificity, positive and negative predictive values of qER in detecting scans with prioritised findings overall and stratified by all six target abnormalities.; Percentage of CT scans that could not be processed by qER due to technical factors.	1 year
Health Economic Assessment	To compare costs and health benefits between pre- and post-implementation of qER, including cost evaluation of fully automatic diagnosis of high confidence normal triage.	1 year

Collaborators and Investigators

• Sponsor: Guy's and St Thomas' NHS Foundation Trust
• Collaborators: Oxford University Hospitals NHS Trust; NHS Greater Glasgow and Clyde; Northumbria Healthcare NHS Foundation Trust; Qure.ai
• Investigators: Study Chair: Haris Shuaib, MSc, Guy's and St.Thomas' Hospitals

Study record dates

Study Major Dates

• Study Start (Estimated): March 27, 2024
• Primary Completion (Estimated): August 1, 2024
• Study Completion (Estimated): August 1, 2024

Study Registration Dates

• First Submitted: August 11, 2023
• First Submitted That Met QC Criteria: August 30, 2023
• First Posted (Actual): September 7, 2023

Study Record Updates

• Last Update Posted (Actual): March 21, 2024
• Last Update Submitted That Met QC Criteria: March 19, 2024
• Last Verified: March 1, 2024

More Information

Terms related to this study

• Additional Relevant MeSH Terms: Pathologic Processes; Cardiovascular Diseases; Vascular Diseases; Cerebrovascular Disorders; Brain Diseases; Central Nervous System Diseases; Nervous System Diseases; Fractures, Bone; Wounds and Injuries; Craniocerebral Trauma; Trauma, Nervous System; Stroke; Brain Injuries; Ischemic Stroke; Hemorrhage; Brain Injuries, Traumatic; Intracranial Hemorrhages; Skull Fractures; Encephalomalacia
• Other Study ID Numbers: 313507

Plan for Individual participant data (IPD)

• Plan to Share Individual Participant Data (IPD)?: NO

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: Yes
• product manufactured in and exported from the U.S.: No