Free Text Prediction Algorithm for Appendicitis

Prospective Study of a Free-text Diagnosis Prediction Algorithm for Appendicitis in the Emergency Department

Computer-aided diagnostic software has been used to assist physicians in various ways. Text-based prediction algorithms have been trained on past medical records through data mining and feature analysis. Currently, all text-based machine learning prediction problem models have been built on extracted data with no research completed on free text based prediction algorithms. This study aims to determine the accuracy of a free text prediction algorithm in predicting the probability of appendicitis in patients presenting to the Emergency Department with abdominal pain and gastrointestinal symptoms.

Study Overview

• Status: Completed
• Conditions: Abdominal Pain; Acute Appendicitis
• Intervention / Treatment: Diagnostic test: Free text prediction algorithm for appendicitis

Detailed Description

Developing machine learning models that have a strong prediction power for diagnosis of appendicitis from physician entered free text input can improve diagnostic accuracy of doctors. It also offers the possibility of using prediction algorithms to improve routine clinical care. In the future, multiple machine learning models can be combined to increase prediction accuracy and prediction algorithms can be extended to other diagnoses.

18,000 cases of emergency department presentations over 10 years were used as a training and validation dataset. To develop the appendicitis prediction model, deep learning neural networks with a customized medical ontology were used. The diagnostic accuracy of the model is expressed as sensitivity (recall), specificity and F1 score (harmonic mean). The developed diagnosis predictive model shows high sensitivity (86.3%), specificity (91.9%) and F1 score (88.8) in diagnosing appendicitis from patients presenting with abdominal pain.

The predictive model algorithm will also highlight words in the free text (entered by the attending physician) that it assigns higher probability for predicting an outcome. The doctors will be instructed to provide a percentage likelihood of appendicitis based on the clinical presentation and any available laboratory investigations. The doctor is then shown the prediction of the algorithm as well as the highlighted words for the patient entered. He/she must then provide another prediction of the likelihood of appendicitis after seeing the algorithm generated prediction.

The aim is to evaluate the performance of the algorithm and to assess if usage of the algorithm is able to help emergency doctors improve their diagnosis of appendicitis. The prediction results will be tabulated to assess accuracy of the algorithm, doctors before algorithm input and doctors after receiving algorithm input. The accuracy will be expressed as sensitivity, specificity, accuracy, positive prediction value, F1 score and F0.5 score.

Approximately 100 emergency doctors will be recruited over the course of 1 year as participants in the study. The doctors will be split randomly assigned to two groups - the algorithm arm and the no algorithm arm. The randomization will be by time (weekly) using variable block randomization of 4 and 6. The patients will be followed up for the final discharge diagnoses.

• Study Type: Observational
• Enrollment (Actual): 689

Contacts and Locations

Study Locations

• Singapore
  • Singapore, Singapore, 119074
    • National University Hospital

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: 21 years to 99 years (ADULT, OLDER_ADULT)
• Accepts Healthy Volunteers: No
• Genders Eligible for Study: All

• Sampling Method: Non-Probability Sample

Study Population

Attending physicians will be recruited as study participants and randomised weekly into "algorithm use" versus "no algorithm use".

Patients who fulfilled the above eligibility criteria will have their data collected and entered into the predictive algorithm.

Description

Eligibility criteria of doctors- Inclusion criteria: Junior doctors working in the Emergency Department Exclusion criteria: Refusal of consent

Eligibility criteria of patients-

Inclusion Criteria:

• Presence of abdominal pain, OR
• Presence of gastrointestinal symptoms such as nausea, vomiting or diarrhea, OR
• Fever with anorexia

Exclusion Criteria:

• Previous history of appendicectomy
• Refusal of consent

Study Plan

How is the study designed?

Number of groups / cohorts

2

Cohorts and Interventions

Group / Cohort	Intervention / Treatment
With algorithm use	Diagnostic test: Free text prediction algorithm for appendicitis; A free-text prediction software that predicts the probability of acute appendicitis
No algorithm use

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Accuracy of predictive algorithm for acute appendicitis	Accuracy of predictive algorithm and accuracy of doctors with input from the algorithm in diagnosing acute appendicitis	30 days

Collaborators and Investigators

• Sponsor: National University Hospital, Singapore
• Investigators: Principal Investigator: Kee Yuan Ngiam, Dr, National University Hospital, Singapore

Study record dates

Study Major Dates

• Study Start (ACTUAL): December 4, 2017
• Primary Completion (ACTUAL): July 1, 2019
• Study Completion (ACTUAL): July 1, 2020

Study Registration Dates

• First Submitted: January 23, 2018
• First Submitted That Met QC Criteria: January 23, 2018
• First Posted (ACTUAL): January 30, 2018

Study Record Updates

• Last Update Posted (ACTUAL): March 3, 2021
• Last Update Submitted That Met QC Criteria: March 2, 2021
• Last Verified: March 1, 2021

More Information

Terms related to this study

• Keywords: Appendicitis, abdominal pain, free text prediction
• Additional Relevant MeSH Terms: Digestive System Diseases; Infections; Pain; Neurologic Manifestations; Signs and Symptoms, Digestive; Gastrointestinal Diseases; Gastroenteritis; Intestinal Diseases; Cecal Diseases; Intraabdominal Infections; Abdominal Pain; Appendicitis
• Other Study ID Numbers: N-171-000-456-001

Plan for Individual participant data (IPD)

• Plan to Share Individual Participant Data (IPD)?: NO
• IPD Plan Description: Individual participant data will not be made available to other researchers.

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No