Artificial Intelligence Augmented Training in Skin Cancer Diagnostics for General Practitioners (AISC-GP)

Artificial Intelligence Augmented Training of Danish General Practitioners in Skin Cancer Diagnostics - A Randomized Superiority Clinical Trial

Background:

The worldwide incidence of skin cancer has been rising for 50 years, in particular the incidence of malignant melanoma has increased approx. 2-7% annually and is the most common cancer amongst Danes aged 15-34. Currently there is a significant amount of misdiagnosis of skin cancer and mole cancer. Our aim is to improve general practitioners' diagnostic skills and accuracy of skin and mole cancer.

Research questions:

In a population of Danish General Practitioners (GPs) what is the dose/response effect of hours spent with an educational platform that offers AI augmented training and clinical feedback on their diagnostic accuracy and accurate clinical management (treatment, dismissal, referral)? Does access to an educational platform that offers AI augmented training and clinical feedback increase the number of malignant skin lesions referred by Danish GPs without simultaneously increasing the number of incorrect benign referrals? Can the participating GPs clinical accuracy be predicted from the MCQ-score by comparing their quiz answers and diagnostic accuracy on their registered lesions with their score on the MCQ?

Method:

90 Danish GPs will at baseline, 1 month and end of trial answer a Multiple Choice Questionnaire (MCQ). There is no change to current clinical practice, but all participating doctors will be asked to register a clinical picture and a dermoscopic image as well as basic information about the lesion and patient (age, gender, location and diagnosis) of all skin lesions examined due to a suspicion for non-melanoma or melanoma skin cancer, raised by the GP or patient.

GPs in the intervention group are besides the registration application (R-app) given access to an AI augmented training and clinical feedback through an educational smartphone app (E-app). Within the E-app the doctor can access quizzes on a library of 10,000+ skin lesions, written articles about the 40 most common skin lesions, and a clinical feedback module that gives the GP feedback on their registered skin lesions.

Feedback on skin lesions with the registered clinical management of referred/excised/biopsied will be provided continuously by independent experts in skin cancer diagnostics (>10 years of experience) through a web-based review system developed by our group. Feedback on the remaining registered cases are withheld until the end of the study period. This is done to simulate a realistic clinical setting during the study.

Study Overview

• Status: Completed
• Conditions: Melanoma; Skin Cancer
• Intervention / Treatment: Other: AI augmented training and clinical feedback

Detailed Description

Statistics and Results Using data from the E-app, scores on the MCQs and the diagnostic accuracy of the GP's a dose/response relationship is calculated, in order to answer the research questions posed above.

Aim The aim of this project is to investigate the dose/response effect on GPs' proficiency from AI augmented training and clinical feedback in skin cancer diagnostics. This project will examine how much training is needed before the GPs' ability to diagnose and correctly refer skin cancer is affected.

Research Questions In a population of Danish GPs what is the dose/response effect of hours spent with an educational platform that offers AI augmented training and clinical feedback on their diagnostic accuracy and accurate clinical management (treatment, dismissal, referral)?

Does access to an educational platform that offers AI augmented training and clinical feedback increase the number of malignant skin lesions referred by Danish general practitioners without simultaneously increasing the number of incorrect benign referrals? Can the participating GPs clinical accuracy be predicted from the MCQ-score by comparing their quiz answers and diagnostic accuracy on their registered lesions with their score on the MCQ?

Design This stratified superiority RCT will include 90 GPs in Denmark. The study period is five months. Three months of active intervention, then two months of post intervention observation period.

Participating GPs must give signed consent before receiving a short introductory course regarding the use of hard- and software. They are then asked to complete a short questionnaire regarding the use of the platform as well as completing a proficiency test using a multiple choice questionnaire (MCQ) at baseline and again after 1 and 3. The MCQ consists of 12 skin lesion cases randomly sampled from a test library developed and validated by our research group.

Stratification and randomization:

Eligible participants will be stratified based on gender, results on the pre-test, age, type of clinic and self-reported years of experience diagnosing skin lesions, before being randomized (allocation ratio 3:1) into either the intervention or control group. Intervention All participating doctors will be asked to register all skin lesions examined due to a suspicion for non-melanoma or melanoma skin cancer, raised by the GP or patient, throughout the study period (Figure 2). Skin lesions will be registered using the R-app that enables photo acquisition and registration of: patient social security number (CPR), location of lesion, tentative diagnosis and chosen clinical action (referral, excision, monitoring, none) ect.

The intervention consists of AI augmented training and clinical feedback through an educational smartphone app (E-app), whilst the control group registers the lesions but otherwise continues its standard clinical practice.

The E-app has two modules:

The training module includes AI enhanced case training on a library of 10,000+ benign and malignant skin lesion cases each coupled to written learning modules. Clinical feedback module gives the user diagnostic feedback on all cases registered in the registration module. Feedback during the trial will be based on either histopathology or the consensus agreement of domain experts (if no biopsy is taken). Feedback on referred or dismissed skin lesions will be provided by independent experts in skin cancer diagnostics ( >10 years of experience) through a web-based review system developed by our group. Statistics The mean improvement in clinical and MCQ-tested diagnostic accuracy measured will be compared between intervention and control groups using independent t-tests. Clinical accuracy will be determined using histopathology and/or expert consensus as gold standard. Both per-protocol and Intention To Treat analysis will be calculated in order to elucidate potentially missed differences and to allow readers to interpret the effect of the intervention.

Power and Sample Size To our knowledge there is no data within the literature that describes the typical number of skin lesions seen by a Danish GP. Based on estimates from clinical experience roughly eight patients are seen by each GP per week with skin lesions suspected of skin cancer. With 90 GPs registering 90% of all lesions this amounts to 8.417 skin lesions registered during the three months. However, the ratio of benign to malignant skin lesions (BMR) referred or removed by GPs is typically somewhere between 10-29:1. Specialised centers in the UK have reported that it is possible to achieve accurate melanoma diagnostics with a BMR of 1.74-6.3:1, based on clinical evaluation aided by dermatoscopy alone. Based on these numbers an effect size of Cohen's d = 0.5-0.8 is realistic. This would require around 25-40 GPs in each group to achieve a minimum 80% power.

Ethical considerations Patient participation contains no immediate strain or discomfort for the patient, and no change to current clinical practise, as dermoscopic evaluation is part of the clinical examination of skin lesions. The images captured are stored safely and anonymously with no risk for the patient.

Participating patients receive current standard of practice or an additional teledermatological evaluation of their skin lesions, giving them an expert opinion rather than only the opinion of their GP, in cases where patients are not referred to a dermatologist. A setup with minimal discomfort for the patient and the evaluation of their skin lesions by an expert justifies the participation of the patients.

With the current low diagnostic accuracy of GP's the educational nature of the intervention justifies the study for the sake of all future patients with skin lesions that are less likely to be misdiagnosed.

Educational interventions on doctors do not require approval by The National Committee on Health Research Ethics in Denmark. However, a waiver has been obtained from the The Committee on Health Research Ethics in the Capital Region of Denmark (case number H-20059977) and ethical considerations have been made and the project is in concordance with the Helsinki Declaration II.

• Study Type: Interventional
• Enrollment (Actual): 115
• Phase: Not Applicable

Contacts and Locations

Study Locations

• Denmark
  • Danmark
    • Valby, Danmark, Denmark, 2500
      • AISC Research Fascility

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: Child; Adult; Older Adult
• Accepts Healthy Volunteers: Yes

Description

Inclusion criteria:

• Participating doctors are required to have a danish doctors authorization and work at least 4 days a week in a general practitioners office.
• The participating doctors may register skin lesions on patients of all ages with skin lesions suspected of skin cancer. A skin lesion is for the purpose of this study defined as a mole or tumor that either patient or GP raises suspicion of skin cancer about.

Exclusion criteria:

• Doctors that have participated in a small qualitative pilot study are excluded.

Study Plan

How is the study designed?

Design Details

• Primary Purpose: Diagnostic
• Allocation: Randomized
• Interventional Model: Parallel Assignment
• Masking: Single

Number of Arms

2

Arms and Interventions

Participant Group / Arm	Intervention / Treatment
Experimental: Intervention; During the three months the intervention group will receive access to the AI augmented digital educational platform and its two modules (Training Module and Clinical Feedback Module). They will receive continuous clinical feedback on their registered lesions.	Other: AI augmented training and clinical feedback; The educational platform has two modules: The training module includes AI enhanced case training on a library of 10,000+ benign and malignant skin lesion cases each coupled to written learning modules. Participants will be able to track their progression through automatically generated performance statistics and discuss difficult cases with peers within the application. Clinical feedback is defined as diagnostic feedback on all cases registered in the registration module. Feedback during the trial will be based on either histopathology or the consensus agreement of domain experts (if no biopsy is taken). Feedback on referred or dismissed skin lesions will be provided by independent experts in skin cancer diagnostics ( >10 years of experience) through a web-based review system developed by our group.
No Intervention: Control; The control group continues its standard clinical practice without access to the E-app, but does register skin lesions throughout the full 3 month period.

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Time spent on educational materials	The participants time spend with the digital educational platform is measured by the platform.	Participants are assessed over a period of 5 months.
Change in diagnostic accuracy	The change in diagnostic accuracy is measured as percentage of correctly diagnosed skin lesions.	Participants are assessed over a period of 5 months.
Benign/Malignant Ratio	All registered lesions will be evaluated by a expert dermatologist. Using the expert opinion as golden standard a ratio of the benign and malignant skin lesions forwarded by the General Practitioner is calculated.	Participants are assessed over a period of 5 months.

Secondary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Multiple-Choice-Questionnaire predictability of diagnostic accuracy	Participants will answer a 12-item multiple choice test at baseline, after 1 and 3 months. The multiple-choice-questionnaire is validated and descriped here: https://doi.org/10.1007/s00403-020-02097-8 A correlation between the participants MCQ score and their clinical diagnostic accuracy is calculated. A high score correlates to high diagnostic accuracy (better).	Participants are assessed over a period of 5 months.

Collaborators and Investigators

• Sponsor: Herlev Hospital
• Investigators: Principal Investigator: Gustav G Nervil, MD, Research Unit of Plastic Surgery, Herlev Hospital

Study record dates

Study Major Dates

• Study Start (Actual): November 1, 2021
• Primary Completion (Actual): January 15, 2022
• Study Completion (Actual): January 31, 2022

Study Registration Dates

• First Submitted: September 24, 2020
• First Submitted That Met QC Criteria: October 5, 2020
• First Posted (Actual): October 6, 2020

Study Record Updates

• Last Update Posted (Actual): July 25, 2023
• Last Update Submitted That Met QC Criteria: July 24, 2023
• Last Verified: July 1, 2023

More Information

Terms related to this study

• Keywords: Skin Cancer; Artificial Intelligence; Education; Diagnostic Accuracy
• Additional Relevant MeSH Terms: Skin Diseases; Neoplasms; Neoplasms by Site; Skin Neoplasms
• Other Study ID Numbers: AISC-GP

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No
• product manufactured in and exported from the U.S.: No