Effect of Intelligent Tutor Induced Pausing on Learning Simulated Surgical Skills

Effect of Intelligent Tutor Induced Pausing on Learning Simulated Surgical Skills Among Medical Students: A Randomized Controlled Trial

Traditional training of surgical technical skills relies on mentorship from experienced surgeons, who continuously evaluate and change trainee performance to prevent errors and potential patient harm by providing verbal instructions. These educators may also pause the procedure, explaining the risks associated with the trainee's actions, and may personally demonstrate proper techniques to the students. Studies examining pausing while providing medical care outline that these approaches allow for learning.

An artificial intelligent (AI) tutoring system, the Intelligent Continuous Expertise Monitoring System (ICEMS), improves learning in a surgical simulated operation by providing trainees with verbal instructions upon error identification. However, the effect of including a pause during this AI teaching has not been studied. Therefore, the ICEMS post-error identification methodology has been altered to include a pause with the intelligent tutor voice instruction.

The aim of this study is to determine the effect of pausing on surgical skill acquisition and transfer among pre-medical and medical students. This will be done by comparing their performance in repeated simulated tumour resection tasks.

Study Overview

• Status: Recruiting
• Conditions: Surgical Education
• Intervention / Treatment: Behavioral: Experimental Group ICEMS verbal feedback with pause group; Behavioral: Experimental Group ICEMS audio feedback with pause and expert video

Detailed Description

Background: Surgical skill assessment is shifting from a quantitative, time-based approach towards a qualitative evaluation of a trainee's competency. During surgical procedures, instructors continuously monitor trainee performance and utilize various teaching methods focused on enhancing acquisition of surgical skills. One such method includes pausing the operation, either to outline the risks associated with the trainee's performance or to personally demonstrate the best practice technique(s). Pausing in such situations has been shown to allow learners to re-assess best practice, interrupt negative momentum, and allow for learning. Specifically, pausing after an error can prevent introduction of new information that may affect one's ability to reflect on their error and reduce stress before continuing.

Rationale: The ICEMS, an AI tutoring system, was developed by our group using a Long-Short Term Memory deep learning algorithm to assess surgical performance and provide guidance. This was then integrated with the NeuroVR simulation platform. Using this AI system, the provision of verbal feedback on error identification demonstrated the potential of intelligent tutoring to improve learning in two previous randomized control trials (RCTs). However, these RCTs did not incorporate pausing methodology post-error identification. To further emulate the mentorship of an experienced surgeon in a clinical setting, the ICEMS platform has been modified to both initiate pausing when learner error is identified and provide a video demonstrating expert performance.

Research aims: To compare the effect of incorporating a pause after intelligent tutor instruction to intelligent tutor instruction alone on medical and pre-medical students' surgical skill acquisition and skill transfer.

Hypotheses:

1. The pause with video group will significantly improve the composite score between the first and sixth repetition of the practice scenario.
2. The pause with video group will have a composite score statistically higher than the control group in the sixth repetition of the practice scenario.
3. The pause with video group will have a composite score statistically non-inferior to the pause without video group in the sixth repetition of the practice scenario.
4. The pause with video group will have a global OSATS score statistically higher than the control group in the realistic scenario.
5. The pause with video group will have a global OSATS score statistically non-inferior to the pause without video group in the realistic scenario.
6. The pause with video group will not elect a difference in emotional stress or cognitive load compared to the control group.

Specific objectives:

1. To assess if the efficacy of AI mediated real-time tailored feedback combined with pausing methodology is statistically superior to AI mediated real-time tailored feedback alone in improving medical students' surgical skills on two virtual reality surgery tasks.
2. To determine if different emotions and cognitive load are elicited by the pausing methodology as compared to only hearing the AI mediated feedback.

Design: A three-arm single blinded randomized controlled trial of AI feedback with pausing methodology and an expert demonstration video versus AI feedback with only pausing methodology versus AI feedback alone.

Setting: Neurosurgical Simulation and Artificial Intelligence Learning Centre.

Participants: Students who are enrolled in a Quebec medical school in a preparatory year, and first and second year.

Task: Using the NeuroVR surgical simulator by CAE Healthcare, resect a simulated practice tumour six times and a complex simulated realistic brain tumour once using an Ultrasonic Aspirator and Bipolar pincers while minimizing bleeding and preserving the surrounding, simulated healthy brain structures.

Intervention: A 90-minute training session where participants will have seven simulated subpial tumour resection attempts (six repetitions of a simple practice scenario and one attempt at a complex realistic scenario). All participants will receive auditory feedback from the ICEMS but will differ in what follows:

1. Continuously perform the procedure (i.e., no pause) (Group 1);
2. A pause followed by a reflection period (Group 2);
3. A pause followed by an expert-level demonstration video and a reflection period (Group 3).

Auditory feedback will be based on 4 metrics:

1. Instrument tip separation distance;
2. Low bipolar force;
3. High aspirator force;
4. High bipolar force. Initially, feedback will only be given based on the first metric (instrument tip separation), but once a repetition is completed without receiving feedback, the subsequent repetition will assess the next metric in the list above, and so on and so forth.

Main outcomes and measures:

The two co-primary outcomes are:

1. The improvement in surgical performance, as dictated by the composite score computed by the previously validated evaluation module of the ICEMS. Performance improvement is measured by the difference in composite score between each of the 6 repetitions of the practice scenario. Transfer of learning will be measured by the participant's composite score for the complex realistic scenario.
2. The performance score of the participants in the complex realistic scenario, assessed by two blinded experts using the Objective Structured Assessment of Technical Skills (OSATS) global rating scale.

The secondary outcome is the differences in the strength of emotions elicited, measured before the practice scenario, immediately before the realistic scenario, and after completion of all attempts using the Duffy's Medical Emotional Scale (MES). Cognitive load will also be measured following completion of all tasks using Leppink's Cognitive Load Index (CLI). Both outcomes are measured using self-reports.

Statistical Analysis Plan: Participant data will be anonymized and stored. The ICEMS will assess the participant's surgical performance and provide a performance score at 0.2 second intervals throughout each repetition of the simulated surgical task. An average composite score will then be calculated for each repetition. Using ANCOVA, improvement in performance and participant learning will be assessed by comparing the composite score of the first practice scenario repetition (baseline) and the composite score of the sixth repetition (summative). Meanwhile, the composite score of the complex realistic scenario will be used to assess the transfer of learning using a one-way ANOVA. With an effect size of 0.25 and a significance of 0.05, a total sample size of 129 provides 80% power to detect a significant interaction.

Videos of participant performance in the complex realistic scenario will be evaluated by two blinded expert raters using the OSATS global rating scale. The OSATS score will be analyzed between groups using a one-way ANOVA to compare efficiency of learning and skill retention.

Emotional changes before, during, and after learning in the simulated scenarios will be evaluated using a two-way mixed ANOVA, while one-way ANOVA will be used to assess cognitive load after learning.

• Study Type: Interventional
• Enrollment (Estimated): 129
• Phase: Not Applicable

Contacts and Locations

• Study Contact: Name: Rolando F Del Maestro, MD, PhD; Phone Number: 519 708 0346; Email: rolando.del_maestro@mcgill.ca
• Study Contact Backup: Name: Vanja Davidovic; Phone Number: 6132964855; Email: vanja.davidovic@mail.mcgill.ca

Study Locations

• Canada
  • Quebec
    • Montreal, Quebec, Canada, H2X 4B3
      • Recruiting
      • Neurosurgical Simulation and Artificial Intelligence Learning Centre
      • Contact: Rolando F Del Maestro, MD, PhD; Phone Number: 519 708 0346; Email: rolando.del_maestro@mcgill.ca
      • Contact: Vanja Davidovic; Phone Number: 6132964855; Email: vanja.davidovic@mail.mcgill.ca

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: Adult; Older Adult
• Accepts Healthy Volunteers: Yes

Description

Inclusion Criteria:

• First and second year medical students who are actively enrolled in any Quebec institution who do not meet the exclusion criteria.
• Students actively enrolled in medical school in a preparatory year in any Quebec institution who do not meet the exclusion criteria.

Exclusion Criteria:

• Participation in previous trials involving the NeuroVR (CAE Healthcare) simulator

Study Plan

How is the study designed?

Design Details

• Primary Purpose: Health Services Research
• Allocation: Randomized
• Interventional Model: Parallel Assignment
• Masking: Double

Number of Arms

3

Arms and Interventions

Participant Group / Arm	Intervention / Treatment
No Intervention: Control Group Intelligent Continuous Expertise Monitoring System (ICEMS) only verbal feedback group; 43 participants. Individuals receive standard information. They perform 6 5-min practice scenario resections with a 5-min break between each one. The 7th attempt is the 13-min realistic scenario. Participants receive no feedback during the first repetition. They will then receive feedback on 4 metrics, one metric a time: instrument tip separation, low bipolar force, high aspirator force, high bipolar force. Once an attempt is completed without receiving feedback, the next repetition will assess the next metric in the list above. During the 5-min break after an attempt is completed without receiving feedback, participants can watch an optional expert-level demonstration video corresponding to the next metric. Participants receive no feedback during the 6th repetition. They will have no feedback in their 7th repetition, the realistic scenario.
Experimental: Experimental Group ICEMS verbal feedback with pause group; 43 participants. Individuals receive standard information. They perform 6 5-min practice scenario resections with a 5-min break between each one. The 7th attempt is the 13-min realistic scenario. Participants receive no feedback during the first repetition. They will then receive feedback on 4 metrics, one metric a time: instrument tip separation, low bipolar force, high aspirator force, high bipolar force. Once an attempt is completed without receiving feedback, the next repetition will assess the next metric in the list above. During the 5-min break after an attempt is completed without receiving feedback, participants can watch an optional expert-level demonstration video corresponding to the next metric. Participants receive no feedback during the 6th repetition. They will have no feedback in their 7th repetition, the realistic scenario.	Behavioral: Experimental Group ICEMS verbal feedback with pause group; While performing the simulated procedure, if participants receive real-time verbal feedback based on the intelligent system error detection, they will be instructed to pause the task, putting their instruments down and reflecting for 22 seconds. A warning will be given when there are 7 seconds remaining in the reflection period so individuals can prepare to resume the task immediately.
Experimental: Experimental Group ICEMS verbal feedback with pause and expert-level video demonstration; 43 participants. Individuals receive standard information. They perform 6 5-min practice scenario resections with a 5-min break between each one. The 7th attempt is the 13-min realistic scenario. Participants receive no feedback during the first repetition. They will then receive feedback on 4 metrics, one metric a time: instrument tip separation, low bipolar force, high aspirator force, high bipolar force. Once an attempt is completed without receiving feedback, the next repetition will assess the next metric in the list above. During the 5-min break after an attempt is completed without receiving feedback, participants can watch an optional expert-level demonstration video corresponding to the next metric. Participants receive no feedback during the 6th repetition. They will have no feedback in their 7th repetition, the realistic scenario.	Behavioral: Experimental Group ICEMS audio feedback with pause and expert video; While performing the simulated procedure, if participants receive real-time verbal feedback based on the intelligent system error detection, they will be instructed to pause the task, putting their instruments down, turning their attention to a 9-second expert-level demonstration video, and reflecting for 13 seconds. A warning will be given when there are 7 seconds remaining in the reflection period so individuals can prepare to resume the task immediately.

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Change in performance	Evaluated by comparing the average composite-score, calculated by the ICEMS, from each practice scenario. Scores range from expert/skilled level (a score of 1.00) to novice/less-skilled level (a score of -1.00).	1 day of study
Transfer of learning	Evaluated by comparing the average composite-score, calculated by the ICEMS, from each practice scenario. Scores range from expert/skilled level (a score of 1.00) to novice/less-skilled level (a score of -1.00).	1 day of study
Objective Structured Assessment of Technical Skills (OSATS) global rating scale	Performance score of the participants in the complex realistic scenario, assessed by two blinded experts using the Objective Structured Assessment of Technical Skills (OSATS) global rating scale on a 7-point Likert scale (1= novice to 7 = expert). Efficacy in learning with pausing methodology and an expert-level demonstration video will be compared to pausing methodology alone and to no pausing methodology.	1 day of study

Secondary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Differences in strength of emotions elicited	Measured by Duffy's Medical Emotional Scale (MES) before, during, and after learning. Participants will self-report the intensity of each emotion on a 5-point Likert scale (1 = not at all to 5 = very strong).	1 day of study
Difference in Cognitive Load	Measured using Leppink's Cognitive Load Index (CLI) after the intervention. Participants will self-report their level of agreement with each statement on a 5-point Likert scale (1 = strongly disagree to 5 = strongly agree).	1 day of study

Collaborators and Investigators

• Sponsor: McGill University
• Investigators: Principal Investigator: Rolando F Del Maestro, MD, PhD, McGill University

Publications and helpful links

General Publications

• Winkler-Schwartz A, Yilmaz R, Mirchi N, Bissonnette V, Ledwos N, Siyar S, Azarnoush H, Karlik B, Del Maestro R. Machine Learning Identification of Surgical and Operative Factors Associated With Surgical Expertise in Virtual Reality Simulation. JAMA Netw Open. 2019 Aug 2;2(8):e198363. doi: 10.1001/jamanetworkopen.2019.8363.
• Yilmaz R, Winkler-Schwartz A, Mirchi N, Reich A, Christie S, Tran DH, Ledwos N, Fazlollahi AM, Santaguida C, Sabbagh AJ, Bajunaid K, Del Maestro R. Continuous monitoring of surgical bimanual expertise using deep neural networks in virtual reality simulation. NPJ Digit Med. 2022 Apr 26;5(1):54. doi: 10.1038/s41746-022-00596-8.
• Fazlollahi AM, Bakhaidar M, Alsayegh A, Yilmaz R, Winkler-Schwartz A, Mirchi N, Langleben I, Ledwos N, Sabbagh AJ, Bajunaid K, Harley JM, Del Maestro RF. Effect of Artificial Intelligence Tutoring vs Expert Instruction on Learning Simulated Surgical Skills Among Medical Students: A Randomized Clinical Trial. JAMA Netw Open. 2022 Feb 1;5(2):e2149008. doi: 10.1001/jamanetworkopen.2021.49008.
• Lee JY, Szulewski A, Young JQ, Donkers J, Jarodzka H, van Merrienboer JJG. The medical pause: Importance, processes and training. Med Educ. 2021 Oct;55(10):1152-1160. doi: 10.1111/medu.14529. Epub 2021 May 1.

Study record dates

Study Major Dates

• Study Start (Actual): January 30, 2024
• Primary Completion (Estimated): September 1, 2024
• Study Completion (Estimated): September 1, 2024

Study Registration Dates

• First Submitted: January 23, 2024
• First Submitted That Met QC Criteria: January 23, 2024
• First Posted (Actual): February 1, 2024

Study Record Updates

• Last Update Posted (Estimated): February 5, 2024
• Last Update Submitted That Met QC Criteria: February 1, 2024
• Last Verified: February 1, 2024

More Information

Terms related to this study

• Keywords: Clinical trial; Artificial intelligence; Virtual reality; Neurosurgery; Medical students; Intelligent tutor; Competency-based feedback
• Other Study ID Numbers: 2010-270, NEU-09-042-Trial 4

Plan for Individual participant data (IPD)

• Plan to Share Individual Participant Data (IPD)?: YES
• IPD Plan Description: Data obtained from primary and secondary outcomes may be shared if other researchers have an interest in this data.
• IPD Sharing Time Frame: Data will be available for 5 years after completion of trial.
• IPD Sharing Access Criteria: Researchers wanting access to the data will need to contact the principal investigator of the trial. Dr. Rolando Del Maestro
• IPD Sharing Supporting Information Type: STUDY_PROTOCOL; SAP; ICF; ANALYTIC_CODE; CSR

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No