Validation of Motion Tracking Software for Evaluation of Surgical Performance in Laparoscopic Cholecystectomy

Sandeep Ganni, Sanne M B I Botden, Magdalena Chmarra, Meng Li, Richard H M Goossens, Jack J Jakimowicz, Sandeep Ganni, Sanne M B I Botden, Magdalena Chmarra, Meng Li, Richard H M Goossens, Jack J Jakimowicz

Abstract

Motion tracking software for assessing laparoscopic surgical proficiency has been proven to be effective in differentiating between expert and novice performances. However, with several indices that can be generated from the software, there is no set threshold that can be used to benchmark performances. The aim of this study was to identify the best possible algorithm that can be used to benchmark expert, intermediate and novice performances for objective evaluation of psychomotor skills. 12 video recordings of various surgeons were collected in a blinded fashion. Data from our previous study of 6 experts and 23 novices was also included in the analysis to determine thresholds for performance. Video recording were analyzed both by the Kinovea 0.8.15 software and a blinded expert observer using the CAT form. Multiple algorithms were tested to accurately identify expert and novice performances. ½ L + [Formula: see text] A + [Formula: see text] J scoring of path length, average movement and jerk index respectively resulted in identifying 23/24 performances. Comparing the algorithm to CAT assessment yielded in a linear regression coefficient R2 of 0.844. The value of motion tracking software in providing objective clinical evaluation and retrospective analysis is evident. Given the prospective use of this tool the algorithm developed in this study proves to be effective in benchmarking performances for psychomotor skills evaluation.

Keywords: Indices of performance; Laparoscopic skills training; Motion tracking; Objective evaluation; Thresholds of performance; Video-based assessment.

Conflict of interest statement

Sandeep Ganni, Sanne MBI Botden, Magdalena K. Chmarra, Meng Li, Richard HM Goossens and Jack J. Jakimowicz have no conflicts of interest or financial ties to disclose.

Figures

Fig. 1
Fig. 1
Plot of Weighted score of videos, p vs expert-assessed CAT score. The linear trendline has a regression coefficient of determination (R2) of 0.844

References

    1. Moorthy K, Munz Y, Sarker SK, Darzi A. Objective assessment of technical skills in surgery. Br J Surg. 2003;327(7422):1032–1037.
    1. van Hove P. D., Tuijthof G. J. M., Verdaasdonk E. G. G., Stassen L. P. S., Dankelman J. Objective assessment of technical surgical skills. British Journal of Surgery. 2010;97(7):972–987. doi: 10.1002/bjs.7115.
    1. Oropesa Ignacio, Sánchez-González Patricia, Lamata Pablo, Chmarra Magdalena K., Pagador José B., Sánchez-Margallo Juan A., Sánchez-Margallo Francisco M., Gómez Enrique J. Methods and Tools for Objective Assessment of Psychomotor Skills in Laparoscopic Surgery. Journal of Surgical Research. 2011;171(1):e81–e95. doi: 10.1016/j.jss.2011.06.034.
    1. Botden SMBI, Jakimowicz JJ. What is going on in augmented reality simulation in laparoscopic surgery? Surg Endosc. 2008;23:1693–1700. doi: 10.1007/s00464-008-0144-1.
    1. Bann S, Darzi A, Munz Y, Kumar BD, Moorthy K. Laparoscopic virtual reality and box trainers: Is one superior to the other? Surg Endosc. 2004;18:485–494. doi: 10.1007/s00464-003-9043-7.
    1. Schijven MP, Jakimowicz JJ, Broeders IAMJ, Tseng LNL. The Eindhoven laparoscopic cholecystectomy training course—Improving operating room performance using virtual reality training: Results from the first E.a.E.S. accredited virtual reality trainings curriculum. Surg Endosc. 2005;19(9):1220–1226. doi: 10.1007/s00464-004-2240-1.
    1. Seymour NE, Gallagher AG, Roman SA, O’Brien MK, Bansal VK, Andersen DK, Satava RM. Virtual reality training improves operating room performance. Ann Surg. 2002;236:458–464. doi: 10.1097/00000658-200210000-00008.
    1. Ganni S, Chmarra MK, Goossens RHM, Jakimowicz JJ. Self-assessment in laparoscopic surgical skills training: Is it reliable? Surg Endosc. 2017;31(6):2451–2456. doi: 10.1007/s00464-016-5246-6.
    1. Gardner Aimee K, Abdelfattah Kareem. Getting better all the time? Facilitating accurate team self-assessments through simulation. BMJ Simulation and Technology Enhanced Learning. 2019;6(1):31–34. doi: 10.1136/bmjstel-2018-000411.
    1. Ganni Sandeep, Botden Sanne M.B.I., Schaap Dennis P., Verhoeven Bas H., Goossens Richard H.M., Jakimowicz Jack J. “Reflection- Before -Practice” Improves Self-Assessment and End-Performance in Laparoscopic Surgical Skills Training. Journal of Surgical Education. 2018;75(2):527–533. doi: 10.1016/j.jsurg.2017.07.030.
    1. Grantcharov T. P., Rosenberg J., Pahle E., Fench-Jensen P. Virtual reality computer simulation. Surgical Endoscopy. 2001;15(3):242–244. doi: 10.1007/s004640090008.
    1. Lamata P, Gomez EJ, Bello F, et al. Conceptual framework for laparoscopic VR simulators. IEEE Comput Graph Appl. 2006;26(6):69–79. doi: 10.1109/MCG.2006.125.
    1. Yamaguchi S, Yoshida D, Kenmotsu H, Yasunaga T, Konishi K, Ieiri S, Nakashima H, Tanoue K, Hashizume M. Objective assessment of laparoscopic suturing skills using a motion-tracking system. Surg Endosc. 2010;25:771–775. doi: 10.1007/s00464-010-1251-3.
    1. Oropesa I, Chmarra MK, Sánchez-González P, Lamata P, Rodrigues SP, Enciso S, Sánchez-Margallo FM, Jansen F-W, Dankelman J, Gómez EJ. Relevance of motion-related assessment metrics in laparoscopic surgery. Surg Innov. 2013;20:299–312. doi: 10.1177/1553350612459808.
    1. Hofstad EF, Våpenstad C, Chmarra MK, Langø T, Kuhry E, Mårvik R. A study of psychomotor skills in minimally invasive surgery: What differentiates expert and nonexpert performance. Surg Endosc. 2012;27(3):854–863. doi: 10.1007/s00464-012-2524-9.
    1. Ghasemloonia A, Maddahi Y, Zareinia K, Lama S, Dort JC, Sutherland GR. Surgical skill assessment using motion quality and smoothness. Journal of Surgical Education. 2017;74(2):295–305. doi: 10.1016/j.jsurg.2016.10.006.
    1. Ganni Sandeep, Botden Sanne M. B. I., Chmarra Magdalena, Goossens Richard H. M., Jakimowicz Jack J. A software-based tool for video motion tracking in the surgical skills assessment landscape. Surgical Endoscopy. 2018;32(6):2994–2999. doi: 10.1007/s00464-018-6023-5.
    1. Kowalewski Karl-Friedrich, Garrow Carly R., Schmidt Mona W., Benner Laura, Müller-Stich Beat P., Nickel Felix. Sensor-based machine learning for workflow detection and as key to detect expert level in laparoscopic suturing and knot-tying. Surgical Endoscopy. 2019;33(11):3732–3740. doi: 10.1007/s00464-019-06667-4.
    1. Baghdadi, A., Hussein, A. A., Ahmed, Y., Cavuoto, L. A., and Guru, K. A., A computer vision technique for automated assessment of surgical performance using surgeon console-feed videos. Int J Comput Assist Radiol Surg, 2018. 10.1007/s11548-1881-9.
    1. Miskovic D, Ni M, Wyles SM, Kennedy RH, Francis NK, Parvaiz A, Cunningham C, Rockall TA, Gudgeon AM, Coleman MG, Hanna GB. Is competency assessment at the specialist level achievable? A study for the national training program in laparoscopic colorectal surgery in England. Ann Surg. 2013;257:476–482. doi: 10.1097/SLA.0b013e318275b72a.
    1. Vassiliou Melina C., Feldman Liane S., Andrew Christopher G., Bergman Simon, Leffondré Karen, Stanbridge Donna, Fried Gerald M. A global assessment tool for evaluation of intraoperative laparoscopic skills. The American Journal of Surgery. 2005;190(1):107–113. doi: 10.1016/j.amjsurg.2005.04.004.
    1. Martin JA, Regehr G, Reznick R, et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg. 1997;84:243–278. doi: 10.1002/bjs.1800840237.
    1. Reiley CE, Lin HC, Yuh DD, et al. Review of methods for objective surgical skill evaluation. Surg Endosc. 2011;25:356. doi: 10.1007/s00464-010-1190-z.
    1. Chmarra MK, Grimbergen CA, Dankelman J. Systems for tracking minimally invasive surgical instruments. Min Inv Ther All Tech. 2007;16(6):328–340. doi: 10.1080/13645700701702135.
    1. Hashimoto Daniel A., Rosman Guy, Volkov Mikhail, Rus Daniela L., Meireles Ozanan R. Artificial Intelligence for Intraoperative Video Analysis: Machine Learning’s Role in Surgical Education. Journal of the American College of Surgeons. 2017;225(4):S171. doi: 10.1016/j.jamcollsurg.2017.07.387.

Source: PubMed

Patrocinadores e Colaboradores

Condições médicas

Intervenções de drogas

3
Se inscrever