Augmented Reality in Orthopedic Surgery Is Emerging from Proof of Concept Towards Clinical Studies: a Literature Review Explaining the Technology and Current State of the Art

Fabio A Casari, Nassir Navab, Laura A Hruby, Philipp Kriechling, Ricardo Nakamura, Romero Tori, Fátima de Lourdes Dos Santos Nunes, Marcelo C Queiroz, Philipp Fürnstahl, Mazda Farshad, Fabio A Casari, Nassir Navab, Laura A Hruby, Philipp Kriechling, Ricardo Nakamura, Romero Tori, Fátima de Lourdes Dos Santos Nunes, Marcelo C Queiroz, Philipp Fürnstahl, Mazda Farshad

Abstract

Purpose of review: Augmented reality (AR) is becoming increasingly popular in modern-day medicine. Computer-driven tools are progressively integrated into clinical and surgical procedures. The purpose of this review was to provide a comprehensive overview of the current technology and its challenges based on recent literature mainly focusing on clinical, cadaver, and innovative sawbone studies in the field of orthopedic surgery. The most relevant literature was selected according to clinical and innovational relevance and is summarized.

Recent findings: Augmented reality applications in orthopedic surgery are increasingly reported. In this review, we summarize basic principles of AR including data preparation, visualization, and registration/tracking and present recently published clinical applications in the area of spine, osteotomies, arthroplasty, trauma, and orthopedic oncology. Higher accuracy in surgical execution, reduction of radiation exposure, and decreased surgery time are major findings presented in the literature. In light of the tremendous progress of technological developments in modern-day medicine and emerging numbers of research groups working on the implementation of AR in routine clinical procedures, we expect the AR technology soon to be implemented as standard devices in orthopedic surgery.

Keywords: AR; Augmented reality; Computer-assisted surgery; HoloLens; Hologram; VR.

Conflict of interest statement

Fabio A. Casari, Nassir Navab, Laura A. Hruby, Philipp Kriechling, Ricardo Nakamura, Romero Tori, Fátima de Lourdes dos Santos Nunes, Marcelo C. Queiroz, Philipp Fürnstahl and Mazda Farshad declare that they have no conflict of interest.

References

    1. Schweizer A, Furnstahl P, Harders M, Szekely G, Nagy L. Complex radius shaft malunion: osteotomy with computer-assisted planning. Hand (N Y) 2010;5(2):171–178.
    1. Schlenzka D, Laine T, Lund T. Computer-assisted spine surgery. Eur Spine J. 2000;9(1):S057–SS64.
    1. Hernandez D, Garimella R, Eltorai AEM, Daniels AH. Computer-assisted orthopaedic surgery. Orthop Surg. 2017;9(2):152–158.
    1. End K, Eppenga R, Kfd K, Groen HC, van Veen R, van Dieren JM, et al. Accurate surgical navigation with real-time tumor tracking in cancer surgery. NPJ Precis Oncol. 2020;4(1):8.
    1. Navab N, Blum T, Wang L, Okur A, Wendler T. First deployments of augmented reality in operating rooms. Computer. 2012;45(7):48–55.
    1. Watkins RG, Gupta A, Watkins RG. Cost-effectiveness of image-guided spine surgery. Open Orthop J. 2010;4:228–233.
    1. Barbash GI, Glied SA. New technology and health care costs — the case of robot-assisted surgery. N Engl J Med. 2010;363(8):701–704.
    1. Qiu B, Liu F, Tang B, Deng B, Liu F, Zhu W, Zhen D, Xue M, Zhang M. Clinical study of 3D imaging and 3D printing technique for patient-specific instrumentation in total knee arthroplasty. J Knee Surg. 2017;30(8):822–828.
    1. Roner S, Bersier P, Furnstahl P, Vlachopoulos L, Schweizer A, Wieser K. 3D planning and surgical navigation of clavicle osteosynthesis using adaptable patient-specific instruments. J Orthop Surg Res. 2019;14(1):115.
    1. Furnstahl P, Vlachopoulos L, Schweizer A, Fucentese SF, Koch PP. Complex osteotomies of tibial plateau malunions using computer-assisted planning and patient-specific surgical guides. J Orthop Trauma. 2015;29(8):e270–e276.
    1. Azuma R, Baillot Y, Behringer R, Feiner S, Julier S, MacIntyre B. Recent advances in augmented reality. IEEE Comput Graph Appl. 2001;21(6):34–47.
    1. Azuma R. Tracking requirements for augmented reality. Commun ACM. 1993;36(7):50–51.
    1. Tan CT, Soh D. Augmented reality games: a review 2011. 212–218 p.
    1. Blackwell M, Morgan F, DiGioia AM. Augmented reality and its future in orthopaedics. Clin Orthop Relat Res. 1998;354:111–122.
    1. • Molina CA, Theodore N, Ahmed AK, Westbroek EM, Mirovsky Y, Harel R, et al. Augmented reality-assisted pedicle screw insertion: a cadaveric proof-of-concept study. J Neurosurg Spine. 2019:1–8 Shows that that AR-navigated pedicles screws are not inferior to conventional navigation systems and robotic-assited screw placement. Results were superior to freehand pedicle screw placement.
    1. Gibby JT, Swenson SA, Cvetko S, Rao R, Javan R. Head-mounted display augmented reality to guide pedicle screw placement utilizing computed tomography. Int J Comput Assist Radiol Surg. 2019;14(3):525–535.
    1. Cherian JJ, Kapadia BH, Banerjee S, Jauregui JJ, Issa K, Mont MA. Mechanical, anatomical, and kinematic axis in TKA: concepts and practical applications. Curr Rev Musculoskelet Med. 2014;7(2):89–95.
    1. Bernard TN, Jr, Seibert CE. Pedicle diameter determined by computed tomography. Its relevance to pedicle screw fixation in the lumbar spine. Spine. 1992;17(6 Suppl):S160–S163.
    1. Lambert DH, Deane RS, Mazuzan JE., Jr Anesthesia and the control of blood pressure in patients with spinal cord injury. Anesth Analg. 1982;61(4):344–348.
    1. Nelson CL, Fontenot HJ. Ten strategies to reduce blood loss in orthopedic surgery. Am J Surg. 1995;170(6A Suppl):64s–68s.
    1. Liebmann F, Roner S, von Atzigen M, Scaramuzza D, Sutter R, Snedeker J, et al. Pedicle screw navigation using surface digitization on the Microsoft HoloLens. Int J Comput Assist Radiol Surg. 2019;14(7):1157–1165.
    1. Elmi-Terander A, Burström G, Nachabe R, Skulason H, Pedersen K, Fagerlund M, et al. Pedicle screw placement using augmented reality surgical navigation with intraoperative 3D imaging: a first in-human prospective cohort study. Spine. 2019;44(7).
    1. Wanivenhaus F, Neuhaus C, Liebmann F, Roner S, Spirig JM, Farshad M. Augmented reality-assisted rod bending in spinal surgery. Spine J. 2019;19(10):1687–1689.
    1. Elmi-Terander A, Burström G, Nachabé R, Fagerlund M, Ståhl F, Charalampidis A, et al. Augmented reality navigation with intraoperative 3D imaging vs fluoroscopy-assisted free-hand surgery for spine fixation surgery: a matched-control study comparing accuracy. Sci Rep. 2020;10(1):707.
    1. Müller F, Roner S, Liebmann F, Spirig JM, Fürnstahl P, Farshad M. Augmented reality navigation for spinal pedicle screw instrumentation using intraoperative 3D imaging. Spine J. 2020;20(4):621–628.
    1. Yoon JW, Chen RE, Han PK, Si P, Freeman WD, Pirris SM. Technical feasibility and safety of an intraoperative head-up display device during spine instrumentation. Int J Med Robot. 2017;13(3).
    1. Elmi-Terander A, Skulason H, Soderman M, Racadio J, Homan R, Babic D, et al. Surgical navigation technology based on augmented reality and integrated 3D intraoperative imaging: a spine cadaveric feasibility and accuracy study. Spine. 2016;41(21):E1303–E1e11.
    1. Wu JR, Wang ML, Liu KC, Hu MH, Lee PY. Real-time advanced spinal surgery via visible patient model and augmented reality system. Comput Methods Prog Biomed. 2014;113(3):869–881.
    1. Kosterhon M, Gutenberg A, Kantelhardt SR, Archavlis E, Giese A. Navigation and image injection for control of bone removal and osteotomy planes in spine surgery. Oper Neurosurg. 2017;13(2):297–304.
    1. Fallavollita P, Brand A, Wang L, Euler E, Thaller P, Navab N, Weidert S. An augmented reality C-arm for intraoperative assessment of the mechanical axis: a preclinical study. Int J Comput Assist Radiol Surg. 2016;11(11):2111–2117.
    1. Ogawa H, Kurosaka K, Sato A, Hirasawa N, Matsubara M, Tsukada S. Does an augmented reality-based portable navigation system improve the accuracy of acetabular component orientation during THA? A randomized controlled trial. Clin Orthop Relat Res. 2020;478(5):935–943.
    1. Alexander C, Loeb AE, Fotouhi J, Navab N, Armand M, Khanuja HS. Augmented reality for acetabular component placement in direct anterior total hip arthroplasty. J Arthroplast. 2020.
    1. Tsukada S, Ogawa H, Nishino M, Kurosaka K, Hirasawa N. Augmented reality-based navigation system applied to tibial bone resection in total knee arthroplasty. J Exp Orthop. 2019;6(1):44.
    1. Ogawa H, Hasegawa S, Tsukada S, Matsubara M. A pilot study of augmented reality technology applied to the acetabular cup placement during total hip arthroplasty. J Arthroplast. 2018;33(6):1833–1837.
    1. Fotouhi J, Alexander CP, Unberath M, Taylor G, Lee SC, Fuerst B, et al. Plan in 2-D, execute in 3-D: an augmented reality solution for cup placement in total hip arthroplasty. J Med Imaging (Bellingham) 2018;5(2):021205.
    1. Liu H, Auvinet E, Giles J, Rodriguez y Baena F. Augmented reality based navigation for computer assisted hip resurfacing: a proof of concept study. Ann Biomed Eng 2018;46(10):1595–1605.
    1. Cho HS, Park MS, Gupta S, Han I, Kim HS, Choi H, Hong J. Can augmented reality be helpful in pelvic bone cancer surgery? An in vitro study. Clin Orthop Relat Res. 2018;476(9):1719–1725.
    1. Cho HS, Park YK, Gupta S, Yoon C, Han I, Kim HS, Choi H, Hong J. Augmented reality in bone tumour resection: an experimental study. Bone Joint Res. 2017;6(3):137–143.
    1. Choi H, Park Y, Lee S, Ha H, Kim S, Cho HS, Hong J. A portable surgical navigation device to display resection planes for bone tumor surgery. Minim Invasive Ther Allied Technol. 2017;26(3):144–150.
    1. Fritz J, P Ut, Ungi T, Flammang AJ, McCarthy EF, Fichtinger G, et al. Augmented reality visualization using image overlay technology for MR-guided interventions: cadaveric bone biopsy at 1.5 T. Investig Radiol 2013;48(6):464–470.
    1. Weidert S, Wang L, Landes J, Sandner P, Suero EM, Navab N, Kammerlander C, Euler E, von der Heide A. Video-augmented fluoroscopy for distal interlocking of intramedullary nails decreased radiation exposure and surgical time in a bovine cadaveric setting. Int J Med Robot. 2019;15(4):e1995.
    1. von der Heide AM, Fallavollita P, Wang L, Sandner P, Navab N, Weidert S, et al. Camera-augmented mobile C-arm (CamC): a feasibility study of augmented reality imaging in the operating room. Int J Med Robot. 2018;14(2).
    1. Shen F, Chen B, Guo Q, Qi Y, Shen Y. Augmented reality patient-specific reconstruction plate design for pelvic and acetabular fracture surgery. Int J Comput Assist Radiol Surg. 2013;8(2):169–179.
    1. Ortega G, Wolff A, Baumgaertner M, Kendoff D. Usefulness of a head mounted monitor device for viewing intraoperative fluoroscopy during orthopaedic procedures. Arch Orthop Trauma Surg. 2008;128(10):1123–1126.
    1. •• Fürnstahl P, Schweizer A, Graf M, Vlachopoulos L, Fucentese S, Wirth S, et al. Surgical treatment of long-bone deformities: 3D preoperative planning and patient-specific instrumentation. In: Zheng G, Li S, editors. Computational radiology for orthopaedic interventions. Cham: Springer International Publishing; 2016. p. 123–49. Describes the process of data preparation and surgical planning for the purpose of 3D surgical navigation of corrective procedures for long bone deformities.
    1. Lorensen W, Cline EH. Marching cubes: a high resolution 3D surface construction algorithm. 1987. 163 p.
    1. Lindgren Belal S, Sadik M, Kaboteh R, Enqvist O, Ulén J, Poulsen MH, Simonsen J, Høilund-Carlsen PF, Edenbrandt L, Trägårdh E. Deep learning for segmentation of 49 selected bones in CT scans: first step in automated PET/CT-based 3D quantification of skeletal metastases. Eur J Radiol. 2019;113:89–95.
    1. Zhou A, Zhao Q, Zhu J, editors. Automatic segmentation algorithm of femur and tibia based on Vnet-C network. 2019 Chinese Automation Congress (CAC); 2019 22–24 Nov. 2019.
    1. Jodeiri A, Zoroofi RA, Hiasa Y, Takao M, Sugano N, Sato Y, et al. Fully automatic estimation of pelvic sagittal inclination from anterior-posterior radiography image using deep learning framework. Comput Methods Prog Biomed. 2020;184:105282.
    1. Bae H-J, Hyun H, Byeon Y, Shin K, Cho Y, Song YJ, et al. Fully automated 3D segmentation and separation of multiple cervical vertebrae in CT images using a 2D convolutional neural network. Comput Methods Prog Biomed. 2020;184:105119.
    1. Kamiya N. Muscle segmentation for orthopedic interventions. In: Zhuang X, Zheng G, Tian W, editors. Intelligent orthopaedics. Singapore: Springer; 2018. p. 1093.
    1. • Kamiya N. Deep learning technique for musculoskeletal analysis. In: G L, H F, editors. Deep learning in medical image analysis. 1213: Springer, Cham; 2020. First approaches of automatic segmentation of bone imaging will allow automatization of currently highly manual process steps in the future.
    1. Dou Q, Yu L, Chen H, Jin Y, Yang X, Qin J, et al. 3D deeply supervised network for automated segmentation of volumetric medical images. Med Image Anal. 2017;41:40–54.
    1. •• Navab N, Bani-Kashemi A, Mitschke M, editors. Merging visible and invisible: two camera-augmented Mobile C-arm (CAMC) applications. Proceedings 2nd IEEE and ACM international workshop on augmented reality (IWAR’99); 1999 20-21 Oct. 1999. It describes the camera augmented surgical c-arm for augmented reality in the operating room.
    1. Navab N, Heining S, Traub J. Camera augmented mobile C-arm (CAMC): calibration, accuracy study, and clinical applications. IEEE Trans Med Imaging. 2010;29(7):1412–1423.
    1. Fotouhi J, Unberath M, Song T, Gu W, Johnson A, Osgood G, Armand M, Navab N. Interactive flying frustums (IFFs): spatially aware surgical data visualization. Int J Comput Assist Radiol Surg. 2019;14(6):913–922.
    1. Wu H-K, Lee SW-Y, Chang H-Y, Liang J-C. Current status, opportunities and challenges of augmented reality in education. Comput Educ. 2013;62:41–49.
    1. Chen Y, Medioni G, editors. Object modeling by registration of multiple range images. Proceedings 1991 IEEE International Conference on Robotics and Automation; 1991 9–11 April 1991.
    1. Milgram P, Takemura H, Utsumi A, Kishino F. Augmented reality: a class of displays on the reality-virtuality continuum: SPIE; 1995.
    1. Schmalstieg D, Höllerer T. Augmented reality - principles and practice: Addison-Wesley professional; 2016.
    1. Chandra ANR, Jamiy FE, Reza H, editors. Augmented reality for big data visualization: a review. 2019 International Conference on Computational Science and Computational Intelligence (CSCI); 2019 5–7 Dec. 2019.
    1. Jih-Fang W, Ronald TA, Gary B, Vernon C, John E, Henry F, editors. Tracking a head-mounted display in a room-sized environment with head-mounted cameras. ProcSPIE; 1990.
    1. Hoff W, Vincent T. Analysis of head pose accuracy in augmented reality. IEEE Trans Vis Comput Graph. 2000;6(4):319–334.
    1. • Florentin Liebmann SR, Marco von Atzigen, Florian Wanivenhaus, Caroline Neuhaus, José Spirig, Davide Scaramuzza, Reto Sutter, Jess Snedeker, Mazda Farshad, Philipp Fürnstahl. Registration made easy -- standalone orthopedic navigation with HoloLens. CVPR 2019 workshop on computer vision applications for mixed reality headsets. 2019. The work shows how a conventional HMD developed for entertainment purposes (Microsoft HoloLens 1) used for surgical navigation purposes might meet clinical accurcy requirements.
    1. Gertzbein SD, Robbins SE. Accuracy of pedicular screw placement in vivo. Spine. 1990;15(1).
    1. Heary RF, Bono CM, Black M. Thoracic pedicle screws: postoperative computerized tomography scanning assessment. J Neurosurg. 2004;100(4 Suppl Spine):325–331.
    1. Schrepp M, Hinderks A, Thomaschewski J. Design and evaluation of a short version of the user experience questionnaire (UEQ-S) Int J Interactive Multimedia Artif Intell. 2017;4:103–108.
    1. Liebmann F, Roner S, von Atzigen M, Scaramuzza D, Sutter R, Snedeker J, et al. Pedicle screw navigation using surface digitization on the Microsoft HoloLens. Int J Comput Assist Radiol Surg. 2019.
    1. Arnd Viehöfer SHW, Stefan Michael Zimmermann, Laurenz Jaberg, Cyrill Dennler, Philipp Fürnstahl, Mazda Farshad. Augmented reality guided osteotomy in Hallux Valgus Correction (Preprint). BMC musculoskeletal disorders. 2020;Preprint.
    1. Roner S, Vlachopoulos L, Nagy L, Schweizer A, Fürnstahl P. Accuracy and early clinical outcome of 3-dimensional planned and guided single-cut osteotomies of malunited forearm bones. J Hand Surg. 2017;42(12):1031. e1–1031. e8.
    1. Cometti C, Païzis C, Casteleira A, Pons G, Babault N. Effects of mixed reality head-mounted glasses during 90 minutes of mental and manual tasks on cognitive and physiological functions. PeerJ. 2018;6:e5847–e.
    1. Navab N, Traub J, Sielhorst T, Feuerstein M, Bichlmeier C. Action- and workflow-driven augmented reality for computer-aided medical procedures. IEEE Comput Graph Appl. 2007;27(5):10–14.

Source: PubMed

Patrocinadores e Colaboradores

Condições médicas

Intervenções de drogas

3
Se inscrever