Development of a real-time endoscopic image diagnosis support system using deep learning technology in colonoscopy

Masayoshi Yamada, Yutaka Saito, Hitoshi Imaoka, Masahiro Saiko, Shigemi Yamada, Hiroko Kondo, Hiroyuki Takamaru, Taku Sakamoto, Jun Sese, Aya Kuchiba, Taro Shibata, Ryuji Hamamoto, Masayoshi Yamada, Yutaka Saito, Hitoshi Imaoka, Masahiro Saiko, Shigemi Yamada, Hiroko Kondo, Hiroyuki Takamaru, Taku Sakamoto, Jun Sese, Aya Kuchiba, Taro Shibata, Ryuji Hamamoto

Abstract

Gaps in colonoscopy skills among endoscopists, primarily due to experience, have been identified, and solutions are critically needed. Hence, the development of a real-time robust detection system for colorectal neoplasms is considered to significantly reduce the risk of missed lesions during colonoscopy. Here, we develop an artificial intelligence (AI) system that automatically detects early signs of colorectal cancer during colonoscopy; the AI system shows the sensitivity and specificity are 97.3% (95% confidence interval [CI] = 95.9%-98.4%) and 99.0% (95% CI = 98.6%-99.2%), respectively, and the area under the curve is 0.975 (95% CI = 0.964-0.986) in the validation set. Moreover, the sensitivities are 98.0% (95% CI = 96.6%-98.8%) in the polypoid subgroup and 93.7% (95% CI = 87.6%-96.9%) in the non-polypoid subgroup; To accelerate the detection, tensor metrics in the trained model was decomposed, and the system can predict cancerous regions 21.9 ms/image on average. These findings suggest that the system is sufficient to support endoscopists in the high detection against non-polypoid lesions, which are frequently missed by optical colonoscopy. This AI system can alert endoscopists in real-time to avoid missing abnormalities such as non-polypoid polyps during colonoscopy, improving the early detection of this disease.

Conflict of interest statement

The authors declare no competing interests.

Figures

Figure 1
Figure 1
Representative images of trained colonic lesions. (a) 10-mm sized pedunculated type. (b) 10-mm sized sessile type. (c) 4-mm sized superficial elevated type. (d) 4-mm sized superficial depressed type. (e) 4-mm sized superficial depressed type. (f) 25-mm sized non-granular type laterally spreading tumor. (g) 18-mm sized granular type laterally spreading tumor. (h) 50-mm sized granular type laterally spreading tumor. i, 6-mm sized sessile serrated lesion.
Figure 2
Figure 2
Represented schematically outline of the developed artificial intelligence system.
Figure 3
Figure 3
Representative images of detected polyps. (a) A 10-mm adenomatous polyp (polypoid type). (b) A 2-mm adenomatous polyp (polypoid type). (c) A 4-mm adenomatous polyp (slightly elevated type). (d) A 5-mm serrated lesion (slightly elevated type).
Figure 4
Figure 4
Comparing diagnostic performance between the AI system and endoscopists, and Intersection over the union (IoU) for the lesion detection. (a) Diagnostic performance was represented by the receiver-operating characteristic curve with AUC = 0.9752. Each orange, gray, and yellow point represents the sensitivity and specificity of an endoscopist. (b) If we defined poor = IoU < 0.5, good ≥0.5, <0.7, excellent ≥0.7, Good and Excellent was 91%, indicating AI flag is almost correct for lesions detection.

References

    1. Hori M, et al. Cancer incidence and incidence rates in Japan in 2009: a study of 32 population-based cancer registries for the Monitoring of Cancer Incidence in Japan (MCIJ) project. Jpn. J. Clin. Oncol. 2015;45:884–891. doi: 10.1093/jjco/hyv088.
    1. Arnold M, et al. Global patterns and trends in colorectal cancer incidence and mortality. Gut. 2017;66:683–691. doi: 10.1136/gutjnl-2015-310912.
    1. Winawer SJ, et al. Prevention of colorectal cancer by colonoscopic polypectomy. The National Polyp Study Workgroup. The New England journal of medicine. 1993;329:1977–1981. doi: 10.1056/NEJM199312303292701.
    1. Zauber AG, et al. Colonoscopic polypectomy and long-term prevention of colorectal-cancer deaths. The New England journal of medicine. 2012;366:687–696. doi: 10.1056/NEJMoa1100370.
    1. Samadder NJ, et al. Characteristics of missed or interval colorectal cancer and patient survival: a population-based study. Gastroenterology. 2014;146:950–960. doi: 10.1053/j.gastro.2014.01.013.
    1. Corley DA, et al. Adenoma detection rate and risk of colorectal cancer and death. N. Engl. J. Med. 2014;370:1298–1306. doi: 10.1056/NEJMoa1309086.
    1. Kaminski MF, et al. Quality indicators for colonoscopy and the risk of interval cancer. N. Engl. J. Med. 2010;362:1795–1803. doi: 10.1056/NEJMoa0907667.
    1. le Clercq CM, et al. Postcolonoscopy colorectal cancers are preventable: a population-based study. Gut. 2014;63:957–963. doi: 10.1136/gutjnl-2013-304880.
    1. le Clercq CM, et al. Metachronous colorectal cancers result from missed lesions and non-compliance with surveillance. Gastrointest. Endosc. 2015;82:325–333 e322. doi: 10.1016/j.gie.2014.12.052.
    1. Stoffel EM, et al. Clinical and Molecular Characteristics of Post-Colonoscopy Colorectal Cancer: A Population-based Study. Gastroenterology. 2016;151:870–878 e873. doi: 10.1053/j.gastro.2016.07.010.
    1. Rex DK, et al. Colonoscopic miss rates of adenomas determined by back-to-back colonoscopies. Gastroenterology. 1997;112:24–28. doi: 10.1016/S0016-5085(97)70214-2.
    1. van Rijn JC, et al. Polyp miss rate determined by tandem colonoscopy: a systematic review. Am. J. Gastroenterol. 2006;101:343–350. doi: 10.1111/j.1572-0241.2006.00390.x.
    1. McCarthy JF, et al. Applications of machine learning and high-dimensional visualization in cancer detection, diagnosis, and management. Ann. N. Y. Acad. Sci. 2004;1020:239–262. doi: 10.1196/annals.1310.020.
    1. Hinton GE, Osindero S, Teh YW. A fast learning algorithm for deep belief nets. Neural Comput. 2006;18:1527–1554. doi: 10.1162/neco.2006.18.7.1527.
    1. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015;521:436–444. doi: 10.1038/nature14539.
    1. He, K., Zhang, X., Ren, S. & Sun, J. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. IEEE International Conference on Computer Vision, 1026–1034 (2015).
    1. Ehteshami Bejnordi B, et al. Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast Cancer. JAMA. 2017;318:2199–2210. doi: 10.1001/jama.2017.14585.
    1. Esteva A, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017;542:115–118. doi: 10.1038/nature21056.
    1. Gulshan V, et al. Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs. JAMA. 2016;316:2402–2410. doi: 10.1001/jama.2016.17216.
    1. Doi K. Computer-aided diagnosis in medical imaging: historical review, current status and future potential. Comput. Med. Imaging Graph. 2007;31:198–211. doi: 10.1016/j.compmedimag.2007.02.002.
    1. Bernal J, et al. Comparative Validation of Polyp Detection Methods in Video Colonoscopy: Results From the MICCAI 2015 Endoscopic Vision Challenge. IEEE Trans. Med. Imaging. 2017;36:1231–1249. doi: 10.1109/TMI.2017.2664042.
    1. Farris AB, et al. Sessile serrated adenoma: challenging discrimination from other serrated colonic polyps. Am J Surg Pathol. 2008;32:30–35. doi: 10.1097/PAS.0b013e318093e40a.
    1. Benson AB, III, et al. Colon. Cancer, Version 1.2017, NCCN Clinical Practice Guidelines in Oncology. J. Natl. Compr. Canc. Netw. 2017;15:370–398.
    1. Voulodimos A, Doulamis N, Doulamis A, Protopapadakis E. Deep Learning for Computer Vision: A Brief Review. Comput Intell Neurosci. 2018;2018:7068349. doi: 10.1155/2018/7068349.
    1. Chartrand G, et al. Deep Learning: A Primer for Radiologists. Radiographics. 2017;37:2113–2131. doi: 10.1148/rg.2017170077.
    1. Erhan, D., Szegedy, C., Toshev, A. & Anguelov, D. Scalable Object Detection using Deep Neural Networks. Computer Vision and Pattern Recognition (2014).
    1. Ren S, He K, Girshick R, Sun J, Faster R- CNN: Towards Real-Time Object Detection with Region Proposal Networks. IEEE Trans Pattern Anal Mach Intell. 2017;39:1137–1149. doi: 10.1109/TPAMI.2016.2577031.
    1. Simonyan, K. & Zisserman, A. Very Deep Convolutional Networks for Large-Scale Image Recognition. Computer Vision and Pattern Recognition (2015).
    1. Kim, Y.-D. et al. Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications. Computer Vision and Pattern Recognition, In ICLR (2016).
    1. Winawer SJ, et al. Randomized comparison of surveillance intervals after colonoscopic removal of newly diagnosed adenomatous polyps. The National Polyp Study Workgroup. N. Engl. J. Med. 1993;328:901–906. doi: 10.1056/NEJM199304013281301.
    1. Tajbakhsh, N., Gurudu, S. R. & Liang, J. Automatic polyp detection in colonoscopy videos using an ensemble of convolutional neural networks. Biomedical Imaging Proc. 2015 IEEE 12th International Symposium, 79–83 (2015).
    1. Tajbakhsh N, Gurudu SR, Liang J. Automated Polyp Detection in Colonoscopy Videos Using Shape and Context Information. IEEE Trans. Med. Imaging. 2016;35:630–644. doi: 10.1109/TMI.2015.2487997.
    1. Tajbakhsh N, et al. Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning? IEEE Trans. Med. Imaging. 2016;35:1299–1312. doi: 10.1109/TMI.2016.2535302.
    1. Wang Y, Tavanapong W, Wong J, Oh JH, de Groen PC. Polyp-Alert: near real-time feedback during colonoscopy. Comput. Methods Programs Biomed. 2015;120:164–179. doi: 10.1016/j.cmpb.2015.04.002.
    1. Riegler, M. et al. EIR - Efficient computer aided diagnosis framework for gastrointestinal endoscopies. Proc. 2016 14th International Workshop on Content-Based Multimedia Indexing (2016).
    1. Bae SH, Yoon KJ. Polyp Detection via Imbalanced Learning and Discriminative Feature Learning. IEEE Trans. Med. Imaging. 2015;34:2379–2393. doi: 10.1109/TMI.2015.2434398.
    1. Mahmud N, Cohen J, Tsourides K, Berzin TM. Computer vision and augmented reality in gastrointestinal endoscopy. Gastroenterol Rep (Oxf) 2015;3:179–184. doi: 10.1093/gastro/gov027.
    1. Fernandez-Esparrach G, et al. Exploring the clinical potential of an automatic colonic polyp detection method based on the creation of energy maps. Endoscopy. 2016;48:837–842. doi: 10.1055/s-0042-108434.
    1. Pu W, et al. Development and validation of a deep-learning algorithm for the detection of polyps during colonoscopy. Nature Biomedical Engineeringvolume. 2018;2:741–748. doi: 10.1038/s41551-018-0301-3.
    1. Urban G, et al. Deep Learning Localizes and Identifies Polyps in Real Time With 96% Accuracy in Screening Colonoscopy. Gastroenterology. 2018;155:1069–1078 e1068. doi: 10.1053/j.gastro.2018.06.037.
    1. Leufkens AM, et al. Effect of a retrograde-viewing device on adenoma detection rate during colonoscopy: the TERRACE study. Gastrointest. Endosc. 2011;73:480–489. doi: 10.1016/j.gie.2010.09.004.
    1. DeMarco DC, et al. Impact of experience with a retrograde-viewing device on adenoma detection rates and withdrawal times during colonoscopy: the Third Eye Retroscope study group. Gastrointest. Endosc. 2010;71:542–550. doi: 10.1016/j.gie.2009.12.021.
    1. Waye JD, et al. A retrograde-viewing device improves detection of adenomas in the colon: a prospective efficacy evaluation (with videos) Gastrointest. Endosc. 2010;71:551–556. doi: 10.1016/j.gie.2009.09.043.
    1. Gralnek IM, et al. Standard forward-viewing colonoscopy versus full-spectrum endoscopy: an international, multicentre, randomised, tandem colonoscopy trial. Lancet Oncol. 2014;15:353–360. doi: 10.1016/S1470-2045(14)70020-8.
    1. Piegl LA. Knowledge-guided Computation for Robust CAD. Computer-Aided Design and Applications. 2005;2:685–695. doi: 10.1080/16864360.2005.10738333.
    1. Soetikno RM, et al. Prevalence of nonpolypoid (flat and depressed) colorectal neoplasms in asymptomatic and symptomatic adults. JAMA. 2008;299:1027–1035. doi: 10.1001/jama.299.9.1027.
    1. Redmon, J., Divvala, S., Girshick, R. & Farhadi, A. You Only Look Once: Unified, Real-Time Object Detection. Computer Vision and Pattern Recognition, arXiv:1506.02640 (2016).
    1. Liu, L. et al. Deep Learning for Generic Object Detection: A Survey. Computer Vision and Pattern Recognition, arXiv:1809.02165 (2019).
    1. Huang, J. et al. Speed/accuracy trade-offs for modern convolutional object detectors. Computer Vision and Pattern Recognition, arXiv:1611.10012 (2017).
    1. Russakovsky O, et al. ImageNet Large Scale Visual Recognition Challenge. Int J Comput Vis. 2015;115:211–252. doi: 10.1007/s11263-015-0816-y.

Source: PubMed

Szponzorok és közreműködők

Egészségi állapot

Kábítószer-beavatkozások

3
Iratkozz fel