Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices

Michael D Abràmoff, Philip T Lavin, Michele Birch, Nilay Shah, James C Folk, Michael D Abràmoff, Philip T Lavin, Michele Birch, Nilay Shah, James C Folk

Abstract

Artificial Intelligence (AI) has long promised to increase healthcare affordability, quality and accessibility but FDA, until recently, had never authorized an autonomous AI diagnostic system. This pivotal trial of an AI system to detect diabetic retinopathy (DR) in people with diabetes enrolled 900 subjects, with no history of DR at primary care clinics, by comparing to Wisconsin Fundus Photograph Reading Center (FPRC) widefield stereoscopic photography and macular Optical Coherence Tomography (OCT), by FPRC certified photographers, and FPRC grading of Early Treatment Diabetic Retinopathy Study Severity Scale (ETDRS) and Diabetic Macular Edema (DME). More than mild DR (mtmDR) was defined as ETDRS level 35 or higher, and/or DME, in at least one eye. AI system operators underwent a standardized training protocol before study start. Median age was 59 years (range, 22-84 years); among participants, 47.5% of participants were male; 16.1% were Hispanic, 83.3% not Hispanic; 28.6% African American and 63.4% were not; 198 (23.8%) had mtmDR. The AI system exceeded all pre-specified superiority endpoints at sensitivity of 87.2% (95% CI, 81.8-91.2%) (>85%), specificity of 90.7% (95% CI, 88.3-92.7%) (>82.5%), and imageability rate of 96.1% (95% CI, 94.6-97.3%), demonstrating AI's ability to bring specialty-level diagnostics to primary care settings. Based on these results, FDA authorized the system for use by health care providers to detect more than mild DR and diabetic macular edema, making it, the first FDA authorized autonomous AI diagnostic system in any field of medicine, with the potential to help prevent vision loss in thousands of people with diabetes annually. ClinicalTrials.gov NCT02963441.

Keywords: Biomedical engineering; Eye manifestations.

Conflict of interest statement

Competing interestsM.D.A. is shareholder, director, and employee of IDx, LLC, and has relevant patents and patent applications assigned to the University of Iowa; P.T.L. received fees from IDx, LLC for statistical consultancy; N.S. and M.B. declare no competing interests. J.C.F. is shareholder of IDx, LLC. Disclosure forms provided by the authors are available with the full text of this article. Patents (all issued) that may be affected by this study are: applied for by the University of Iowa, inventor M.D.A., 7,474,775, Automatic Detection of Red Lesions in Digital Color Fundus Photographs; 7,712,898, Methods and Systems for Optic Nerve Head Segmentation; 8,340,437, Methods and Systems for Determining Optimal Features for Classifying Patterns or Objects in Images; 9,924,867, Automated Determination of Arteriovenous Ratio in Images of Blood Vessels; applied by IDx, inventor M.D.A., 9,155,465, Snapshot Spectral Domain Optical Coherence Tomographer; 9,782,065, Parallel optical coherence tomography apparatuses, systems and related methods; 9,814,386, Systems and methods for alignment of the eye for ocular imaging.

Figures

Fig. 1
Fig. 1
Waterfall diagram showing the final disposition of each participant in the enrolled, intention to screen (ITS), and fully analyzable populations

References

    1. Hendricks LE, Hendricks RT. Greatest fears of type 1 and type 2 patients about having diabetes: implications for diabetes educators. Diabetes Educ. 1998;24:168–173. doi: 10.1177/014572179802400206.
    1. Fong DS, et al. Diabetic retinopathy. Diabetes Care. 2003;26:226–229. doi: 10.2337/diacare.26.1.226.
    1. Centers for Disease Control and Prevention. Diabetes Report Card 2012. Atlanta, GA: U.S. Department of Health and Human Services; 2012.
    1. Bragge P, Gruen RL, Chau M, Forbes A, Taylor HR. Screening for Presence or Absence of Diabetic Retinopathy: A Meta-analysis. Arch Ophthalm. 2011;129:435–444. doi: 10.1001/archophthalmol.2010.319.
    1. National Health Service Diabetic Retinopathy Programme Annual Report, April 2007-March 2008 (2008).
    1. Liew, G., Michaelides, M. & Bunce, C. A comparison of the causes of blindness certifications in Englandand Wales in working age adults (16–64 years), 1999–2000 with 2009–2010. BMJ Open 4, e004015, 10.1136/bmjopen-2013-004015 (2014).
    1. Centers for Disease Control and Prevention. Centers for Disease Control and Prevention. National diabetes fact sheet: national estimates and general information on diabetes and prediabetes in the United States. Atlanta, GA: U.S. Department of Health and Human Services, Centers for Disease Control and Prevention; 2008.
    1. Hazin R, Colyer M, Lum F, Barazi MK. Revisiting diabetes 2000: challenges in establishing nationwide diabetic retinopathy prevention programs. Am. J. Ophthalmol. 2011;152:723–729. doi: 10.1016/j.ajo.2011.06.022.
    1. Lawrence MG. The accuracy of digital-video retinal imaging to screen for diabetic retinopathy: an analysis of two digital-video retinal imaging systems using standard stereoscopic seven-field photography and dilated clinical examination as reference standards. Trans. Am. Ophthalmol. Soc. 2004;102:321–340.
    1. Abràmoff MD, Suttorp-Schulten MS. Web-based screening for diabetic retinopathy in a primary care population: the EyeCheck project. Telemed. J. E. Health. 2005;11:668–674. doi: 10.1089/tmj.2005.11.668.
    1. Scanlon PH. The English national screening programme for sight-threatening diabetic retinopathy. J. Med. Screen. 2008;15:1–4. doi: 10.1258/jms.2008.008015.
    1. Abràmoff MD, et al. Automated analysis of retinal images for detection of referable diabetic retinopathy. JAMA Ophthalmol. 2013;131:351–357. doi: 10.1001/jamaophthalmol.2013.1743.
    1. Abràmoff MD, et al. Automated early detection of diabetic retinopathy. Ophthalmology. 2010;117:1147–1154. doi: 10.1016/j.ophtha.2010.03.046.
    1. Figueiredo IN, Kumar S, Oliveira CM, Ramos JD, Engquist B. Automated lesion detectors in retinal fundus images. Comput. Biol. Med. 2015;66:47–65. doi: 10.1016/j.compbiomed.2015.08.008.
    1. Oliveira CM, Cristovao LM, Ribeiro ML, Abreu JR. Improved automated screening of diabetic retinopathy. Ophthalmologica. 2011;226:191–197. doi: 10.1159/000330285.
    1. Abràmoff MD, et al. Improved automated detection of diabetic retinopathy on a publicly available dataset through integration of deep learning. Invest. Ophthalmol. Vis. Sci. 2016;57:5200–5206. doi: 10.1167/iovs.16-19964.
    1. Gulshan V, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA. 2016;316:2402–2410. doi: 10.1001/jama.2016.17216.
    1. Abràmoff MD, et al. Evaluation of a system for automatic detection of diabetic retinopathy from color fundus photographs in a large population of patients with diabetes. Diabetes Care. 2008;31:193–198. doi: 10.2337/dc07-1312.
    1. Hansen MB, et al. Results of automated retinal image analysis for detection of diabetic retinopathy from the Nakuru Study, Kenya. PLoS ONE. 2015;10:e0139148. doi: 10.1371/journal.pone.0139148.
    1. Diabetic Retinopathy Clinical Research Network et al. Three-year follow-up of a randomized trial comparing focal/grid photocoagulation and intravitreal triamcinolone for diabetic macular edema. Arch. Ophthalmol. 127, 245–251 (2009).
    1. PKC-DRS Group, et al. Effect of ruboxistaurin on visual loss in patients with diabetic retinopathy et al. Ophthalmology 113, 2221–2230 (2006).
    1. Li HK, et al. Comparability of digital photography with the ETDRS film protocol for evaluation of diabetic retinopathy severity. Invest. Ophthalmol. Vis. Sci. 2011;52:4717–4725. doi: 10.1167/iovs.10-6303.
    1. Gangaputra S, et al. Comparison of standardized clinical classification with fundus photograph grading for the assessment of diabetic retinopathy and diabetic macular edema severity. Retina. 2013;33:1393–1399. doi: 10.1097/IAE.0b013e318286c952.
    1. Diabetic Retinopathy Clinical Research Network et al. Aflibercept, bevacizumab, or ranibizumab for diabetic macular edema. N. Engl. J. Med. 2015;372:1193–1203. doi: 10.1056/NEJMoa1414264.
    1. Wilkinson CP, et al. Proposed international clinical diabetic retinopathy and diabetic macular edema disease severity scales. Ophthalmology. 2003;110:1677–1682. doi: 10.1016/S0161-6420(03)00475-5.
    1. American Academy of Ophthalmology Retina/Vitreous Panel & Hoskins Center for Quality Eye Care. Diabetic Retinopathy PPP - Updated 2017 (San Francisco, CA: American Academy of Ophthalmology, 2017).
    1. Pugh JA, et al. Screening for diabetic retinopathy: The wide-angle retinal camera. Diabetes Care. 1993;16:889–895. doi: 10.2337/diacare.16.6.889.
    1. Lin DY, Blumenkranz MS, Brothers RJ, Grosvenor DM. The sensitivity and specificity of single-field nonmydriatic monochromatic digital fundus photography with remote image interpretation for diabetic retinopathy screening: a comparison with ophthalmoscopy and standardized mydriatic color photography. Am. J. Ophthalmol. 2002;134:204–213. doi: 10.1016/S0002-9394(02)01522-2.
    1. Diabetes Prevention Program Research Group. Long-term effects of lifestyle intervention or metformin on diabetes development and microvascular complications over 15-year follow-up: the Diabetes Prevention Program Outcomes Study. Lancet Diabetes Endocrinol.3, 866–875 (2015).
    1. [No authors listed] The relationship of glycemic exposure (HbA1c) to the risk of development and progression of retinopathy in the diabetes control and complications trial. Diabetes44, 968–983 (1995).
    1. Ahmed J, et al. The sensitivity and specificity of nonmydriatic digital stereoscopic retinal imaging in detecting diabetic retinopathy. Diabetes Care. 2006;29:2205–2209. doi: 10.2337/dc06-0295.
    1. Wang YT, Tadarati M, Wolfson Y, Bressler SB, Bressler NM. Comparison of prevalence of diabetic macular edema based on monocular fundus photography vs optical coherence tomography. JAMA Ophthalmol. 2016;134:222–228. doi: 10.1001/jamaophthalmol.2015.5332.
    1. Klonoff DC, Schwartz DM. An economic analysis of interventions for diabetes. Diabetes Care. 2000;23:390–404. doi: 10.2337/diacare.23.3.390.
    1. Moyer VA, U.S. Preventive Services Task Force Screening for glaucoma: U.S. Preventive ServicesTask Force Recommendation Statement. Ann. Intern. Med. 2013;159:484–489.
    1. Chou R, Dana T, Bougatsos C, Grusing S, Blazina I. Screening for impaired visual acuity in older adults: updated evidence report and systematic review for the US Preventive Services Task Force. JAMA. 2016;315:915–933. doi: 10.1001/jama.2016.0783.
    1. Hollands H, et al. Do findings on routine examination identify patients at risk for primary open-angle glaucoma? The rational clinical examination systematic review. JAMA. 2013;309:2035–2042. doi: 10.1001/jama.2013.5099.
    1. Age-Related Eye Disease Study Research Group. A randomized, placebo-controlled, clinical trial of high-dose supplementation with vitamins C and E, beta carotene, and zinc for age-related macular degeneration and vision loss: AREDS Report 8. Arch. Ophthalmol.119, 1417–1436 (2001).
    1. Abràmoff MD, et al. Automated segmentation of the optic disc from stereo color photographs using physiologically plausible features. Invest. Ophthalmol. Vis. Sci. 2007;48:1665–1673. doi: 10.1167/iovs.06-1081.
    1. Shah, Abhay, et al. "Susceptibility to misdiagnosis of adversarial images by deep learning based retinal image analysis algorithms." In Biomedical Imaging (ISBI 2018), 2018 IEEE 15th International Symposium on, pp. 1454–1457. IEEE, 2018.
    1. Friedenwald J, Day R. The vascular lesions of diabetic retinopathy. Bull. Johns. Hopkins Hosp. 1950;86:253–254.
    1. US Food and Drug Administration. FDA permits marketing of artificial intelligence-based device to detect certain diabetes-related eye problems. April 12, 2018 (Washington, DC, 2018).
    1. Niemeijer M, Abràmoff MD, van Ginneken B. Image structure clustering for image quality verification of color retina images in diabetic retinopathy screening. Med. Image Anal. 2006;10:888–898. doi: 10.1016/j.media.2006.09.006.
    1. Abràmoff, M. D., Staal, J., Suttorp, M. S. A., Polak, B. C. & Viergever, M. A. Low level screening of exudates and hemorrhages in background diabetic retinopathy. Comp. Assi. Fun. Image Anal., 15 (2000).
    1. Niemeijer M, van Ginneken B, Russell SR, Suttorp-Schulten MSA, Abràmoff MD. Automated Detection and Differentiation of Drusen, Exudates, and Cotton-Wool Spots in Digital Color Fundus Photographs for Diabetic Retinopathy Diagnosis. Invest. Ophthalmol. Vis. Sci. 2007;48:2260–2267. doi: 10.1167/iovs.06-0996.
    1. Niemeijer M, van Ginneken B, Staal J, Suttorp-Schulten MS, Abràmoff MD. Automatic detection of red lesions in digital color fundus photographs. IEEE Trans. Med. Imaging. 2005;24:584–592. doi: 10.1109/TMI.2005.843738.
    1. Niemeijer M, Abràmoff MD, van Ginneken B. Information fusion for diabetic retinopathy CAD in digital color fundus photographs. IEEE Trans. Med. Imaging. 2009;28:775–785. doi: 10.1109/TMI.2008.2012029.
    1. Krizhevsky, A., Sutskever, I. & Hinton, G. E. ImageNet Classification with Deep Convolutional Neural Networks; in Advances in neural information processing systems 1097–1105 (Neural Information Processing Systems Foundation, Inc., California, 2012).
    1. Quellec G, Russell SR, Abràmoff MD. Optimal filter framework for automated, instantaneous detection of lesions in retinal images. IEEE Trans. Med. Imaging. 2011;30:523–533. doi: 10.1109/TMI.2010.2089383.
    1. Quellec G, Abràmoff MD. Estimating maximal measurable performance for automated decision systems from the characteristics of the reference standard. application to diabetic retinopathy screening. Conf. Proc. Ieee. Eng. Med. Biol. Soc. 2014;2014:154–157.
    1. Gauthier I, Anderson AW, Tarr MJ, Skudlarski P, Gore JC. Levels of categorization in visual recognition studied using functional magnetic resonance imaging. Curr. Biol. 1997;7:645–651. doi: 10.1016/S0960-9822(06)00291-0.
    1. Polk TA, Farah MJ. The neural development and organization of letter recognition: evidence from functional neuroimaging, computational modeling, and behavioral studies. Proc. Natl Acad. Sci. USA. 1998;95:847–852. doi: 10.1073/pnas.95.3.847.
    1. Farah MJ, Aguirre GK. Imaging visual recognition: PET and fMRI studies of the functional anatomy of human visual recognition. Trends Cogn. Sci. 1999;3:179–186. doi: 10.1016/S1364-6613(99)01309-1.
    1. Harley EM, et al. Engagement of fusiform cortex and disengagement of lateral occipital cortex in the acquisition of radiological expertise. Cereb. Cortex. 2009;19:2746–2754. doi: 10.1093/cercor/bhp051.
    1. Lynch SK, Abràmoff MD. Diabetic retinopathy is a neurodegenerative disorder. Vision Res. 2017;139:101–107. doi: 10.1016/j.visres.2017.03.003.
    1. American Diabetes Association. Classification and diagnosis of diabetes. Diabetes Care38 Suppl, S8–S16 (2015).
    1. American Diabetes Association. Diagnosis and classification of diabetes mellitus. Diabetes Care 37 Suppl 1, S81–90 (2014).
    1. Chew EY, et al. Evaluation of the age-related eye disease study clinical lens grading system AREDS report No. 31. Ophthalmology. 2010;117:2112–2119 e2113. doi: 10.1016/j.ophtha.2010.02.033.
    1. Early Treatment Diabetic Retinopathy Study Research Group. Fundus photographic risk factors forprogression of diabetic retinopathy. ETDRS report number 12. Ophthalmology98, 823–833 (1991).
    1. Li HK, et al. Monoscopic versus stereoscopic retinal photography for grading diabetic retinopathy severity. Invest. Ophthalmol. Vis. Sci. 2010;51:3184–3192. doi: 10.1167/iovs.09-4886.
    1. Firth D. Bias reduction of maximum-likelihood-estimates. Biometrika. 1993;80:27–38. doi: 10.1093/biomet/80.1.27.

Source: PubMed

Спонсоры и соавторы

Заболевания

Медикаментозные вмешательства

Подписаться