Accuracy of a Machine-Learning Algorithm for Detecting and Classifying Choroidal Neovascularization on Spectral-Domain Optical Coherence Tomography

Abstract

Background: To evaluate the performance of a machine-learning (ML) algorithm to detect and classify choroidal neovascularization (CNV), secondary to age-related macular degeneration (AMD) on spectral-domain optical coherence tomography (SD-OCT) images.

Methods: Baseline fluorescein angiography (FA) and SD-OCT images from 1037 treatment-naive study eyes and 531 fellow eyes, without advanced AMD from the phase 3 HARBOR trial (NCT00891735), were used to develop, train, and cross-validate an ML pipeline combining deep-learning-based segmentation of SD-OCT B-scans and CNV classification, based on features derived from the segmentations, in a five-fold setting. FA classification of the CNV phenotypes from HARBOR was used for generating the ground truth for model development. SD-OCT scans from the phase 2 AVENUE trial (NCT02484690) were used to externally validate the ML model.

Results: The ML algorithm discriminated CNV absence from CNV presence, with a very high accuracy (area under the receiver operating characteristic [AUROC] = 0.99), and classified occult versus predominantly classic CNV types, per FA assessment, with a high accuracy (AUROC = 0.91) on HARBOR SD-OCT images. Minimally classic CNV was discriminated with significantly lower performance. Occult and predominantly classic CNV types could be discriminated with AUROC = 0.88 on baseline SD-OCT images of 165 study eyes, with CNV from AVENUE.

Conclusions: Our ML model was able to detect CNV presence and CNV subtypes on SD-OCT images with high accuracy in patients with neovascular AMD.

Keywords: age-related macular degeneration; choroidal neovascularization; classification; machine learning; optical coherence tomography.

Conflict of interest statement

A. Maunz, F. Benmansour, Y. Li, T. Albrecht, Y.-P. Zhang, F. Arcadu, and J. Sahni, F. Hoffmann-La Roche Ltd., Basel, Switzerland (E); Y. Zheng, F. Hoffmann-La Roche Ltd., Basel, Switzerland (F); S. Madhusudhan, F. Hoffmann-La Roche Ltd., Basel, Switzerland (F).

Figures

Figure 1

Example of the automated layer segmentation of 13 retinal layers. BM, Bruch’s membrane; BMEIS, boundary of myoid and ellipsoid inner segments; GCL-IPL, ganglion cell layer-inner plexiform layer; IB OPR, inner boundary outer photoreceptor; IB RPE, inner boundary retinal pigment epithelium; ILM, internal limiting membrane; INL-OPL, inner nuclear layer-outer plexiform layer; IPL-INL, inner plexiform layer-inner nuclear layer; ISJ OSJ, inner segment/outer segment junction; OB OPR, outer boundary outer photoreceptor; OB RPE, outer boundary retinal pigment epithelium; OPL-HFL, outer plexiform layer-Henle’s fiber layer; and RNFL-GCL, retinal nerve fiber layer-ganglion cell layer.

Figure 2

Segmentation pipeline. Sketch of the segmentation pipeline, involving training, prediction, and feature calculation for both fluidic and layer features. See Supplementary Table S2 for a detailed feature list.

Figure 3

ROC analysis of predominantly classic versus occult (best-tuned performance). Sensitivity versus specificity for all possible ROC cutoff points with respect to the predicted occult scores in HARBOR, including 95% CIs (bootstrapped). The location of the red crosshair indicates the operating point of the model. AUROC, area under the receiver operating characteristic; FA, fluorescein angiography; NEG, negative; POS, positive; ROC, receiver operating characteristic; Sens, sensitivity; and Spec, specificity.

Figure 4

(A) Recursive feature elimination cross-validation. Optimal performance for predominantly classic versus occult was reached with 101 features out of 106, and only 21 features were necessary to sustain the average model performance of 91% AUROC. (B) Distribution of the top 20 feature values in the training data (predominantly classic vs. occult classes).

Figure 4

(A) Recursive feature elimination cross-validation. Optimal performance for predominantly classic versus occult was reached with 101 features out of 106, and only 21 features were necessary to sustain the average model performance of 91% AUROC. (B) Distribution of the top 20 feature values in the training data (predominantly classic vs. occult classes).

Figure 5

Representative cases showing comparison of machine algorithm with angiography. (A–D) Central SD-OCT B-scans (top), with segmented pixel masks of volumetric measures and Bruch’s membrane (middle left), en-face projections (middle center), and thickness maps (middle right), as well as corresponding FAs (bottom). Colors on the SD-OCT images indicate volumetric measures as follows—intraretinal fluid (red), subretinal fluid (green), PED (blue), and SHRM (cyan). Bruch’s membrane is shown as a red line. In (A), FA shows an area of hypofluorescence due to hemorrhage, and a well-demarcated area of hyperfluorescence due to a predominantly classic CNV that leaks in later frames. This was also identified as classic CNV by our ML algorithm, due to increased SHRM height and volume. In (B), FA demonstrates an ill-defined area of stippled hyperfluorescence, due to an occult CNV that leaks diffusely in mid and late frames, and was also identified as occult CNV by the ML algorithm, due to the presence of the PED. In (C), FA shows an area of well-defined hyperfluorescence in mid frames that stains and leaks in late frames due to fibrosis. The image was classified as classic CNV by the reading center, but was identified as occult CNV by the ML algorithm due to low SHRM height and volume. In (D), FA shows an area of hypofluorescence due to hemorrhage and a poorly demarcated area of hyperfluorescence due to the CNV. This lesion was defined as minimally classic by the reading center, but was identified as classic CNV by the ML algorithm due to the SHRM created by the hemorrhage. CNV, choroidal neovascularization; FA, fluorescein angiogram; ML, machine learning; PED, pigment epithelium detachment; SD-OCT, spectral-domain optical coherence tomography; and SHRM, subretinal hyperreflective material.

Figure 5

Representative cases showing comparison of machine algorithm with angiography. (A–D) Central SD-OCT B-scans (top), with segmented pixel masks of volumetric measures and Bruch’s membrane (middle left), en-face projections (middle center), and thickness maps (middle right), as well as corresponding FAs (bottom). Colors on the SD-OCT images indicate volumetric measures as follows—intraretinal fluid (red), subretinal fluid (green), PED (blue), and SHRM (cyan). Bruch’s membrane is shown as a red line. In (A), FA shows an area of hypofluorescence due to hemorrhage, and a well-demarcated area of hyperfluorescence due to a predominantly classic CNV that leaks in later frames. This was also identified as classic CNV by our ML algorithm, due to increased SHRM height and volume. In (B), FA demonstrates an ill-defined area of stippled hyperfluorescence, due to an occult CNV that leaks diffusely in mid and late frames, and was also identified as occult CNV by the ML algorithm, due to the presence of the PED. In (C), FA shows an area of well-defined hyperfluorescence in mid frames that stains and leaks in late frames due to fibrosis. The image was classified as classic CNV by the reading center, but was identified as occult CNV by the ML algorithm due to low SHRM height and volume. In (D), FA shows an area of hypofluorescence due to hemorrhage and a poorly demarcated area of hyperfluorescence due to the CNV. This lesion was defined as minimally classic by the reading center, but was identified as classic CNV by the ML algorithm due to the SHRM created by the hemorrhage. CNV, choroidal neovascularization; FA, fluorescein angiogram; ML, machine learning; PED, pigment epithelium detachment; SD-OCT, spectral-domain optical coherence tomography; and SHRM, subretinal hyperreflective material.

Figure 5

Representative cases showing comparison of machine algorithm with angiography. (A–D) Central SD-OCT B-scans (top), with segmented pixel masks of volumetric measures and Bruch’s membrane (middle left), en-face projections (middle center), and thickness maps (middle right), as well as corresponding FAs (bottom). Colors on the SD-OCT images indicate volumetric measures as follows—intraretinal fluid (red), subretinal fluid (green), PED (blue), and SHRM (cyan). Bruch’s membrane is shown as a red line. In (A), FA shows an area of hypofluorescence due to hemorrhage, and a well-demarcated area of hyperfluorescence due to a predominantly classic CNV that leaks in later frames. This was also identified as classic CNV by our ML algorithm, due to increased SHRM height and volume. In (B), FA demonstrates an ill-defined area of stippled hyperfluorescence, due to an occult CNV that leaks diffusely in mid and late frames, and was also identified as occult CNV by the ML algorithm, due to the presence of the PED. In (C), FA shows an area of well-defined hyperfluorescence in mid frames that stains and leaks in late frames due to fibrosis. The image was classified as classic CNV by the reading center, but was identified as occult CNV by the ML algorithm due to low SHRM height and volume. In (D), FA shows an area of hypofluorescence due to hemorrhage and a poorly demarcated area of hyperfluorescence due to the CNV. This lesion was defined as minimally classic by the reading center, but was identified as classic CNV by the ML algorithm due to the SHRM created by the hemorrhage. CNV, choroidal neovascularization; FA, fluorescein angiogram; ML, machine learning; PED, pigment epithelium detachment; SD-OCT, spectral-domain optical coherence tomography; and SHRM, subretinal hyperreflective material.

Figure 6

SHAP analysis external validation. SHAP analysis for the CNV type predictions in AVENUE. Every prediction contributes exactly one dot to each row. Blue and red colors indicate lower and higher feature values, respectively. SHAP values (x-axis) add up to the predicted probability for occult (only 20 features with highest SHAP variance shown here). BM, Bruch’s membrane; CNV, choroidal neovascularization; HFL, Henle’s fiber layer; IB, inner boundary; ILM, inner limiting membrane; IRF, intraretinal fluid; max, maximum; OB, outer boundary; OPL, outer plexiform layer; PED, pigment epithelial detachment; RPE, retinal pigment epithelium; SHAP, SHapley Additive exPlanations; and SHRM, subretinal hyperreflective material.

Figure 7

ROC analysis of predominantly classic versus occult external validation. Sensitivity versus specificity for all possible cutoff points with respect to predicted occult scores in AVENUE, including 95% CIs (bootstrapped). The location of the red crosshair indicates the operating point of the model. AUROC, area under the receiver operating characteristic; FA, fluorescein angiography; NEG, negative; POS, positive; ROC, receiver operating characteristic; Sens, sensitivity; and Spec, specificity.