Tumor detection of the thyroid and salivary glands using hyperspectral imaging and deep learning

Abstract

The performance of hyperspectral imaging (HSI) for tumor detection is investigated in ex-vivo specimens from the thyroid (N = 200) and salivary glands (N = 16) from 82 patients. Tissues were imaged with HSI in broadband reflectance and autofluorescence modes. For comparison, the tissues were imaged with two fluorescent dyes. Additionally, HSI was used to synthesize three-band RGB multiplex images to represent the human-eye response and Gaussian RGBs, which are referred to as HSI-synthesized RGB images. Using histological ground truths, deep learning algorithms were developed for tumor detection. For the classification of thyroid tumors, HSI-synthesized RGB images achieved the best performance with an AUC score of 0.90. In salivary glands, HSI had the best performance with 0.92 AUC score. This study demonstrates that HSI could aid surgeons and pathologists in detecting tumors of the thyroid and salivary glands.

Conflict of interest statement

The authors have no relevant financial interests in this article and no potential conflicts of interest to disclose. Informed consent was obtained from all patients in accordance with Emory IRB policies under the Head and Neck Satellite Tissue Bank protocol.

© 2020 Optical Society of America under the terms of the OSA Open Access Publishing Agreement.

Figures

Fig. 1.

Average hyperspectral signatures for the tissues in the thyroid cohort (a) and the salivary cohort (b). Subplots show the spectral signature with standard deviation for each tissue sub type in both cohorts.

Fig. 2.

A representative specimen of thyroid cancer. (a) Left to right: RGB image from standard RGB camera; HSI-synthesized RGB human-eye multiplex image made from reflectance HSI using Vos et al. 1978 method; HSI-synthesized Gaussian RGB multiplex image made from reflectance HSI. (b) Spectral signatures of human-eye color perception of red (R), green (G), and blue (B) colors proposed by Vos et al. 1978. (c) Gaussian kernels used for generating Gaussian RGB multiplex images.

Fig. 3.

Schematic depicting the experimental design of fully-independent training, validation, and testing paradigms for the 76 patient thyroid tumor cohort.

Fig. 4.

Flow diagram of intra-patient experiments of the salivary gland, with representative tumor of the parotid gland. Intra-patient T and N tissues were used for MLP (multilayer perceptron) training, and TN tissue specimens were used for testing. The histological ground truth is shown with tumor contour in green. The predicted tumor heat-map overlay onto the RGB image is shown with tumor predictions (red) and normal predictions (green). Areas of specular glare in the heat-map are not classified, and the ground-truth tumor contour is in blue.

Fig. 5.

Schematic diagram of the modified inception v4 CNN architecture. The CNN was customized to operate on the 25×25×91 patch-size selected. The receptive field size and number of convolutional filters is shown at bottom of each inception block. The convolutional kernel size used for convolutions is shown in italics inside each convolution box. Squeeze-and-excitation modules were added to the CNN to increase performance.

Fig. 6.

Average and median AUC scores from thyroid tumor detection. (a) average AUC scores for thyroid tumor detection across all tissue specimens grouped by tumor subtype; statistical significance (*, p < 0.05) is shown above. (b) median AUC scores of tumor subtype detection.

Fig. 7.

Representative tissue images and corresponding classification heat-maps from all modalities from patients with thyroid carcinoma. Columns from left to right: histology, HSI with heat-map, HSI-synthesized Gaussian-RGB multiplex with heat-map, HSI-synthesized human-eye RGB multiplex with heat-map, autofluorescence with heat-map, 2-NBDG dye image with heat-map, Proflavin dye image with heat-map. Rows from top to bottom: papillary thyroid carcinoma (PTC), medullary thyroid carcinoma (MTC), follicular thyroid carcinoma, and poorly differentiated thyroid carcinoma. The contours in white (in heat-maps) and green (on histology) outline the cancerous regions. Predicted tumor heat-maps range from dark blue (predicted normal) to dark red (predicted cancer).

Fig. 8.

Differences in AUC score performance comparing HSI against HSI-synthesized Gaussian-RGB multiplex and HSI-synthesized human-eye RGB multiplexing. (a) Histogram of percent difference in AUC scores of tissue specimens between HSI and HSI-synthesized Gaussian-RGB multiplex imaging. The arrows show the bins that contain the patient specimens shown in the right panels, which are the two worst performing tissues. (b) RGB image of tissue specimen with large difference in AUC score performance between heat-maps produced from HSI, HSI-synthesized Gaussian multiplex, and HSI-synthesized human-eye multiplex image. (c) Histogram of percent difference in AUC scores of tissue specimens between HSI and HSI-synthesized human-eye RGB multiplex imaging. The arrows show the bins that contain the patient specimens shown in the right panels, which are the two worst performing tissues. (d) RGB image of the tissue specimen with the largest difference in AUC score performance between heat-maps produced from HSI, HSI-synthesized Gaussian multiplex, and HSI-synthesized human-eye multiplex image. The tumor margin is delineated in white.

Fig. 9.

AUC score results from one fold of the testing data comparing different methods using HSI. The HSI-synthesized RGB multiplex images represent RGB imaging with different parameters. From left to right in the plot: original HSI method, 3-band Gaussian-RGB from HSI, original HSI-synthesized human-eye RGB from HSI, human-eye RGB from HSI synthesized with half of the 400-500 nm red component, and last the human-eye RGB from HSI synthesized with none of the 400-500 nm red component. Values shown are average AUC score from all tissues in one fold of testing data with 95% confidence interval error bars.

Fig. 10.

Mean spectral signatures of correctly-classified normal thyroid tissues (a) and thyroid tumors (b). The saliency of spectral features is identified below each plot using the grad-CAM technique. Red hues represent the most important features for correctly predicting each tissue class, and blue color hues represent less important wavelengths for correctly predicting each class.