Hyperspectral imaging for tissue classification, a way toward smart laparoscopic colorectal surgery

Abstract

In the last decades, laparoscopic surgery has become the gold standard in patients with colorectal cancer. To overcome the drawback of reduced tactile feedback, real-time tissue classification could be of great benefit. In this ex vivo study, hyperspectral imaging (HSI) was used to distinguish tumor tissue from healthy surrounding tissue. A sample of fat, healthy colorectal wall, and tumor tissue was collected per patient and imaged using two hyperspectral cameras, covering the wavelength range from 400 to 1700 nm. The data were randomly divided into a training (75%) and test (25%) set. After feature reduction, a quadratic classifier and support vector machine were used to distinguish the three tissue types. Tissue samples of 32 patients were imaged using both hyperspectral cameras. The accuracy to distinguish the three tissue types using both hyperspectral cameras was 0.88 (STD = 0.13) on the test dataset. When the accuracy was determined per patient, a mean accuracy of 0.93 (STD = 0.12) was obtained on the test dataset. This study shows the potential of using HSI in colorectal cancer surgery for fast tissue classification, which could improve clinical outcome. Future research should be focused on imaging entire colon/rectum specimen and the translation of the technique to an intraoperative setting.

Keywords: colorectal cancer; hyperspectral imaging; machine learning; margin assessment; support vector machine.

Figures

Fig. 1

(a) Hyperspectral image, with two spatial dimensions (x,y) and one spectral dimension (λ). (b) On the right side, the spectra of the two selected pixels are shown.

Fig. 2

Registration of the HSI, RGB, and pathology images. In the upper row from left to right, annotated pathology image (yellow = fat, green = healthy colorectal wall, red = tumor, blue = mucosa), RGB image and HSI. The second row from left to right, the annotated pathology image registered to the RGB image and the RGB image registered to the HSI.

Fig. 3

Spectral bands determined for the dataset with the combination of visual and near-infrared camera images are shown together with the mean spectra of fat (yellow), healthy colorectal wall (green), and tumor (red). The 13 spectral bands all have a different gray value and are separated by black vertical lines. Between 950 and 970 nm, a gap is shown in the data. This region is not covered by the cameras.

Fig. 4

Classification of the tissue samples of one patient from the test set of the combined data set of the visual and near-infrared camera. In the first column, the RGB image of each tissue sample is shown. The second column shows the registered annotated pathology image (yellow = fat, green = muscle or healthy colorectal wall, red = tumor, blue = mucosa). In the third column, the classification based on the visual and near-infrared spectra is shown projected on the binary mask of the RGB image (yellow = fat, green = muscle of healthy colorectal wall, red = tumor). The first row shows the healthy tissue including fat and healthy colorectal wall. The second row shows the tumor tissue sample. Tissue annotated by the pathologist as mucosa (blue) is not classified and is shown a white in the third column.

Fig. 5

ROC curves of the training results of the quadratic classifier distinguishing fat from all other tissue types. The three datasets are shown as the visual camera (green), near-infrared camera (red), and the combination of the visual and near-infrared camera (blue).

Fig. 6

ROC curves of the training results of the SVMs distinguishing tumor from healthy colorectal tissue. The three datasets are shown as the visual camera (green), near-infrared camera (red), and the combination of the visual and near-infrared camera (blue).

Fig. 7

Classification of the tissue samples of one patient from the test dataset of the combined dataset. In the first column, the RGB image of each tissue sample is shown. The second column shows the registered annotated pathology H&E image (yellow = fat, green = muscle or healthy colorectal wall, red = tumor, blue = mucosa). In the third to fifth column from the classification based on the visual image only, the classification based on the near-infrared image only and the classification based on the combined visual and near-infrared image are shown, respectively, projected on the binary mask of the RGB image (yellow = fat, green = muscle of healthy colorectal wall, red = tumor). From top to bottom, fat, healthy colorectal wall, and tumor tissue are shown. Tissue annotated as mucosa (blue) by the pathologist is not classified and shown as white in the third to fifth column.