Assessing Rectal Cancer Treatment Response Using Coregistered Endorectal Photoacoustic and US Imaging Paired with Deep Learning

Abstract

Background Conventional radiologic modalities perform poorly in the radiated rectum and are often unable to differentiate residual cancer from treatment scarring. Purpose To report the development and initial patient study of an imaging system comprising an endorectal coregistered photoacoustic (PA) microscopy (PAM) and US system paired with a convolution neural network (CNN) to assess the rectal cancer treatment response. Materials and Methods In this prospective study (ClinicalTrials.gov identifier NCT04339374), participants completed radiation and chemotherapy from September 2019 to September 2020 and images were obtained with the PAM/US system prior to surgery. Another group's colorectal specimens were studied ex vivo. The PAM/US system consisted of an endorectal imaging probe, a 1064-nm laser, and one US ring transducer. The PAM CNN and US CNN models were trained and validated to distinguish normal from malignant colorectal tissue using ex vivo and in vivo patient data. The PAM CNN and US CNN were then tested using additional in vivo patient data that had not been seen by the CNNs during training and validation. Results Twenty-two patients' ex vivo specimens and five patients' in vivo images (a total of 2693 US regions of interest [ROIs] and 2208 PA ROIs) were used for CNN training and validation. Data from five additional patients were used for testing. A total of 32 participants (mean age, 60 years; range, 35-89 years) were evaluated. Unique PAM imaging markers of the complete tumor response were found, specifically including recovery of normal submucosal vascular architecture within the treated tumor bed. The PAM CNN model captured this recovery process and correctly differentiated these changes from the residual tumor. The imaging system remained highly capable of differentiating tumor from normal tissue, achieving an area under the receiver operating characteristic curve of 0.98 (95% CI: 0.98, 0.99) for data from five participants. By comparison, the US CNN had an area under the receiver operating characteristic curve of 0.71 (95% CI: 0.70, 0.73). Conclusion An endorectal coregistered photoacoustic microscopy/US system paired with a convolutional neural network model showed high diagnostic performance in assessing the rectal cancer treatment response and demonstrated potential for optimizing posttreatment management. © RSNA, 2021 Supplemental material is available for this article. See also the editorial by Klibanov in this issue.

Conflict of interest statement

Disclosures of Conflicts of Interest: X.L. Activities related to the present article: disclosed no relevant relationships. Activities not related to the present article: disclosed no relevant relationships. Other relationships: Washington University filed a provisional U.S. patent application based on this work. K.M.S.U. Activities related to the present article: disclosed no relevant relationships. Activities not related to the present article: disclosed no relevant relationships. Other relationships: Washington University filed a provisional U.S. patent application based on this work. W.C. Activities related to the present article: disclosed no relevant relationships. Activities not related to the present article: disclosed no relevant relationships. Other relationships: Washington University filed a provisional U.S. patent application based on this work. H.L. disclosed no relevant relationships. S.K. Activities related to the present article: disclosed no relevant relationships. Activities not related to the present article: disclosed no relevant relationships. Other relationships: Washington University filed a provisional U.S. patent application based on this work. E.A. disclosed no relevant relationships. G.Y. disclosed no relevant relationships. D.C. disclosed no relevant relationships. A.S. disclosed no relevant relationships. S.H. disclosed no relevant relationships. M.M. disclosed no relevant relationships. Q.Z. Activities related to the present article: disclosed no relevant relationships. Activities not related to the present article: disclosed no relevant relationships. Other relationships: Washington University filed a provisional U.S. patent application based on this work.

Figures

Graphical abstract

Figure 1:

Acoustic-resolution (AR) photoacoustic microscopy (PAM) system. A, Photograph of the AR PAM/US imaging probe. B, Rotation components of the AR PAM probe. C, Schematic drawing of the AR PAM system. D, Imaging head covered by a water-inflated balloon. E, Cross-section of the imaging head. The design of the AR PAM probe is based on a Food and Drug Administration–approved conventional endorectal US probe (BK Medical). With this design, our AR-PAM/US probe fits perfectly into standard rigid proctoscopes and is easily operated by surgeons familiar with performing standard endorectal US examinations. DAQ = data acquisition, Nd:YAG = neodymium-doped yttrium aluminum garnet, PA = photoacoustic, PC = personal computer.

Figure 2:

Design of the convolutional neural network (CNN) architecture for normal layered or layerlike colorectal tissue identification. Both the photoacoustic microscopy (PAM) CNN and the US CNN contain two sequential feature extraction layers and two fully connected layers. The second fully connected layer is the output layer, which has only two outputs, corresponding to either normal colorectal tissue (referred to as a layered tissue structure for US and a layerlike vascular distribution for PAM) or to abnormal or malignant colorectal tissue. The output layer has a softmax activation function that predicts the probability of an input image being in a certain class (eg, a layered normal tissue structure for US and layerlike vascular distribution for PAM).

Figure 3:

An example of region-of-interest (ROI) selections for training the US and photoacoustic microscopy (PAM) convolutional neural network (CNN) models. A 68-year-old man with highly invasive rectal cancer treated initially with chemotherapy and radiation was imaged immediately prior to large pelvic resection. A, T2- and diffusion-weighted (DWI) MRI scans show a residual tumor with corresponding intermediate-to-low T2 signal intensity and diffusion restriction from the 10- to 6-o’clock positions (arrows). B, US image of the rectum, including the treated cancer region. Boxes indicate ROIs selected uniformly in the tumor bed area. C, Coregistered PAM/US image. PAM images in boxes are selected to train and validate CNNs. D, Tissue specimen shows the interface of normal and cancer regions, with blood vessels (arrows) in the normal region (hematoxylin-eosin [H&E] stain). E, US image, F, coregistered PAM/US image, and,G, hematoxylin-eosin–stained specimen of the normal rectum. Boxes are ROIs uniformly selected from a normal distal region. The color map in, F, is set to hot in the MATLAB code, ranging in magnitude from black (minimum) to red to yellow to white (maximum). All figures envelop data in a logarithmic scale, in which the maximum pixel value (white) refers to 1 V, which corresponds to the maximum signal level. AR = acoustic resolution.

Figure 4:

US and photoacoustic microscopy (PAM)/US images of normal human rectum walls from five testing patients unseen by the convolutional neural networks (CNNs). A1–A3, Patient 1.A1, US image of a normal rectum region proximal to tumor bed. A2, PAM/US image of A1. A3, Image of hematoxylin-eosin (H&E)–stained normal area shows rich vessels in the submucosa (arrows).B1–B3, Patient 2.B1, US image distal to tumor bed. B2, PAM/US image. B3, Image of hematoxylin-eosin–stained specimen from the normal rectum. C1–C3, Patient 3.C1, US image distal to tumor bed. C2, PAM/US image. C3, Image of the hematoxylin-eosin–stained normal rectum.D1–D3, Patient 4. D1, US image distal to tumor bed. D2, PAM/US image.E1–E3, Patient 5. E1, US image proximal to tumor bed. E2, PAM/US image.D3 and E3 are corresponding images of the hematoxylin-eosin–stained specimens from the normal rectum. The numbers in the US and PAM panels are the corresponding average US CNN and PAM CNN outputs indicating the probability of normal findings (Table).

Figure 5:

Endoscopic images, T2- and diffusion-weighted (DWI) MRI images, US images, photoacoustic microscopy (PAM)/US images, and images of hematoxylin–eosin (H&E)–stained specimens from the tumor beds of five human rectum walls; none of these images had previously been seen by the convolutional neural networks (CNNs) during validation and training. The percentages in the US and PAM panels are the corresponding average US CNN and PAM CNN outputs. A1–A5, Patient 1. A1, Endoscopy reveals residual ulceration in the lateral rectal wall.A2, T2-weighted MRI (left) and DWI (right) scans show an intermediate T2 signal and DWI hyperintensity corresponding to diffusion restriction in residual cancer (arrow). A3, US image of the rectum with the tumor bed (box outline) (an air bubble is trapped inside the balloon at the 12-o’clock position) shows distorted morphologic characteristics. A4, PAM/US image of A3shows a dearth of photoacoustic signal intensity. A5, Tissue specimen of the residual tumor region shows the tumor region lacks blood vessels (hematoxylin-eosin stain). B1–B5, Patient 2. B1, Posttreatment endoscopy of original tumor bed. B2, T2-weighted MRI scan (left) shows low T2 signal intensity in the right rectal wall. DWI scan (right) shows no hyperintensity to suggest a recurrent tumor. B3, PAM/US image shows some acoustic attenuation contrast (box outline). B4, PAM image reveals a region lacking vasculature. B5, Tissue specimen reveals a recurrence tumor of approximately 1.2 cm underneath the surface (hematoxylin-eosin stain). C1–C5, Patient 3.C1, Posttreatment endoscopic image of the rectal tumor bed shows fibrosis and scarring. C2, T2-weighted MRI scan (left) shows low T2 signal. DWI scan (right) shows completely normal hypointense signals.C3, US image shows normal-appearing rectal tissue in the treated tumor bed. C4, PAM image reveals a microvascularly deficient region corresponding to the fibrotic tumor bed.C5, Tissue specimen reveals scattered tumor cells throughout the tumor bed and shows invasion through the muscularis (hematoxylin-eosin stain). D1–D5, Patient 4.D1, Endoscopic images show scar tissue of the treated tumor. D2, T2-weighted MRI scan (left) shows a nonviable hypointense treated tumor along the left posterior margin of the rectum (red arrow). Additionally, a nodule of intermediate T2 signal intensity is present along the anterior left margin (blue arrow). DWI scan (right) shows hyperintensity within the nodule (blue arrow), suspicious for residual viable tumor. D3, Endorectal US image shows normal-appearing rectal tissue in the treated tumor; however, inD4, a PAM image reveals a paucity of blood vessels within the tumor bed region. D5, Tissue specimen reveals residual cancer (hematoxylin-eosin stain). E1–E5, Patient 5. E1, Endoscopic image. E2, T2-weighted MRI scan (left) shows a reduction in the volume of the mass; however, persistent focus of intermediate T2 signal intensity along the left wall of the rectum, and the DWI scan (right) shows residual hyperintensity corresponding to the abnormal T2 signal (arrow), suggestive of residual viable tumor. E3, US image appears normal in the tumor bed region as marked. E4, PAM image shows uniform layerlike submucosal microvascular distribution. E5, Surgical pathologic analysis reveals ulcer and granulation tissue in the treated tumor bed; however, no residual cancer is found. AR = acoustic resolution.

Figure 6:

Receiver operating characteristic curves in five testing participants with data not previously seen by the convolutional neural networks (CNNs). A, Photoacoustic microscopy–CNN testing results show an area under the receiver operating characteristic curve (AUC) of 0.98. B, US CNN testing results show an AUC of 0.71.