Review of fluorescence guided surgery visualization and overlay techniques

Abstract

In fluorescence guided surgery, data visualization represents a critical step between signal capture and display needed for clinical decisions informed by that signal. The diversity of methods for displaying surgical images are reviewed, and a particular focus is placed on electronically detected and visualized signals, as required for near-infrared or low concentration tracers. Factors driving the choices such as human perception, the need for rapid decision making in a surgical environment, and biases induced by display choices are outlined. Five practical suggestions are outlined for optimal display orientation, color map, transparency/alpha function, dynamic range compression, and color perception check.

Keywords: (170.1610) Clinical applications; (170.3880) Medical and biological imaging; (170.4730) Optical pathology; (260.2510) Fluorescence; (330.1730) Colorimetry; (330.5020) Perception psychology.

Figures

Fig. 1

(A) The number of publications in “fluorescence-guided surgery” or “fluorescence-guided resection” in the past 25 years, showing the expotential growth in the field. (B) The Novadaq SPY Elite fluorescence imaging system, which has been at the forefront of the effort to expand fluorescence guided surgery capabilities, leading the commercial market. (C) Laproscopic images acquired under white-light and (D) by exciting indocyanine green which has been pseudocolored blue and overlaid onto C. (Source: Luigi Boni, MD [6]) showing a novel use of this for perfusion imaging of tissue.

Fig. 2

(A) The sensitivity of human photoreceptors for different wavelengths of light is shown (NB: the abscissa is defined according to a logarithmic scale). (B) The emission spectra of four common FGS fluorophores: fluorescein sodium (FS), protoporphyrin IX (PpIX), IRDye® 800CW, and indocyanine green (ICG). (C) The CIE 1931 x,y chromaticity map showing the sRGB gamut used by most LED and LCD monitors, the trajectory of the exemplary color map (koufonisi), and the gamut representing brain tissue. The koufonisi colormap is perceptually balanced and has mid-high colors which circumscribe the brain tissue gamut, giving a uniform chromatic contrast. The average brain tissue gamut was characterized from intracranial images acquired from 10 patients, of which (D) is an example.

Fig. 3

(A) A representative sample of color maps used in medical imaging overlays (available in the OiM Overlay GUI), which include sequential, diverging, and categorical palettes. Examples of their use from literature include: (B) Doppler ultrasound image of placentia previa of blood flow using an opaquely-overlaid diverging hot/cold color map centered about a luminance nadir [34], (C) Axial fused PET/MR image, with FDG SUV values encoded by a hot color map and blended by a uniform transparency function [35], (D) Zeiss OPMI infrared 800 blood flow module showing a pseudocolor representation of ICG wash-in delay during arteriovenous malformation [36], (E) the resection of a sentinel lymph node in laproscopic surgery detected using ICG [37].

Fig. 4

(A) The visual fluorescence emission from PpIX under blue light excitation on the Zeiss Pentero OPMI 800 surgical microscope during glioma resection, and (B) the RGB image acquired with white-light (~5500 K) illumination. (C) The PpIX concentration map recovered using hyperspectral imaging. (D) The [PpIX] is visualized using the multivariate koufonisi colormap and overlaid on the RGB image using the logistic function in (H) [max = 0.78, midpoint = 11.6 ug/ml, k = 11.8]. (E) The same information as in Panel C, but visualized using the myCarta cube1 color map (F) and as a single-value [RGB (7, 246, 64)] color map blended into the RGB image with the same transparency function (Panel H). (G) The 1931 CIE xy chromaticity plot showing the trajectories of the three color maps and the gamut from the RGB image.

Fig. 5

(A) Color map image overlay of quantitative fluorescence (qFI) during ALA-induced PpIX human glioma resection [51]. (B) Intensity image of folate conjugated to fluoresceinisothiocyanate (FITC), pseudocolored and overlaid onto an RGB image during ovarian cancer resection [52], (C) Pseudocolored ICG overlay during breast cancer lymph node resection [53] (D) Sentinal lymph node mapping of non-small cell lung cancer metastesis which has been pseudocolored and overlaid onto an RGB image intraoperatively [46].

Fig. 6

The four transparency functions or look-up tables widely used in creating overlays.

Fig. 7

The relationship between normal distributions of measured parameter values in normal and tumor regions, and the true positive rate (TPR) if that value was selected as the clinical threshold for diagnosis. The resulting TPR logistic function for (A) scarcely overlapping and (B) greatly overlapping distributions are shown.

Fig. 8

The same fluorescence map overlaid using a uniform colormap but with different transparency functions. (A) Logistic function with x = 5 μg/ml (B) logistic function with x = 10 μg/ml, (D) linear function intersecting point (x = 14 μg/ml, y = 50%), (C) logistic function with x = 15 μg/ml. How the transparency function is defined will have large effects on the perceived margin of malignant tissue, highlighting the need for standardization.

Fig. 9

Lymphatic uptake of fluorophore in a mouse is shown as the green overlay on grayscale white-light images. The leftmost column shows the original image and each subsequent column shows a processed image. Window-level adjustment with contrast-limited adaptive histogram equalization (CLAHE) was applied to the original images for comparison with log-compressed images. Well plates with various concentrations of fluorophore ranging over 3 orders of magnitude in concentration and fluorescence intensity are shown, and histograms corresponding to lymphatic uptake images. White arrows indicate a lymph vessel that is hard to detect at early time points without log compression. 5mm scale bars are shown.

Fig. 10

(A) An early prototype of a HUD unit that was integrated into a clinical operating microscope allowing display of the augmented information into the surgeons microscope view. (B) An example contour representation of the data in Fig. 8 at threshold = 70%, showing the outline of the region for the surgeon. (C) A density point-cloud representation of the same data.

Fig. 11

(A) Screenshot of the main window of Overlay GUI. A number of different sliders, radio-buttons and drop-down menus enable the user to quickly make fully customized color overlays, selecting from 18 different color maps and the four different transparency functions discussed in Section 4. (B) The normalized scalar magnitude vs. red, green and blue values as well as the lightness (L*) are shown for the present colormap (koufonisi) along with the Pyramid Test for lightness uniformity [28]. (C) The 1931 CIE x,y chromaticity plot showing the gamut for the current bottom image (contour plot overlay) and the trajectory of the current color map through the color space.