Semantic consistency generative adversarial network for cross-modality domain adaptation in ultrasound thyroid nodule classification

Abstract

Deep convolutional networks have been widely used for various medical image processing tasks. However, the performance of existing learning-based networks is still limited due to the lack of large training datasets. When a general deep model is directly deployed to a new dataset with heterogeneous features, the effect of domain shifts is usually ignored, and performance degradation problems occur. In this work, by designing the semantic consistency generative adversarial network (SCGAN), we propose a new multimodal domain adaptation method for medical image diagnosis. SCGAN performs cross-domain collaborative alignment of ultrasound images and domain knowledge. Specifically, we utilize a self-attention mechanism for adversarial learning between dual domains to overcome visual differences across modal data and preserve the domain invariance of the extracted semantic features. In particular, we embed nested metric learning in the semantic information space, thus enhancing the semantic consistency of cross-modal features. Furthermore, the adversarial learning of our network is guided by a discrepancy loss for encouraging the learning of semantic-level content and a regularization term for enhancing network generalization. We evaluate our method on a thyroid ultrasound image dataset for benign and malignant diagnosis of nodules. The experimental results of a comprehensive study show that the accuracy of the SCGAN method for the classification of thyroid nodules reaches 94.30%, and the AUC reaches 97.02%. These results are significantly better than the state-of-the-art methods.

Keywords: Cross-modality domain adaptation; Domain knowledge; Self-attention mechanism; Semantic consistency; Thyroid nodule classification.

Conflict of interest statement

Conflict of InterestsThe authors declare that they have no conflict of interest.

© The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2021.

Figures

Fig. 1

From left to right, the original ultrasound images of the four benign/malignant nodules randomly sampled in the dataset, the corresponding “Ultrasound Findings” in the ultrasonography report, and the domain knowledge are shown. Among them, the red text description is based on the relevant disease keywords selected by TI-RADS as the standard

Fig. 2

Overview of our proposed SCGAN, consisting of a text encoder (top left), a domain adaptation generator Gs→t(bottom left), a discriminator Dt(top right), and a classifier (bottom right). Gs→t has two inputs, Z and Tsenc generated by the text encoder, both of which are implemented in upblocks (gray boxes) for cross-domain fusion. The SACAM contained in upblocks promotes semantic alignment during the fusion process. Similarly, Dt distinguishes the authenticity of an image by a series of downblocks (gray boxes, Is→t represents the synthesized image, It represents the real image). The classifier is the modified classification model ResNet-50. In particular, the adversarial loss refers to the hinge version of the adversarial loss, ℒvd is the visual discrepancy loss, and ℒCD−GP is the cross-domain fusion zero-centered gradient penalty function

Fig. 3

Details of the cross-modal alignment self-attention module (CASAM). The semantic alignment layer can focus on the source domain features corresponding to the target domain pixels. ⊗ denotes dot product operation

Fig. 4

The overall process of image preprocessing. Where the blue dashed line indicates the vertical diameter of the nodule and the green dashed line indicates the horizontal diameter

Fig. 5

ROC analysis of different image generation models with our SCGAN and its variants for thyroid nodule classification

Fig. 6

The loss curve of generator and discriminator in SCGAN

Fig. 7

Inception score (IS) analysis for different parameters of the loss function

Fig. 8

Box and whisker plot analysis of the inception score (IS) for different image generation methods

Fig. 9

Visualization results of 24 images generated using DAGAN, CASAM (i.e., variant SCGAN−ℒvd−ℒCD−GP), and SCGAN