Utility of deep learning networks for the generation of artificial cardiac magnetic resonance images in congenital heart disease

Abstract

Background: Deep learning algorithms are increasingly used for automatic medical imaging analysis and cardiac chamber segmentation. Especially in congenital heart disease, obtaining a sufficient number of training images and data anonymity issues remain of concern.

Methods: Progressive generative adversarial networks (PG-GAN) were trained on cardiac magnetic resonance imaging (MRI) frames from a nationwide prospective study to generate synthetic MRI frames. These synthetic frames were subsequently used to train segmentation networks (U-Net) and the quality of the synthetic training images, as well as the performance of the segmentation network was compared to U-Net-based solutions trained entirely on patient data.

Results: Cardiac MRI data from 303 patients with Tetralogy of Fallot were used for PG-GAN training. Using this model, we generated 100,000 synthetic images with a resolution of 256 × 256 pixels in 4-chamber and 2-chamber views. All synthetic samples were classified as anatomically plausible by human observers. The segmentation performance of the U-Net trained on data from 42 separate patients was statistically significantly better compared to the PG-GAN based training in an external dataset of 50 patients, however, the actual difference in segmentation quality was negligible (< 1% in absolute terms for all models).

Conclusion: We demonstrate the utility of PG-GANs for generating large amounts of realistically looking cardiac MRI images even in rare cardiac conditions. The generated images are not subject to data anonymity and privacy concerns and can be shared freely between institutions. Training supervised deep learning segmentation networks on this synthetic data yielded similar results compared to direct training on original patient data.

Trial registration: ClinicalTrials.gov NCT00266188.

Conflict of interest statement

Not applicable.

Figures

Fig. 1

Study overview illustrating the use of original cardiac magnetic resonance (CMR) images for generation of synthetic short axis (SAX) and long axis (LAX) images using a progressive generative adversarial network (PG GAN). The resulting images were subjected to visual inspection by CMR experts and general cardiologists. In addition, deep learning segmentation networks (with U-Net design) were built based, both, on PG GAN and actual CMR frames. The accuracy of the resulting segmentation networks was finally compared on a separate data set not used for training of either network

Fig. 2

Illustration of the network design of the U-Net segmentation network. The network accepts a greyscale frame (128 × 128 pixels) and produces segmentation maps of equal size for the heart chambers involved. The network consists of a contracting path with multiple 3 × 3 convolutions followed by ReLU (Rectified Linear Unit) activation and a max. Pooling operation (2 × 2). The number of channels is doubled at each step of the contraction path. In the expanding part, the feature maps are upscaled symmetrically, with 2 × 2 up-convolutions. In addition, channels of the expanding path are combined with the corresponding part of the contracting path through concatenation. The number on top corresponds to the number of channels, while the dimensions are given on the left of the respective boxes. For details see Ref. [13]

Fig. 3

Overview over the training of the progressive adversarial network (PG GAN) using increasing image resolution of 4 × 4, 8 × 8, 16 × 16, 32 × 32, 64 × 64 and 128 × 128 pixels. Finally, a maximal resolution of 256 × 256 pixels is achieved (right panel)

Fig. 4

Comparison of synthetic cardiac magnetic resonance (CMR) images (top row) produced by the progressive generative adversarial network (GAN) with actual CMR images from patients with tetralogy of Fallot with the highest degree of statistical similarity (Wasserstein distance; for details see Method section)