Deep learning workflow in radiology: a primer

Abstract

Interest for deep learning in radiology has increased tremendously in the past decade due to the high achievable performance for various computer vision tasks such as detection, segmentation, classification, monitoring, and prediction. This article provides step-by-step practical guidance for conducting a project that involves deep learning in radiology, from defining specifications, to deployment and scaling. Specifically, the objectives of this article are to provide an overview of clinical use cases of deep learning, describe the composition of multi-disciplinary team, and summarize current approaches to patient, data, model, and hardware selection. Key ideas will be illustrated by examples from a prototypical project on imaging of colorectal liver metastasis. This article illustrates the workflow for liver lesion detection, segmentation, classification, monitoring, and prediction of tumor recurrence and patient survival. Challenges are discussed, including ethical considerations, cohorting, data collection, anonymization, and availability of expert annotations. The practical guidance may be adapted to any project that requires automated medical image analysis.

Keywords: Cohorting; Convolutional neural network; Deep learning; Medical imaging; Review article.

Conflict of interest statement

Samuel Kadoury has an industry research grant from Elekta Ltd. and NuVasive inc. The other authors declare that they have no competing interests.

Figures

Fig. 1

Potential clinical uses of deep learning techniques. Tasks such as monitoring of treatment response or prediction of survival, can be derived from lesion detection, classification, and longitudinal follow-up

Fig. 2

Expertise of team members

Fig. 3

Concept of case selection based on clinical indication (left), imaging (middle), or pathology (right) findings

Fig. 4

Types of learning. With supervised learning, the number of inputs (CT images in this example) equals numbers of targets (malignancy status of a lesion here). With semi-supervised, the number of inputs is greater than the number of targets (dataset includes unlabeled samples). With unsupervised learning, none of the inputs are labeled (e.g., clustering, manifold learning, restricted Boltzmann machines). N.A. indicates not available information

Fig. 5

Example of data visualization: projection on first two dimensions of linear discriminant analysis (LDA) applied to radiomics features extracted from various types of lesions [58]

Fig. 6

Division of dataset into training, validation, and test datasets. It is recommended to perform splitting at the very beginning of the workflow, keeping test data unseen to the model until final performance evaluation

Fig. 7

Data sampling strategies. a The whole dataset is split in two distinct subsets for training and testing purposes. Training dataset is subdivided to perform cross-validation, (b) k-fold cross-validation. The training dataset is divided in k subsets of equal size. Training is performed sequentially, considering at each iteration a specific subset as validation set

Fig. 8

Commonly used deep learning architectures. a Fully connected neural networks. b Convolutional neural networks for detection or classification. c U-net, for segmentation. d Legend illustrating the building blocks in various deep learning architectures