RADIOGAN: DEEP CONVOLUTIONAL CONDITIONAL GENERATIVE ADVERSARIALNETWORK TO GENERATE PET IMAGES

A. Amyar 1, 2, S. Ruan 2 , P. Vera 2, 3, P. Decazes 2, 3, R. Modzelewski 2, 3

1 General Electric Healthcare, Buc, France2 LITIS - EA4108 - Quantif, University of Rouen, Rouen, France

3 Nuclear Medicine Department, Henri Becquerel Center, Rouen, France

ABSTRACT

One of the most challenges in medical imaging is the lackof data. It is proven that classical data augmentation methodsare useful but still limited due to the huge variation in images.Using generative adversarial networks (GAN) is a promisingway to address this problem, however, it is challenging totrain one model to generate different classes of lesions. Inthis paper, we propose a deep convolutional conditional gen-erative adversarial network to generate MIP positron emissiontomography image (PET) which is a 2D image that representsa 3D volume for fast interpretation, according to different le-sions or non lesion (normal). The advantage of our proposedmethod consists of one model that is capable of generatingdifferent classes of lesions trained on a small sample size foreach class of lesion, and showing a very promising results. Inaddition, we show that a walk through a latent space can beused as a tool to evaluate the images generated.

Index Terms— Machine Learning, Deep learning, Gen-erative adversarial Networks, Positron Emission Tomography

1. INTRODUCTION

A generative adversarial network (GAN) is a class of machinelearning system invented by Ian Goodfellow in 2014 [1]. Twoneural networks a discriminator and a generator compete witheach other in a mini-max game. Given a training set, the gen-erator tends to learn how to produce new data with the samestatistics as the training set, while the discriminator tries todistinguish between real data and generated one as shown inFig. 1. Generative methods (in particular, GANs) are cur-rently used in various places for data augmentation [2]. Theirpotential is vast; they can learn to mimic any distribution ofdata across any domain: photographs, drawings, music, andprose. However, theres no ground truth data to predict, whichmake it difficult to measure the model performance, espe-cially in the medical field where the images may contain dif-ferent lesions. In addition, it is not evident to distinguish amemorizing GAN, which memorize training examples froma generating one that learned meaningful boundary between

Fig. 1. Generative Adversarial Network Framework.

real and generated images [3]. Walking through a latent spacecan be used as a tool that can evaluate memorization degree.

Several authors have used GANs to build models for med-ical imaging synthesis, such as generating computed tomog-raphy (CT) from magnetic resonance imaging (MRI) [4] orlearning with GANs for chest X-ray classification [5]. In PETimages, Ben-Cohen et al. [6] proposed a framework to gen-erate PET images from CT data using a deep convolutionalnetwork. However, to our knowledge, there was no studyabout the generation of different class of lesions with the samemodel, and neither the verification if the model able to gen-erate a complete new data or its memorizing the images fromthe training set.

To perform fast lesion classification physicians usuallyuse MIP PET images which are 2D images that represents the3D images. In this paper, we focus on the generation of MIPPET images. Deep learning methods trained to perform sucha task are data hungry and need a lot of data to accomplishthis task due the variability in images with different lesions.GANs are a promising tool that can solve this problem by in-creasing the size of data. However, we are faced with twomain challenges when training a GAN: first, each class of le-sions presents a small dataset in its own when used separately.Second, it is usually hard to verify if the GAN is generaliz-ing a complete new data or it is just memorizing the dataset.Moreover, medical imaging presents a unique class of diffi-culties due to the huge variability in images, the presence of

arX

iv:2

003.

0866

3v1

[ee

ss.I

V]

19

Mar

202

0

Fig. 2. Examples of PET-MIP input images used to train GAN. In the upper line from left to right: A: head & neck cancer,B: oesophagus cancer, C: lung cancer, D: lymphoma and E: normal patient. In the second line, showing multiple cases for thesame category (normal).

lesions in a specific anatomical point, the lack of integrationof medical information and other constraints.

In this work, we develop a new GAN architecture, calledRadioGAN, to generate FDG-PET images of normal patientsand lesions. The idea is to combine different classes of le-sions to train a condtional model which can capture not onlycommon features in PET images but also specific ones foreach lesion. We address both issues of data generation andclass specified data augmentation. Moreover, we propose aframework to evaluate the result of a GAN and to verify ifit is generating a complete new images or if it is memoriz-ing the images seen in training by using a walk through alatent space [7]. If a GAN memorizes images, then choosingrandom seeds in latent space creates basic blends of trainingimages. However, if a GAN generalizes good images, thenchoosing random seeds produces exciting images that utilizepatterns and components from the training dataset but are notsimple blend.

2. DATA

We have one thousand six hundred and six patients (1606) inwhich six hundred and seventy five (675) are normal (onlyphysiological uptakes) and nine hundred and thirty one (931)with pathological uptakes (cancer). We have four classes ofcancer: lymphoma (225), lung cancer (189), oesophagus can-cer (97) and head and neck cancer (422). Patient informationwas anonymized prior for analysis. The voxel size of PETexams is 4.06 × 4.06 × 2.0 mm3 with a size of 429 x 168 x168 before preprocessing. The lung cancer and head & neckdatasets are from the cancer imaging archive (TCIA) [8]. Theimages were acquired at the Henri Becquerel Center Institu-tion and the study was approved as a retrospective study.

3. METHOD

The object is to generate a specific class of image from onemodel trained with multiple classes (conditions).

3.1. Data preprocessing

Images were spatially normalized by resampling all thedataset to an isotropic resolution of 2 × 2 × 2 mm3 usingthe k-nearest neighbor interpolation algorithm. PET imagesgray level intensity were normalized to absolute Standard-ized Uptake Value (SUV) level between [0 30] and translatedbetween [0 1] to be used in GAN architectures. Maximumintensity Projection (MIP) transformation was used to com-press a 3D exam into a 2D representation. Five classes:normal, head & neck cancer, oesophagus cancer, lung cancerand lymphoma are shown in Figure 2.

3.2. Proposed RadioGAN architecture

RadioGAN is inspired from deep convolutional generativeadversarial network (DCGAN) which is a class of convolu-tional neural networks that generate images in an unsuper-vised learning way [7] and conditional generative adversarialnetwork (CGAN) which is a conditional version of GAN con-structed by feeding the GAN the data and the label to condi-tion both the generator and the discriminator [9].

The design of a GAN can be a challenging part, espe-cially the generator design. The Generator takes as input arandom noise and then map it into an image such that the dis-criminator cannot tell which images came from the dataset(real) and which ones came from the generator (fake).Severalstudies showed that using Convolutional Layers in the gener-ator network produces better results [7]. To transform a GAN

Fig. 3. RadioGANarchitecture

to a DCGAN we follow the steps as in [7]: replace all maxpooling with convolutional stride, use transposed convolutionfor upsampling, eliminate fully connected layers, use of batchnormalization except for the output layer for the generator andthe input layer of the discriminator, use ReLU in the genera-tor except for the output which uses tanh and LeakyReLU inthe discriminator.

Conditional GAN uses image class information as a con-dition. For example, when we give our Generator a randomvector Z, at the same time, we also give the class label Y=[Normal, Esophagus cancer, head & neck cancer, lung can-cer, lymphoma]. Then the Discriminator will evaluate howwell the Generator draws a patient of the corresponding classLabel. After that, we use the evaluated error to update theGenerator’s weights (which includes new weights to accom-modate the input Y).

In RadioGAN we take advantage of both DCGAN andCGAN to create a novel architecture DCCGAN (Deep Con-volutional Conditional Generative adversarial Network) togenerate MIP PET images for different classes of cancer andalso for normal patient. The class label is added as an ad-ditional input layer for both generator and discriminator. Tospecify the class of the lesion for the generator, we encodedit as a condition using an embedding layer, which is a densevector of a fixed size of 50. The five classes of images: lungcancer, esophagus cancer, head & neck cancer, lymphomaand normal are mapped to a different 50 element vector rep-resentation. Then, the output is followed by a Dense layerwith a linear activation. The Dense layer has 15 neuronsto match the 5 x 3 feature map activation of the generatormodel, and added as a second channel. For the discriminator,a new Embedding layer of a size of 50 is added, followed bya Dense layer to scale up to image dimension with a linearactivation and added to the input image as a second channelas shown in Fig. 3.

The discriminator is an eight layers deep convolutional

Fig. 4. An example of a memorizing GAN. The first seed isfor a Normal patient and the last one for head & neck cancer.The GAN model is blending images between the two seeds togo from a normal patient to head & neck cancer. The imageframed in red is not realistic.

neural network in which each layer is composed of a convo-lution with a filter of 3 X 3 and a stride of 2 followed by aLeakyReLU and a Dropout with no pooling, and a BatchNor-malization with a momentum of 0.8 except for the first and thetwo last layers. The last layer is a Dense layer with a Sigmoidfunction to classify the image for real or fake. The generatortakes a random noise Z and passed to a Dense layer, followedby a Reshape and a BatchNormalization of a momentum of0.8. Then a series of transposed convolution followed by aBatchnormalization are applied except for the last layer whichis a convolutional layer with one channel and a tanh activationfunction. Then, the adversarial network is extended to a con-ditional model by conditioning both the discriminator and thegenerator on the lesions, normal labels.

We have trained our model for 300 epochs and used thebinary crossentropy loss and Adam optimizer with a learningrate of 0.0002 and beta1 of 0.5. Accuracy is used as the metricfor the discriminator.

4. RESULTS

4.1. Evaluation of the memorization vs generalisation

Once the model is obtained, we can input a random vector(seed) of length 100 into the Generator and we get a patientimage. If we input seed Z1 = [1, 0, 0...0] for example, wemay get a patient with oesophagus cancer image and if weinput seed Z10 = [10, 0, 0...0] we may obtain a normal pa-tient image. What happens when we input the seeds betweenZ1 and Z10 ? For example Z2 = [2, 0, 0...0], Z3 = [3, 0,

Fig. 5. An example of images generated with RadioGAN. The model is moving slowly from lung cancer (A) to lymphoma (J)while generating a sequence of realistic patient images with coherence..

0...0], , Z9 = [9, 0, 0...0] ? A high quality generator mustalways output realistic images, so we should not see a halfpatient, a patient with 3 legs or a head instead of the stom-ach. Instead, we should see a sequence of realistic patientimages that slowly transform from oesophagus cancer to nor-mal. Latent walks can help us to distinguish simple memo-rizing GANs from complex generalizing GANs. In Fig. 4 anexample of images generated with a standard CGAN whichtends to memorize, we can see the image framed in red isnot realistic. While in Fig. 5 an example of images gener-ated with our proposed RADIOGAN which tends to generaterealistic new images and not only blending them.

5. DISCUSSION AND CONCLUSION

We have developed an end-to-end convolutional conditionalgenerative adversarial network to generate MIP PET images.The advantage of this strategy is to train one model whichcan capture common features, but also specific ones for eachlesion and for normal class. The integration of class lesionsand normal class information is an interesting and challengingapproach due to the difficulty to evaluate a GAN model. Wehave approached this problem through a walk in the latentspace to evaluate the quality of the images generated.

Generative adversarial networks are a very promising toolto tackle the lack of data in the medical imaging field, and itcan be also used to provide an advantage for other problemssuch as domain adaptation where data come from differentdistribution. RadioGAN shows promising results and can beextended to whole 3D body images to generate PET imagesfrom PET for data augmentation or from another modalitysuch as CT for data synthesis. These results can be confirmedon a larger database.

6. ACKNOWLEDGMENTS

The authors acknowledge financial support from the Can-ceropole Nord-Ouest for emerging projects(https://www.canceropole-nordouest.org/).

7. REFERENCES

[1] I Goodfellow et al., “Generative adversarial nets,” in Advances in neuralinformation processing systems, 2014, pp. 2672–2680.

[2] L Perez et al., “The effectiveness of data augmentation in image classi-fication using deep learning,” arXiv preprint arXiv:1712.04621, 2017.

[3] A Brock et al., “Large scale gan training for high fidelity natural imagesynthesis,” arXiv preprint arXiv:1809.11096, 2018.

[4] D Nie et al., “Medical image synthesis with context-aware genera-tive adversarial networks,” in International Conference on Medical Im-age Computing and Computer-Assisted Intervention. Springer, 2017, pp.417–425.

[5] A Madani et al;, “Semi-supervised learning with generative adversar-ial networks for chest x-ray classification with ability of data domainadaptation,” in 2018 IEEE 15th International Symposium on BiomedicalImaging (ISBI 2018). IEEE, 2018, pp. 1038–1042.

[6] A Ben-Cohen et al., “Virtual pet images from ct data using deep convo-lutional networks: initial results,” in International Workshop on Simula-tion and Synthesis in Medical Imaging. Springer, 2017, pp. 49–57.

[7] A Radford et al., “Unsupervised representation learning withdeep convolutional generative adversarial networks,” arXiv preprintarXiv:1511.06434, 2015.

[8] K Clark et al., “The cancer imaging archive (tcia): maintaining andoperating a public information repository,” Journal of digital imaging,vol. 26, no. 6, pp. 1045–1057, 2013.

[9] M Mirza et al., “Conditional generative adversarial nets,” arXiv preprintarXiv:1411.1784, 2014.