Energy-based Generative Adversarial Network

“We impose the following restrictions on EBGAN models: (i)-using learning rate 0.001 and Adam (Kingma & Ba, 2014) for both G and D ; (ii)- nLayerD represents the total number of layers combining Enc and Dec .”

“Finally, we trained EBGANs to generate high-resolution images on ImageNet (Russakovsky et al., 2015).”

“For training both EBGANs and GANs for the grid search, we use the following setting: • Batch normalization (Ioffe & Szegedy, 2015) is applied after each weight layer, except for the generator output layer and the discriminator input layer (Radford et al., 2015).”

“Generative Adversarial Networks (GAN) (Goodfellow et al., 2014) have led to significant improvements in image generation (Denton et al., 2015; Radford et al., 2015; Im et al., 2016; Salimans et al., 2016), video prediction (Mathieu et al., 2015) and a number of other domains.”

“(2010) and Rifai et al. (2011). In quest for better techniques for stabilizing GAN training, several papers was presented with helpful techniques in Salimans et al.”

“Generative Adversarial Networks (GAN) (Goodfellow et al., 2014) have led to significant improvements in image generation (Denton et al., 2015; Radford et al., 2015; Im et al., 2016; Salimans et al., 2016), video prediction (Mathieu et al., 2015) and a number of other domains.”

“We also trained a network to generate images of size 256 × 256 , on a dog-breed subset of ImageNet, using the wordNet IDs provided by Vinyals et al. (2016).”

“By adding regularizers, several techniques were addressed to resolve such problem (Vincent et al., 2010; Rifai et al., 2011; Ng; MarcAurelio Ranzato & Chopra, 2007; Kavukcuoglu et al., 2010).”

“To access the capacity of generating 64× 64 RGB images on the basis of using deep convolutional models, we have trained EBGANs on the LSUN bedroom dataset (Yu et al., 2015) and the large scale face dataset CelebA under alignment (Liu et al.”

“, 2014) have led to significant improvements in image generation (Denton et al., 2015; Radford et al., 2015; Im et al., 2016; Salimans et al., 2016), video prediction (Mathieu et al., 2015) and a number of other domains. The basic idea of GAN is to simultaneously train a discriminator and a generator. The discriminato…”

“To model the discriminator, D(x), as an energy function that assigns low energies to regions close to the data manifold and greater energies to other regions, a family of GANs called EBGANs has been proposed [31].”

“adversarial network [24], wasserstein generative adversarial networks [18], least squares generative adversarial networks [25] and coupled generative adversarial networks [26].”

“Therefore, we re-implement the six alternative architectures: U-Net [2], HED [4], FastImgProc [9], Deep En-Decoder [5] and the networks based on generative adversarial networks (GANs), Pixel2Pixel [10] and EBGAN [11].”

“[8] presented an energy-based GAN that offers optimal discriminator learning by minimizing an energy function where the energy is computed by an auto-encoder structured discriminator.”

“works (GANs) [7, 21, 33, 4, 20] have drawn substantial attention from both the deep learning and computer vision society.”

“t is particularly tting to our proposal, namely InfoGAN [8]. InfoGAN is an extension of the GAN (Generative Adversarial Networks) framework [17] which has been applied to a variety of problems (e.g., [12,18,21,22,23]). We rst describe the original GAN framework before moving on to InfoGAN. The GAN framework (depicted…”

“The energy-based GAN (EBGAN) [17] and boundary equilibrium GAN (BE-GAN) [18] employ a pre-trained auto-encoder as the discriminator, learning to distinguish between real and generated samples.”

“The energy-based GAN (EBGAN) is a comparatively novel type of GAN was proposed by Zhao et al. 28 .”

“Example of IPM based GAN would be WGAN [2], RGAN [22], fisherGAN [43] etc. while non IPM based GAN would be fGAN [46], UGAN [41], EBGAN [76].”

“However, GAN has many variants, such as training generative adversarial network using variational divergence minimization (f-GAN) (Nowozin et al., 2016) and the energy-based generative adversarial network model (EBGAN) (Zhao et al., 2017), thus a natural question arises: what is the effect of deploying a different adv…”

“(Junbo, Michael, and Yann 2016) developed an Energy-based GAN or EBGAN in which regions near a data manifold have low energy and other regions have higher energy.”

“GANs [12] have achieved great success in many fields, e.g. , image inpainting [21], image generation [23, 37], image-to-image translation [3, 18, 22, 28, 46].”

“To further verify the effectiveness of the proposed method, we compare it with widely-used learning objectives, i.e. Std-GAN, WGAN, HingeGAN, LSGAN, DCGAN, WGANGP and Realness GAN.”

“Analogous to energy based GAN(EGAN) [22], we have used two loss functions in our proposed methodology: • L T n loss function to train the translation network T n .”

“Besides, GAN is widely used in the field of image generation tasks [1, 22, 30].”

“A boundary equilibrium generative adversarial network (BEGAN) [15] was inspired by WGAN and EBGAN, and it produced high-quality data.”

“Examples include the Wasserstein GAN, Energy Based GANs, and Boundary Equilibrium GAN [Arjovsky et al., 2017, Zhao et al., 2016, Berthelot et al., 2017].”

“With the encoder-decoder architecture [6], the generator is achieved; Infrared images are generated by decoding latent variables from encoder.”

“in [45] called Energy-based GANs.”

“[5] and [6] proposed using least square and absolute deviation as the measure.”

“…the classic generative models to avoid missing modes, especially in the framework of GANs, by altering objective functions (Che et al., 2016; Zhao et al., 2016; Mao et al., 2017; Arjovsky et al., 2017; Gulrajani et al., 2017; Saatci & Wilson, 2017), changing training methods (Heusel et al.,…”

“StackGAN++ [12] takes it one step further where multiple discriminators and multiple generators 2018 First International Conference on Secure Cyber Computing and Communication (ICSCCC)”

“In addition to (Zhao, Mathieu, and LeCun 2016) which view the discriminator as an energy function that attributes low energies to the regions near the data manifold and higher energies to other regions.”

“The structure of EBGAN is shown in Fig.”

“Some works focus on the architecture of original GAN for better performances [31, 4, 20, 2].”

“Özellikle 2014 yılında geliştirilen Çekişmeli Üretici Ağlar-ÇÜA (Generative Adversarial Network) [6] popülarite kazanmaya başlamış ve günümüze kadar çeşitli varyasyonları geliştirilmiştir [7], [8].”

“Per-pixel classification or regression issues are frequently used to solve image-to-image translation challenges [48]–[62].”

“With the rapid development of computer vision [1, 2, 3, 4, 5, 2, 3, 6, 7], generative models [8, 9, 10, 11, 12, 13, 14, 15] have emerged as a foundational technical framework that bridging data understanding, content creation, and interactive applications.”

“When the discriminator is viewed as an energy function, a variety of architectures and loss functions may be used in addition to the traditional binary classifier with logistic output [92].”

“Transfer learning (TL) is another effective approach that applies previously acquired knowledge and skills to new learning or problem-solving tasks [35,36].”

“To solve this issue, EBGAN [23] and other models were proposed.”

“Early studies on generating face images mostly were capable of proposing models that generate high-quality, low-resolution images [50][60][8], where recent studies propose more realistic face images that are high-quality, large-resolution images, even up to 1,024 × 1,024.”

“The dual codecs are implemented in two subnetworks [15].”

“image to image task, which aims to transform the input image from one domain to another [11], [12], [13], [14], [15], [16], [17].”

“In EBGAN (Zhao et al., 2017a), the discriminator is interpreted as an energy functional and is also cast in an auto-encoder framework, leading to improvements in semi-supervised learning tasks.”

“To achieve this, the paper proposed to increase the diversity of generated samples by increasing generator entropy, via introducing a new loss term pull-away term (PT) first proposed in [31]:”

“GANs enable a wide variety of application such as image generation [22, 23], image editing [24] and representation learning [25, 26].”

“In last few years, many variations have been invented to improve the original GAN, such as f-GAN [15], Energy based GAN [20], info GAN [17], and WGAN [21], which proves that GAN is a promising model.”

“Many recent works, e.g., [5,15], have been dedicated to making progress toward stable training of GANs.”

“Based on the improved GAN model, an auto-encoder, first proposed in EBGAN [18], is used as a discriminator in this paper.”

“Since its introduction, the use of GAN [12], [46] has seen”

“Based on J witness points in the input space X , we propose to learn the generator G θ by solving min θ max { v j } Jj =1 L λ ( θ , { v j } Jj =1 ) where the loss function is defined as Here, D ( P X , Q Y ) is an arbitrary GAN objective and λ > 0 is a trade-off parameter.”

“Many approaches have been proposed in the literature to solve standard GAN problems, either using different network architectures [4], [23], [24], [25] or objective functions [26], [27], [28].”

“A flurry of recent work has proposed improvements over the original GAN work for image generation [4-9].”

“Since the input and output of PSceneGAN both contain images, and the input image is a part of the output image, the structure of the encoder and decoder cannot only pass a lot of information, however also extract the characteristics of the image [17], [29].”

“Contrary to traditional GANs, which try to match the real data distributions directly, the boundary equilibrium generative adversarial network (BEGAN) aims to match the distributions of the autoencoder loss with a loss derived from the Wasserstein distance proposed in [14] and uses an autoencoder as a discriminator wh…”

“[26a, 28] Next, we replace the algorithmic RNG with our hardware-based RNGs during inference for image generation.”

“Furthermore, an autoencoder is implemented as a discriminator in which the image could be passed as real if the reconstruction error is low [3, 63].”

“From the perspective of energy-based generative models [51,53], this conditional gradient can be interpreted as deriving from an energy function E ( x t , t, y ) , which encodes the discrepancy between the current state of x t and the conditioning input y .”

“One common issue in early works is that the objective function based on the probability distances (e.g., JS divergence [1] and total variation [74]) suffer from discontinuity.”

“There are a lot of GAN models, such as Vanilla GAN, BIGAN, CGAN [8], InfoGAN [9], CycleGAN [10], EBGAN [11], and LSGAN [12].”

“Improve quality LapGAN [32] cGAN + Laplacian pyramid Better photo-realistic images EBGAN [174] Auto-Encoder + GAN Better convergence pattern and scalability to generate high-res image BEGAN [13] Energy based, MCMC sampler Improve distribution modeling and sample quality Bayesian GAN [114]…”

“Generative adversarial networks (GANs) [6, 7] are powerful generative models tenerativhat have achieved impressive results in extensive computer vision tasks, such as image restoration [8, 9], image generation [10, 11], image editing [12], representation learning [11, 13], image-to-image…”

“[11] treat the discriminator as an energy function that attributes lower energies to the regions near the data manifold and higher energies to other regions.”

“EBGAN [26] is a GAN explained from an energy perspective, which treats the discriminator as an energy function.”

“1 INTRODUCTION Recent years have witnessed the development of generative adversarial networks (GANs) [5, 10, 14, 21, 22, 24, 35, 39, 42, 48, 50], which learn a function to generate high-dimension images from randomly sampled noises.”

“Moreover, there are a massive training data x received from p data , and the objective of the G is to approximate p data while using p g [50], [51], [52], [53], [54], [55], [56], [57], [58], [59], as well as the intuition behind them.”

“Generative models have found remarkable applications in image analysis [13–16], and a well-trained generative model closely captures the notion of a signal (or image) being ‘natural’ [17].”

“Representative CNN structures include FCN[5], ResNet[6], Faster RCNN[7], SSD[8], GAN[9], etc. FCN is a more suitable neural network structure for the semantic segmentation of medical images.”

“A simple solution to this requirement is to implement a repelling regularizer [12] which maintains production of samples that are not clustered in one or only few models of the batch data distribution p data .”

“2017], where conditional image generation is also studied while some recent techniques focus on stabilizing the training procedure [Brock et al. 2019; Chen and Koltun 2017; Dosovitskiy and Brox 2016; Karras et al. 2018; Mescheder et al. 2018, 2017; Salimans et al. 2016; Wei et al. 2018; Zhao et al. 2017].”

“Synthetic data is generated using the Pix2Pix based on the Generative Adversarial Networks (GANs) [6-9].”

“EBGAN [18] and BEGAN [19] models use an energy-based function to calculate the loss, resulting in stable convergence of the GAN.”

“…alternative formulations as well as improving understanding of their underlying mathematical foundations (Li et al., 2015; Dziugaite et al., 2015; Zhao et al., 2016; Nowozin et al., 2016; Roth et al., 2017; Arjovsky et al., 2017; Gulrajani et al., 2017; Roth et al., 2018; Mescheder et al., 2018).”

“Energy-based generative adversarial networks [9] proposed an idea called energy-based, which treats discriminator as an energy function.”

“The hinge loss function is also used in the energy-based GAN [17].”

“introduced energy-based GAN (EBGAN) using auto-encoder in discriminator [8].”

“Various GAN architectures were proposed during the last few years to solve the mentioned issues [10, 21-24].”

“Energy-Based Generative Adversarial Networks (EBGAN) [51]uses the discriminator as an energy function that interprets energy magnitude as per the data manifold.”

“One typically known problem with GANs is training instability; however, recent studies provided improvement in multiple aspects [16, 66, 74, 98, 4, 5, 55, 22, 36, 86, 58, 56, 91, 11, 14, 37].”

“Of special interest to us are energy-function based adversaries [18] and Wasserstein-distance based adversaries [17].”

“Recently, several studies have focused on improving the stability of the original GANs, mostly by adopting a heuristic approach [2,5], which is extremely sensitive to modifications and variants of the data and hard to apply to new domains [9].”

“3) Energy-Based GAN: The Energy Based GAN (EBGAN) [5] is an implementation in which the discriminator uses a function to assign energy values to inputs received.”

“Despite their success, GANs have known training instabilities [ 11 ], and many recent works address this issue by either modifying the objective function, the network architecture or training dynamics [ 1 , 2 , 4 , 8 , 11 , 15 , 18 , 20 , 22 , 25 , 28 , 30 , 39 ].”

“Since Generative Adversarial Network (GAN) was presented by Goodfellow in 2014[9], GAN is proved to be very efficient in image generation tasks in many research fields[31][30][18][20].”