Energy-based Generative Adversarial Network

“We impose the following restrictions on EBGAN models: (i)-using learning rate 0.001 and Adam (Kingma & Ba, 2014) for both G and D ; (ii)- nLayerD represents the total number of layers combining Enc and Dec .”

“Finally, we trained EBGANs to generate high-resolution images on ImageNet (Russakovsky et al., 2015).”

“For training both EBGANs and GANs for the grid search, we use the following setting: • Batch normalization (Ioffe & Szegedy, 2015) is applied after each weight layer, except for the generator output layer and the discriminator input layer (Radford et al., 2015).”

“Generative Adversarial Networks (GAN) (Goodfellow et al., 2014) have led to significant improvements in image generation (Denton et al., 2015; Radford et al., 2015; Im et al., 2016; Salimans et al., 2016), video prediction (Mathieu et al., 2015) and a number of other domains.”

“(2010) and Rifai et al. (2011). In quest for better techniques for stabilizing GAN training, several papers was presented with helpful techniques in Salimans et al.”

“Generative Adversarial Networks (GAN) (Goodfellow et al., 2014) have led to significant improvements in image generation (Denton et al., 2015; Radford et al., 2015; Im et al., 2016; Salimans et al., 2016), video prediction (Mathieu et al., 2015) and a number of other domains.”

“We also trained a network to generate images of size 256 × 256 , on a dog-breed subset of ImageNet, using the wordNet IDs provided by Vinyals et al. (2016).”

“By adding regularizers, several techniques were addressed to resolve such problem (Vincent et al., 2010; Rifai et al., 2011; Ng; MarcAurelio Ranzato & Chopra, 2007; Kavukcuoglu et al., 2010).”

“To access the capacity of generating 64× 64 RGB images on the basis of using deep convolutional models, we have trained EBGANs on the LSUN bedroom dataset (Yu et al., 2015) and the large scale face dataset CelebA under alignment (Liu et al.”

“, 2014) have led to significant improvements in image generation (Denton et al., 2015; Radford et al., 2015; Im et al., 2016; Salimans et al., 2016), video prediction (Mathieu et al., 2015) and a number of other domains. The basic idea of GAN is to simultaneously train a discriminator and a generator. The discriminato…”

“BEGAN uses an autoencoder architecture for the discrimi-nator which was first proposed in EBGAN [82] (see Fig.”

“Our method is competitive to DCGAN [150], EBGAN [217], Wasserstein GAN [5], InfoGAN [20], VAE [102], the method of [99].”

“The flexibility of the GAN models has allowed them to be extended to structured prediction [189], [190], training energy based models [191], [192], generating adversarial examples for malware [193], [194], and robust malware detection [45], [195].”

“Table 1 displays the Inception scores of the CoopNets, DCGAN (Radford, Metz, and Chintala 2015), EBGAN (Zhao, Mathieu, and LeCun 2016), Wasserstein GAN (Arjovsky, Chintala, and Bottou 2017), InfoGAN (Chen et al. 2016), VAE (Kingma and Welling 2014), DDGM (Kim and Bengio 2016), and separate training by Algorithm G and…”

“The energy function is trained to push down on the energy of real vectors x , and to push up on the energy of generated vectors G(z) (Zhao et al. 2016).”

“Some works improve the architecture of original GAN for better performances [6, 8, 33, 47].”

“Generative models such as Energy-Based Models [18, 50, 92], Variational Autoencoders (VAE) [45–47], Generative Adversarial Networks (GAN) [16, 27–29, 41, 87], and normalizing flows [48, 62, 88] have shown significant potential in creating realistic images.”

“…explore multiple directions for stabilizing GANs; either by adding regularization terms (Arjovsky et al., 2017; Li et al., 2015; Che et al., 2017; Zhao et al., 2017), a myriad of training “hacks” (Salimans et al., 2016; Gulrajani et al., 2017), re-engineering network architectures (Zhao et al.,…”

“ons of such a process, the generator can generate realistic samples from random noises. While it is introduced recently, GAN has shown many promising results not only for generating realistic samples [1, 2, 3], but also for machine translation [4] and image super-resolution [5]. However, for generating realistic sampl…”

“…our improvement in stability is achieved solely by changing the parameterization of the network, without modifying the loss function (as in Wasserstein GAN [1], LSGAN [12] and BEGAN [3]), training objective (as in improved WGAN [7] and Softmax GAN [10]), or network structure (as in EBGAN [17]).”

“Other classical approach to address training instabilities include energy-based GANs [37] and associative adversarial networks [38].”

“One trend is to optimize the model of [4]–[7] from different aspects, such as information theory [8] and an energy-based model [9].”

“improve the performance of GAN, such as Kullback Leibler divergence [39], least squares [40], Wasserstein distance [41,42], and absolute deviation [43].”

“[17] introduced energy based GAN framework which is more stable to train.”

“(11), e.g., the quadratic loss (Mao et al., 2016) and the hinge loss (Zhao et al., 2016; Lim & Ye, 2017; Miyato et al., 2018).”

“Moreover, many GAN variants have been developed in recent years, such as BiGAN [45], VAE-GAN [46], f-GAN [47], EBGAN [48], unrolled GAN [49], and other models [50]–[52].”

“Several papers have pointed out the connection between EBMs and GANs (Zhao et al., 2016; Finn et al., 2016), which is similar to the Electric/ELECTRA connection.”

“In contrast to typical VAE-GAN approaches (Larsen, Sønderby, and Winther 2015; Zhao et al. 2016; Berthelot et al. 2017), adversarial autoencoders (Makhzani et al. 2015), and adversarial generative encoders (Heljakka, Solin, and Kannala 2018; Ulyanov, Vedaldi, and Lempitsky 2018), we retain the…”

“Recently generative adversarial networks (Goodfellow et al. 2014; Arjovsky, Chintala, and Bottou 2017; Zhao, Mathieu, and LeCun 2017) have achieved significant progress.”

“Instead of optimizing Jensen-Shannon divergence as suggested in the original GAN framework, Energy based GAN [12] and Wasserstein GAN [11] show improvement in the stability by optimizing total variation distance and Earth mover distance respectively, together with regularizing the discriminator to limit the discrimina…”

“Specifically, generation techniques based on GANs were proposed for generating sharper, more diverse and better realistic images in a series of works [5, 20, 26, 28, 32, 40, 44, 53, 60, 64].”

“Other techniques aim to find a faster convergence, modeling the discriminator D ( x ) as an energy function [5] or introducing new loss definitions [1].”

“Energy-Based Models (EBMs) have been widely used in the area of generative modeling [41, 8, 1, 4].”

“Here, the encoder can be perceived as a frozen energy network as used in energy-based GAN [66].”

“Also, an alternative viewpoint for the discriminator function is explored by Zhao et al. (2016) where they deviate from the traditional probabilistic interpretation of the discriminator model.”

“Recent efforts [9, 51, 33, 4, 21] are made to improve the stability of GAN in training.”

“However, the previous approaches are not directly applicable to the general case of deep generative models, as many of the models (e.g., GANs, many auto-regressive networks) are not straightforwardly formulated with the probabilistic Bayesian framework and do not possess a posterior distribution or even meaningful lat…”

“Recent extensions of GAN have focused on boosting the performance of image generation by improving the generator [5], discriminator [30] or the training algorithm [9, 22, 31].”

“Energy-Based GANs [41] and their later improvement BEGANs [4] optimize an energy landscape based on auto-encoders.”

“, image editing [235], feature learning [48, 170], and image generation [44, 163, 232].”

“Therefore, we used pull-away-term [38] to fulfill the condition. where x i , x j are batch samples and m is the batch size.”

“Similar to EBGAN [40], we add the gaussian noise to the shared layers and generator.”

“In medical image analysis where scan sizes are large and data are few, inspired from an energy-based adversarial generation framework (Zhao et al., [11]), it is preferable to employ an adversary providing an anatomically-inspired supervision instead of the usual binary adversarial supervision…”

“…neural network realm has been proposed, such as: Deep AutoEncoders by Baldi in 2012 [18], Inception ConvNet by Szegedy et al. in 2014 [19], VGG-16 by Simonyan and Zisserman in 2014 [20], Enhanced LSTM by Chen et al. in 2016 [21], and Energy-based Generative Adversarial Network by Zhao et al. [22].”

“DCGAN [7], EBGAN [8], WGAN [9], BEGAN [10], and the more recent PGGAN [11] and StyleGAN [12] have basically all made their name as the state-of-the-art generative adversarial networks by showcasing their human face generation results.”

“[27] proposed to view the discriminator as an energy function, thus enables various loss functions other than binary classifier.”

“Lastly, we note that our method can be straightforwardly combined with other objective function extensions [24, 4, 25, 26, 27, 43] and network architectures [11, 44, 9], ones that specifically focus on addressing other challenges such as instability, to leverage their advantages.”

“A number of followup work [2, 6, 16, 26, 28, 41] propose new training procedures and network architectures to improve training stability and convergence rate.”

“Unfortunately, the other variants of GAN either do not accept an input condition (such as DCGAN [38], EBGAN [39], LSGAN [40], WGAN [41], etc.) or they do not achieve a one-to-one mapping between inputs and outputs (such as CycleGAN [42], StyleGAN [43], etc).”

“The generator model can also be learned jointly with a discriminator model, as in the generative adversarial networks (GAN) [19], as well as deep convolutional GAN (DCGAN) [20], energy-based GAN (EB-GAN) [21], Wasserstein GAN (WGAN) [22].”

“Arjovsky suggests Wasserstein distance rather than DCGAN’s KLD and EBGAN (Energy-Based Generative Adversarial Networks)’s TV (Total Variance) distances [15,20,87].”

“designed a defect enhancement GAN (DEGAN) based on deep convolutional GAN (DCGAN) [80] and energy-based GAN (EBGAN) [133] to generate microcrack defective samples.”

“Several alternative loss functions and GAN models have been proposed over the years, claiming superior stability and sample quality over the original GAN (e.g., [16], [25], [3], [2], [25], [11]).”

“One wellknown problem with GANs is training instability; however, the recent studies have been making a great stride in solving this problem [11, 43, 51, 74, 3, 4, 35, 15, 24, 62, 38, 36, 69, 7].”

“To alleviate mode collapse in GANs, some methods propose to improve the minimax loss function [35,2,17,1], some methods apply constraints or regularization terms along with the minimax objectives [8,5,46,32], and some other methods aim to modify the discriminator designs [44,54,36].”

“Our implementation is based on Energy-based generative neural networks introduced in [25] which if trained optimally, can be shown to minimize a measure of divergence between probability measures that directly relates to a theoretical bound shown in this section that underlies our method.”

“Boundary-equilibrium GANs [2] and energybased GANs [58] use an autoencoder as the discriminator to stabilize the training.”

“Another close idea was Energy-Based GAN (EBGAN), which uses an autoencoder as the discriminator (Zhao et al., 2016).”

“PD-GAN[48] GAN Imageinpainting GANforimageinpaintingandimagerestoration CGAN[49] GAN Imagegeneration GANinasupervisedwayforimagegeneration DCGAN[50] GAN Imagegeneration GANinanunsupervisedwayforimagegeneration BiGAN[51] GAN Imagegeneration GANwithencoderinanunsupervisedwayforimagegeneration EBGAN[53] GAN Imagegenerati…”

“Energy-based GAN [8] uses the total variation distance.”

“Other GAN variants in this direction include the EnergyBased GAN (EBGAN) [59], the Least Squares GAN (LSGAN) [60], the Loss-Sensitive GAN (LS-GAN) [61], the Correlational GAN (CorrGAN) [62], and the Mode Regularized GAN (MDGAN) [63], to name but a few.”

“[10], the addition of a “pullaway” regularization term fPT(ξ) to the autoencoder internal representation ξ greatly reduced the preponderance of mode collapse in our tests.”

“Energy-based GAN (EBGAN) [67] introduces an energy function into the discriminator and trains the generator and discriminator by optimizing the energy distance.”

“ADAN finds low-dimensional manifolds using a nonlinear autoencoder, and aligns the empirical distribution of the day-k recordings (the source domain) to those of day-0 (the target domain) by aligning the distributions of residuals (as in (17)) between neural firing rates and their nonlinear autoencoder reconstructions…”

“Other authors have proposed changing generator’s or discriminator’s objectives [3, 12, 15, 26].”

“The method requires no other techniques such as batch normalization[22], or layer-wise noises[9] to help with the training.”

“It can be thought of as a loss function that can be learned in training [43].”

“This enables N → t i to learn a better approximation of the intended cross-task energy manifold, even in absence of negative examples (target samples) [64].”

“The launching point of our work is the well-known Energy-Based Models (EBMs) ([14]).”

“2 can then be written as (conditioned on x for notational brevity): We learn ( ω, θ ) in a GAN-style minimax game: The embedding model behind l ω ( x, y + , y − ) is similar to the discriminator in (conditional) GAN (or critic in Wasserstein (Arjovsky et al., 2017) or Energy-based GAN (Zhao et al., 2016), while g θ (…”

“Similar to recent works such as Zhao et al. (2016); Berthelot et al. (2017), the discriminator D has the structure of an autoencoder.”

“From the perspective of energy-based model [17], autoencoder is an energy function [27, 42] that maps the correct variables to low energies.”

“However, there exist multiple notions of energy-based learning in the machine learning literature, including non-probabilistic interpretations (LeCun et al., 2006; Zhao et al., 2017).”

“This loss is also closely connected with Diffusion-GAN [Wang et al., 2022b] and EB-GAN [Zhao et al., 2016], where the discriminator loss is considered as: In our model, the process of adding noise, α t a θ + σ t ϵ , functions as an encoder, and µ ϕ ( ·| s ) acts as a decoder.”

“Gradient based training of energy networks (LeCun et al., 2006; Zhao et al., 2016; Du and Mordatch, 2019) has been successful at training models for inducing distributions over continuous domains but are not suitable for training discrete sequence models for text.”

“There are many methods [39, 24, 13, 3] recently proposed to stabilize the training procedure of GANs and mitigating model collapse issues, via using alternative objective functions.”

“GANs [32, 33] are extensively used to generate images where generated image could be a handwritten digit [32], a bedroom image [34] or be conditioned with a text [14].”

“For example, RED [45] estimates the reward using an auto-encoder, which is exactly the way of energy modeling in EBGAN [46].”

“Generative Adversarial Networks (GAN) are first introduced to generate images from random noise [24], and have been widely used in many fields including image synthesis [25]–[27], image editing [28], semantic inpaint-”

“As an alternative, we can choose the following f and g [6, 4]: f(u) = max(0,−1− u), g(u) = −u.”

“Furthermore, perturbing μθ in parameter space draws strong similarities with (Zhao et al., 2017), in which the authors add gaussian noise to the layers of the generator.”

“After that, the coupled generative adversarial network [27], wasserstein generative adversarial networks [17], energy-based generative adversarial networks [28], deep convolutional generative adversarial networks [29], and the least squares generative adversarial network [30] are followed.”

“The difficulty of training GANs has spurred research that includes reformulating the minimax objective [3, 38, 42, 43, 47, 54, 56], devising training heuristics [28, 35, 50, 48], proving the existence of equilibria [4], and conducting local stability analyses [25, 39, 40, 44].”

“le adversaries, one can change GAN objective to directly promote sample diversity. WGAN [3] and MMD GAN [22] proposed to optimize distance measurements to stabilize training. On the other hand, EBGAN [36] and Coulomb GANs [33] reformulated the original GAN problem using an energy-based objective to promote sample vari…”

“When using hinge loss, we can get a non-degenerate discriminator while matching the data distribution (Dai et al., 2017; Zhao et al., 2016).”

“2) Generalization on GANs: To assess the algorithm’s ability to generalize across different GAN models, we employ four well-established GAN networks: DCGAN [20], WGAN-GP [34], EBGAN [44], and BEGAN [45] for experimentation.”

“For the first time in energy-based GAN (EBGAN) [27], an autoencoder was used as the discriminator.”

“such as DCGAN [54], energy-based GAN [79], Wasserstein GAN (WGAN) [2], [61], WGAN-GP [22], BEGAN [4],”

“Only a few works have used total variation as a cost function, such as [85].”

“…sample G ( z ), the discriminator loss LD and the generator loss LG are formally defined by: where {·} + is the hinge loss function, the generator is trained to produce contrastive samples with minimal energies, while the discriminator is trained to assign high energies to generated samples [59].”

“Recently, several variants were proposed [6,7,22,23,24,25,26] to improve the image quality generated by GAN models.”

“Later, variants of GANs have been proposed to increase the resolutions and quality of generated images [44], [45], [46], [47], [48].”

“[22] and Berthelot et al.”

“GANs [25,26] has achieved remarkable results in image generation [27,28], image editing [29] and feature learning [28,30,31].”