Energy-based Generative Adversarial Network

“We impose the following restrictions on EBGAN models: (i)-using learning rate 0.001 and Adam (Kingma & Ba, 2014) for both G and D ; (ii)- nLayerD represents the total number of layers combining Enc and Dec .”

“Finally, we trained EBGANs to generate high-resolution images on ImageNet (Russakovsky et al., 2015).”

“For training both EBGANs and GANs for the grid search, we use the following setting: • Batch normalization (Ioffe & Szegedy, 2015) is applied after each weight layer, except for the generator output layer and the discriminator input layer (Radford et al., 2015).”

“Generative Adversarial Networks (GAN) (Goodfellow et al., 2014) have led to significant improvements in image generation (Denton et al., 2015; Radford et al., 2015; Im et al., 2016; Salimans et al., 2016), video prediction (Mathieu et al., 2015) and a number of other domains.”

“(2010) and Rifai et al. (2011). In quest for better techniques for stabilizing GAN training, several papers was presented with helpful techniques in Salimans et al.”

“Generative Adversarial Networks (GAN) (Goodfellow et al., 2014) have led to significant improvements in image generation (Denton et al., 2015; Radford et al., 2015; Im et al., 2016; Salimans et al., 2016), video prediction (Mathieu et al., 2015) and a number of other domains.”

“We also trained a network to generate images of size 256 × 256 , on a dog-breed subset of ImageNet, using the wordNet IDs provided by Vinyals et al. (2016).”

“By adding regularizers, several techniques were addressed to resolve such problem (Vincent et al., 2010; Rifai et al., 2011; Ng; MarcAurelio Ranzato & Chopra, 2007; Kavukcuoglu et al., 2010).”

“To access the capacity of generating 64× 64 RGB images on the basis of using deep convolutional models, we have trained EBGANs on the LSUN bedroom dataset (Yu et al., 2015) and the large scale face dataset CelebA under alignment (Liu et al.”

“, 2014) have led to significant improvements in image generation (Denton et al., 2015; Radford et al., 2015; Im et al., 2016; Salimans et al., 2016), video prediction (Mathieu et al., 2015) and a number of other domains. The basic idea of GAN is to simultaneously train a discriminator and a generator. The discriminato…”

“They further motivate a series of variants, which adopt various DNN architectures to learn more complicated generative models (Radford et al., 2015; Chen et al., 2016; Zhao et al., 2016; Dai et al., 2017; Jiang et al., 2018).”

“The flexibility of GANs has also enabled various extensions, for instance to support structured prediction [28, 29], to train energy based models [48] and combine adversarial loss with an information loss [6].”

“Among these methods, we specifically study Generative Adversarial Network (GAN) [7], [44], [2], [43], [19] and Variational Auto-encoder (VAE) [23], [13].”

“GAN performs a forward pass to reconstruct an image, which is generally faster than INNg where synthesis is carried out using backpropagation.”

“opments Based on DCGAN[10] CGAN[52], infoGAN[53], ACGAN[40], SGAN[54] AAE[55], BiGAN [56], ALI[28], AGE[57], VAE-GAN[58] Unrolled GAN[29], f-GAN[59], Mode-Regularized [35], Least-Square GAN[60], EBGAN[61], WGAN[44], WGAN-GP[45], WGAN-LP[46] Fig. 9. Categorization of GAN models. rial N) [ image. The proposed method rea…”

“Previous works have successfully applied GAN framework to various applications such as image generation [4, 22, 39], image editing [27, 40] and feature learning [5, 26].”

“One way to mitigate these problems is to have generator and discriminator of the same type, i.e. autoencoders, as proposed in EBGAN [1] and BEGAN [2] for generating realistic images.”

“3 ADVERSARIAL DOMAIN ADAPTATION NETWORK (ADAN) In addition to matching the probability distributions of latent variables of day-k to those of day-0, we seek an alternative approach: to match the probability distributions of the residuals of the reconstructed firing rates (Zhao et al., 2016), as a proxy for matching th…”

“to model sequential data which are prevalent in the speech and NLP domain. Recently GANs have also been used in the context of topic modeling [5]. The proposed model is based on the Energy-Based GANs [21], but uses a Denoising Autoencoder as the discriminator network. There are several prior works on improving the tra…”

“EBGAN [243] changes the discriminator to an energy function that will assign low energy values to regions with high data density and higher energy values to other regions.”

“Note that the two criteria rarely coincide in other models, for example, denoising autoencoders (DAE, Vincent et al. (2008)), VAE (Kingma & Welling, 2014)), and DSEBMs (Zhao et al., 2016), causing confusion on which of the decision rules should be employed for outlier detection.”

“For (G1), we introduce the class of Energy Based Model (EBM), which is an important family of generative models [62, 17, 57].”

“…audio generation (Dathathri et al., 2020; Ho et al., 2020; Song & Ermon, 2019; Song et al., 2021), often outperforming their contemporaries, including GANs, VAEs, normalizing flows, and energy-based models (Goodfellow et al., 2014; Kingma & Welling, 2013; Rezende & Mohamed, 2015; Zhao et al., 2017).”

“Interest in GANs has since intensified and several novel approaches have been proposed towards improving training [24–29] and architecture [30–36].”

“Extensive methods have been proposed to improve GAN in various aspects, such as better objective functions [30]–[33], multi-stage generation [34]–[37], and combination with variational auto-encoders [38], [39].”

“These problems have been addressed previously in existing work including energy based GANs [6] and the Wasserstein GAN [7, 8].”

“Some studies propose new objective functions for better convergence of the training process, such as adding new constraints (Arjovsky et al., 2017; Gulrajani et al., 2017) and using smoother non-vanishing or non-exploding gradients (Mao et al., 2017; Zhao et al., 2016).”

“, instability during training) [Arjovsky et al. 2017; Berthelot et al. 2017; Denton et al. 2015; Gurumurthy et al. 2017; Salimans et al. 2016; Springenberg 2016; WardeFarley and Bengio 2017; Zhao et al. 2017].”

“owing at an incredible rate, and a full overview of all the related works such as those by Arjovsky et al. (2017); Arjovsky and Bottou (2017); Huszár (2015); Nowozin et al. (2016); Dai et al. (2017); Zhao et al. (2017); Arora and Zhang (2017); Salimans et al. (2016); Gulrajani et al. (2017); Radford et al. (2015) is b…”

“The generator and discriminator are trained simultaneously and competitively in a mini-max game while convergence is achieved when the Nash equilibrium is reached, i.e. no network can improve through further training if one remains unchanged (Zhao et al., 2017).”

“(Kim and Bengio, 2016; Zhao et al., 2016; Haarnoja et al., 2017) pro-pose learning a separate network to generate samples, which makes these methods closely connected to generative adversarial networks as shown by (Finn et al., 2016), but these methods do not have the advantages of implicit…”

“Existing work includes but is not limited to designing new learning procedures or network architectures (Denton et al., 2015; Radford et al., 2015; Zhang et al., 2017; Zhao et al., 2017; Arora et al., 2017; Tao et al., 2018; Brock et al., 2018), seeking alternative distribution discrepancy measures as loss criteria in…”

“Recently, GANs have seen major successes [18,19,20,21,22,23,24,25,26,27,28] in this task.”

“In particular, we take a similar formulation of the energy-based generative adversarial nets (Zhao et al., 2016) where in our case we use the discriminator’s output as the energy to distinguish the ground-truth translation and the generator’s generation, w.r.t θ : where D is the empirical…”

“Nevertheless, study [26] developed the Energy-Based Generative Adversarial Network (EBGAN), which is capable of producing low-energy samples by incorporating energy functions into both the generator and discriminator components of the network.”

“P [ 14 ] S o P h i e [ 15 ] E n e r gy [ 19 ] S EE M S o c i a l G A N -”

“BEGANs [19] used auto-encoder as the discriminator, as in [16].”

“EBGAN(Zhao et al., 2016) and BEGAN(Berthelot et al., 2017) suggested an energy-based function to develop the discriminator, by which their networks showed stable convergence and were less sensitive to parameter selection.”

“Inspired by the recent success in Generative Adversarial Networks (GAN) [11], [12], [13], [14], [15], we propose to design the “discriminator” as the discriminator network in GAN while the regression network functions as the generative network.”

“In image generation, generative adversarial networks (GANs) [25], which are a form of generative model [26,27], have been widely employed to produce state-of-the-art, realistic images [28,29] for applications such as GAN-based image inpainting [30] and video generation [31,32].”

“Generative adversarial network (GAN) proposed by Goodfellow et al. (Goodfellow, et al. 2014) is a powerful neural network for image generation and has attracted increasing attention (Arjovsky, et al. 2017; Berthelot, et al. 2017; Chen, et al. 2016; Zhao, et al. 2016).”

“In this manner, the model can learn domain-invariant features and transfer them across domains when the Nash Equilibrium is achieved (Zhao et al., 2017).”

“GANs allow for various applications (e.g., image generation [36], [37]; representation learning [38]; image manipulation [39]; object detection [40]; and video application [41]).”

“To facilitate network training, several methods, such as energy-based GANs [45], minibatch discrimination [36], Wasserstein GANs [1, 11], boundary equilibrium GANs [2], and spectral normalization [32], were presented.”

“Inspired by the automatic loss function learning power of Generative Adversarial Networks (GAN) [12], [14], [17], [18], [19], [20], [21] and the capability of neural memory models [10], [17], [22], [23] to store and retrieve semantic level abstractions from historical agent behaviour, we propose a…”

“Models such as GANs [7, 34] and VAEs [23, 15] achieve impressive results in image generation.”

“…Metz, and Chintala 2015; Mirza and Osin-dero 2014; Chen et al. 2016; Makhzani et al. 2015) and the type of objective function used (Metz et al. 2016; Arjovsky, Chintala, and Bottou 2017; Mao et al. 2017; Dziugaite, Roy, and Ghahramani 2015; Zhao, Mathieu, and LeCun 2016; Gulrajani et al. 2017).”

“The GANs framework can be also easily modified with other loss functions and learning dynamics, triggering numerous variants [9, 10, 11, 12, 13, 14, 15].”

“Then, we present works related to adversarial training.”

“Commonly used GAN network models include WGAN, 63 DCGAN, 64 InfoGAN, 65 EBGAN, 66 and LSGAN.”

“An energy-based generative adversarial network (EBGAN) [47] and boundary equilibrium generative adversarial network (BEGAN) [7] modify the loss function with hinge loss.”

“Generative Adversarial Networks (GAN) (e.g., (I. Goodfellow et al., 2014; Zhao, Mathieu, & LeCun, 2017; Arjovsky, Chintala, & Bottou, 2017)) is sometimes mentioned together with adversarial machine learning.”

“Previous researches have fully proved that we can get plausible image results from noise input [20, 28, 49, 24].”

“Some current works on the topic pinpoint this specific problem [22], where conditional image generation is also tackled while other recent techniques [17], [4], [6], [23], [10] try to stabilize the training process.”

“Still other papers suggest the use of auxiliary structures into a GAN to encourage the generator to explore more modes of the true data distribution, including an autoencoder [20, 21, 22, 23], multiple generators [24, 25], a conditional augmentation technique [3], etc.”

“…Autoencoders (VAEs) (Kingma & Welling, 2013), flow-based models such as NICE (Dinh et al., 2014), and their variants (Dziugaite et al., 2015; Zhao et al., 2016; Durugkar et al., 2016; Arjovsky et al., 2017; Mao et al., 2017; Yang et al., 2019; Fabius & van Amersfoort, 2014; van den Oord et…”

“…GAN-based anomaly detection model is adequate for fall detection from among nine recently introduced models because the performance may vary depending on the application (AnoGAN [22], fAnoGAN [23], BiGAN [24], ALAD [25], EBGAN [26], ENCEBGAN [27], SENCEBGAN [27], GANomaly [28], Skip-GANomaly [29]).”

“Generative Adversarial Networks (GAN) [17], [18] are a kind of particular generative models, which utilize adversarial learning strategies to control the weight updating of the generator and discriminator, until a dynamic equilibrium is constructed to achieve an optimal solution.”

“As a typical representative of deep generative models, generative adversarial networks (GANs) are considered to have the ability to generate new data by effectively learning the data distribution, and many efforts have been made in images synthesis, such as EBGAN [6], LSGAN [7], and pix2pix [8].”

“GAN based generative models did not take long to reach impressive image quality [13, 14, 17, 10] at least for some spe-cific datasets.”

“It is interesting that the Energy-Based GAN (EBGAN) [25] makes use of the reconstruction error of a denoising autoencoder in a way compatible with the interpretation proposed here.”

“[15] proposed the conception of Generative Adversarial network (GAN) in 2014, and later several successful revamped versions [2, 66] improved the stability of training process.”

“In contrast to typical VAE-GAN Electronic copy available at: https://ssrn.com/abstract=3421771 approaches (Larsen, Sønderby, and Winther 2015; Zhao et al. 2016; Berthelot et al. 2017), adversarial autoencoders (Makhzani et al. 2015) and adversarial generative encoders (Heljakka, Solin, and Kannala…”

“The critic function was parameterized by neural networks, and then, the energy distance was maximized [30, 41].”

“In a few years, many variations have been invented to improve the original GAN until now, such as f-GAN [10], Energy based GAN [11], info GAN [12], and w-GAN [13], which proves that GAN is a promising model.”

“Impressive results have been obtained when using GANs for image generation [34], [35], image editing [36], and representation learning [37].”

“However, EBGANs also regularize the distribution distance as JS divergence and share the same problem that the discriminator cannot be trained well enough.”

“Indeed, a wide technical documentation about GANs as well as other deep learning generative techniques is available elsewhere (Goodfellow et al., 2014; Radford et al., 2015; Zhao et al., 2016).”

“Recently, generative adversarial networks (GANs) [1] have attracted a tremendous amount of attention and they are successfully applied to many signal generation tasks, such as image generation [2] and image to image translation [3] [4] [5].”

“Kim & Bengio (2016); Zhao et al. (2016) are the two most directly related efforts that connect GANs with EBMs.”

“Examples of such GAN frameworks are Energy-based GAN (EBGAN) [60] and Boundary Equilibrium GAN (BEGAN) [51].”

“Here we utilize the pullaway term (PT) [46] that has been successfully applied in generative adversarial networks:”

“To strive for a more stable GAN training, the energy-based generative adversarial networks (EBGAN) [38] is proposed which views the discriminator as an energy function that assigns low energy to the regions near the data manifold and higher energy to other regions.”

“GANMF (Dervishaj and Cre-monesi 2022) notices that a single-valued feedback from discriminator is deficient in generating high-dimensional data, and models the difference between the real and fake profiles by an autoencoder like EBGAN (Zhao, Mathieu, and LeCun 2016).”

“The generative adversarial network (GAN) is a kind of deep learning models that learns the data distribution of training images and generate synthesized images under the same distribution [21], [36], [37].”

“The energy function is trained to push down on the energy of real samples x , and to push up on the energy of generated samples ˆx [18].”

“2) Adversarial Loss [11], [33]: To generate stylized images similar to the domain Y , we apply the adversarial loss function to update the generator.”

“Second, the different objectives of GAN are compared with the multilevel perspective discrimination in HP-ALF, including the standard GAN (GAN) [21], WGAN [55], HingeGAN [56] and LSGAN [57].”

“Another category of the approach is Energy based model (EBM) [11][12][13].”

“un c ond iti on a l GLO [43] 49.60 57.70 VAE [44] 21.85 69.84 BEGAN [45] 13.54 15.90 EBGAN [46] 11.10 41.32 GLANN [47] 8.60 13.10 WGAN [48] 7.07 28.17 LSGAN [49] 6.75 14.72 DCGAN [12] 4.54 8.22 InfoGAN [50] 28.09 - GLF [51] 5.80 10.30 We study the influences of different choices of some…”

“The energy-based model (EBM) (LeCun et al., 2006; Ngiam et al., 2011; Kim & Bengio, 2016; Zhao et al., 2016; Xie et al., 2016; Gao et al., 2018; Kumar et al., 2019b; Nijkamp et al., 2019; Du & Mordatch, 2019; Finn et al., 2016; Atchadé et al., 2017; De Bortoli et al., 2021; Song & Ou, 2018) defines an unnormalized pro…”

“The losspt denotes the pull-away term loss [9], which is designed to make the generated samples distributed averagely in the central area instead of overfitting at the central point.”

“Much of the fundamental work in GANs have focused on unsupervised learning settings [6, 39, 1].”

“Define the density of X by an energy-based model based on an autoencoder (Gu and Zhu, 2001; Zhao et al., 2017; Berthelot et al., 2017): where and θ ∈ Θ is the unknown parameter and V ( θ ) is the log normalization constant.”

“Several works [12,13,14,15,16] have aimed to tackle such problems.”

“o adays, there are so e i p oved odels have b en p po d t t i t li DCG [ ], CG [58], InfoG [59], G [60], LSG [61], EBG [62], .”

“instability issues such as mode collapse and vanishing gradients [1, 5]. „ere has been considerable work investigating the stabilization of GAN training via the use of di‡erent network architectures [4, 16, 27, 35] and objective functions [1, 6, 34]. However, the interaction between the underlying network architecture…”

“Many research efforts are conducted to improve GAN in stablizing the training process and increasing the generated diversity [29], [20], [1], [2], [7], improving the visual quality and resolution of generated images [25], [11], introducing controllability by adding conditional label information [22], [5], and increasi…”

“, 2017) or an energy defined in terms of a reconstruction loss of an autoencoder (Berthelot et al., 2017; Zhao et al., 2017).”

“The literature on GANs has been growing at an incredible rate, and due to space constraints, we cannot give a full overview of all the related works, such as [1, 2, 4, 9, 16, 20, 26, 29, 35, 40].”

“Other examples not used here are the Boundary Equilibrium GAN [3] and Energy Based GAN [16].”