Energy-based Generative Adversarial Network

“We impose the following restrictions on EBGAN models: (i)-using learning rate 0.001 and Adam (Kingma & Ba, 2014) for both G and D ; (ii)- nLayerD represents the total number of layers combining Enc and Dec .”

“Finally, we trained EBGANs to generate high-resolution images on ImageNet (Russakovsky et al., 2015).”

“For training both EBGANs and GANs for the grid search, we use the following setting: • Batch normalization (Ioffe & Szegedy, 2015) is applied after each weight layer, except for the generator output layer and the discriminator input layer (Radford et al., 2015).”

“Generative Adversarial Networks (GAN) (Goodfellow et al., 2014) have led to significant improvements in image generation (Denton et al., 2015; Radford et al., 2015; Im et al., 2016; Salimans et al., 2016), video prediction (Mathieu et al., 2015) and a number of other domains.”

“(2010) and Rifai et al. (2011). In quest for better techniques for stabilizing GAN training, several papers was presented with helpful techniques in Salimans et al.”

“Generative Adversarial Networks (GAN) (Goodfellow et al., 2014) have led to significant improvements in image generation (Denton et al., 2015; Radford et al., 2015; Im et al., 2016; Salimans et al., 2016), video prediction (Mathieu et al., 2015) and a number of other domains.”

“We also trained a network to generate images of size 256 × 256 , on a dog-breed subset of ImageNet, using the wordNet IDs provided by Vinyals et al. (2016).”

“By adding regularizers, several techniques were addressed to resolve such problem (Vincent et al., 2010; Rifai et al., 2011; Ng; MarcAurelio Ranzato & Chopra, 2007; Kavukcuoglu et al., 2010).”

“To access the capacity of generating 64× 64 RGB images on the basis of using deep convolutional models, we have trained EBGANs on the LSUN bedroom dataset (Yu et al., 2015) and the large scale face dataset CelebA under alignment (Liu et al.”

“, 2014) have led to significant improvements in image generation (Denton et al., 2015; Radford et al., 2015; Im et al., 2016; Salimans et al., 2016), video prediction (Mathieu et al., 2015) and a number of other domains. The basic idea of GAN is to simultaneously train a discriminator and a generator. The discriminato…”

“sed in [9]. The generators and discriminators are typically deep neural networks. Recent work has generalized the objective in Eq. (1) to arbitrary f-divergences [18] and integral probability metrics [28, 24]. In particular, Wasserstein GAN (WGAN, [1]), where the minimax is over the Wasserstein distance between the mo…”

“Firstly, some works managed to integrate some structures into the framework to do inference, i.e., projecting sample in data space back into the space of latent features, such as BiGAN (Donahue, Kr¨ahenb¨uhl, and Darrell 2017), ALI (Dumoulin et al. 2017) and EBGAN (Zhao, Mathieu, and LeCun 2016).”

“We also use an additional PatchGAN discriminator D 1 to facilitate adversarial training, which takes a pair of images as input and outputs a vector of true/fake probabilities.”

“Lastly, the mode collapse loss is introduced to prevent the model from producing clustered samples [60], by forcing the generated samples to be different in terms of their activations in the style network.”

“Up to now, many works [32, 33, 34, 35, 36, 4, 5, 6] have been proposed to improve the vanilla GAN.”

“TheuseandapplicationofDLmodelsinvolvesseveralothertech-niques that have grown at the same time, and among them Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) (Goodfellow et al., 2014; Kingma & Welling, 2013; Radford, Zhao, Mathieu, & LeCun, 2016) have attracted much interest in recent times.”

“In recent times, GAN-based approaches have been successfully used to generate impressively realistic faces [26, 20], house-numbers [60], bedrooms [35] and a variety of other image categories [15, 65] through a two-player game between a generator G and discriminator D.”

“Many improvements have been suggested which range from changes in the architecture of generator and discriminator to modifications in the loss function and adoption of good training practices [2, 41, 42, 14].”

“Particularly promising results have recently been reported for WGAN, which uses the Wasserstein distance to reportedly overcome issues of mode collapse (Arjovsky et al., 2017); and EBGAN (Zhao et al., 2016), which is (to our knowledge) the first to generate plausible images at size 256 x 256 over 1000 semantic classes.”

“Image warping and morphing are two approaches that can be used to accomplish this goal (Zhao et al. 2016; Berthouzoz et al. 2011; Xu et al. 2022).”

“One is the theoretical thread that tries to alleviate the instability and mode collapse problems of GAN [5] [6] [7] [8] [9] [10], or reformulate it from different angles like information theory [11] and energy-based models [12].”

“Similar to WGAN, EBGAN (Energy-Based GAN)[19] uses energy values as a measure from the energy model; BEGAN (Boundary Equilibrium GAN)[20] also proposes a “boundary balance” architecture based on EBGAN and WGAN, using standard training steps to achieve fast and stable convergence.”

“Similarly, the LSGAN [47] method leads to minimizing the χ 2 - divergence [56] and the EBGAN [81] scheme minimizes the total variation distance [2].”

“The discriminator loss LDeb in EBGAN is given as, LDeb = Ex̂∼pdata [ ||DAE (x̂) − x̂|| ] +Ez∼pnoise [ max(0, m − ||DAE (G(z)) − G(z)|”

“In contrast, the auto-encoders in [1, 37] are much deeper and involve non-linear mappings, since the data structure remains unknown and thus complex autoencoders are necessary to learn the data structure.”

“Generative adversarial network (GAN) [9, 29, 6, 48] is one of the most popular deep generative models for image generation, and has shown impressive results in many applications [44, 1, 15, 49, 27].”

“GANs have found remarkable applications in modeling image distributions [10, 11, 12, 13], and a well-trained GAN closely captures the notion of a signal (or image) being ‘natural’ [14].”

“, VAE [1], [2], [3], [4], EBM [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], [21], GAN [22], [23], [24], normalizing flow [25], [26], [27], [28], [29], [30] and diffusion models [31], [32], [33], [34], [35] , have shown great potential in creating new patterns that humans ca…”

“Details BEGAN [10] L Wasserstein distance between loss distributions EBGAN [143] Margin m is adjusted in EBGAN’s training Progressive GAN [56] succeeds in to generating high resolution images in a single GAN, making training faster and more stable.”

“Zhao et al. (2017); Che et al. (2020); Song et al. (2020); Arbel et al. (2020) formulate EBMs with GANs, and use the discriminator to assign an energy.”

“To incorporate per-pixel supervisions [1, 35], this work employs an auto-encoder as discriminator D, which is introduced in [2].”

“Guo et al. (2018) added a reconstruction error corresponding to standard VAE loss, a style loss (Gatys et al., 2015), which evaluates how well the generated shape match the style of the real shape, and a mode-collapse loss (Zhao et al., 2016) to prevent only similar samples from being generated.”

“This kind of adversarial training achieves a win-win situation when the generator and discriminator reach a Nash Equilibrium (Zhao et al., 2016; Arora et al., 2017; Guimaraes et al., 2017).”

“To minimize −H( p G ) , we adopt the pull-away term (PT) proposed by [9, 50] that encourages the generated feature vectors to be orthogonal.”

“Later, variants of GANs have been proposed to increase the resolutions and quality of generated images [5, 32, 1, 3].”

“[35] used an auto-encoder architecture to improve the stability of GANs learning.”

“Another branch of work uses autoencoders as discriminators such as EBGAN [42].”

“There are numerous works inspired by the original GANs, [2,3,4,5] to name a few.”

“As compared to the other GAN architectures, the conditional GANs have significant performance on the multi-modal data in comparison with [11,62,107].”

“Recently, a few approaches have tried to learn an energy-based model [20, 19] using deep neural networks [3, 30].”

“Several techniques have been proposed to stabilize the training techniques [24, 21, 27, 1, 43] and generate compelling results.”

“…an expressive family of probabilistic models that have attracted much attention in recent research of deep generative models (Kim & Bengio, 2016; Zhao et al., 2016; Goyal et al., 2017; Xie et al., 2016b; Finn et al., 2016; Gao et al., 2018; Kumar et al., 2019; Nijkamp et al., 2019b; Du &…”

“Zhao et al. proposed an energy-based generative adversarial network (EBGAN) [25] , which views the discriminator as an energy function instead of a probability function.”

“In the 2D image domain, people have developed successful methods for deep generative models such as generative adversarial networks (GANs) [Arjovsky et al. 2017; Goodfellow et al. 2014; Salimans et al. 2016; Zhao et al. 2016], variational autoencoders [Kingma et al.”

“, 2017) and Energybased GAN (EGAN) (Salimans et al., 2016; Gulrajani et al., 2017; Zhao et al., 2017; Berthelot et al., 2017).”

“Unsupervised learning of generative image models is an active research area, and many papers evaluate on the celebA dataset [24] of face images [24, 25, 26, 27].”

“06 EBGAN [58] 2.”

“Instead, dropout layers are employed during both dataset training and prediction phases to introduce randomness; the generator is then directed by the loss functions throughout the training progression [135].”

“Zhao et al. [37] proposed an energy-based generative adversarial network, using the discriminator in GANs as an energy function, and training a single-scale network architecture to generate high-resolution images.”

“Recent developments have witnessed the promising performance of generative adversarial networks [Goodfellow et al. , 2014; Arjovsky et al. , 2017; Zhao et al. , 2017] in unsupervised learning [Peng et al. , 2017; 2016].”

“At the Nash equilibrium [50], the distribution generated by G equals to the real data distribution.”

“Unconditional GANs [16, 37, 44, 58, 1, 18] can generate realistic images without any supervision.”

“Another avenue for improving the stability of GANs has been the replacement of the classifying discriminator with the regression-based one as in energy-based GANs [30] and Wasserstein GANs [1].”

“ucing diverse samples that are close in the embedding space to the ground truth, due to perceptual similarity learned in D. One can also draw connections between our work and Energy Based GAN (EBGAN) [54] – without the adversarial training aspect. The “energy” in our case is a deep metric-learning based scoring mechan…”

“Besides, in order to fully represent the diversity of the selected frames without summary annotations, we use the repelling loss Lrep [35] to enhance the diversity among video frames.”

“Energy based generative adversarial network architecture with autoencoder taken from [27] introduction of the repelling regularizer term that is added to the generator loss in the hopes of obtaining orthogonal representations of the pairwise samples.”

“On the other end of the spectrum of deep generative models lies the family of energy-based models (EBMs) [42, 54, 29, 77, 73, 13, 38, 55, 9, 11, 16, 18, 6].”

“Energy Based GAN (EBGAN) aims to model the discriminator D(x) as an energy function [20].”

“Energy-based GAN (EBGAN) [18] introduces the idea of energy minimization into a GAN architecture.”

“The connection between GANs and EBMs has been studied by many authors [45–48].”

“EBGAN [66] has been proposed to model the discriminator as an energy function.”

“Examples of the latter include Conditional GAN (CGAN) [50], CycleGAN [10], Bidirectional GAN (BiGAN) [51], DualGAN [11], Disco-GAN [52], Pix2Pix [9], InfoGAN [53], Energy-based GAN (EBGAN) [54], Wasserstein GAN (WGAN) [55], and Super-Resolution GAN [15].”

“As it is well-known that GANs are hard to train and unstable, there have been lots of variations of GANs [43, 2, 9, 25, 17].”

“We leverage pull-away term losspt [39] to satisfy it.”

“Boundary Equilibrium GAN uses an autoencoder architecture for the discriminator which was first proposed in [54].”

“[46] propose energy-based GANs (EBGANs), in which the discriminator can be viewed as an energy function.”

“This choice is motivated by Energy-based GAN (Zhao et al., 2016) in which the autoencoder loss is used as a discriminator but without a probabilistic interpretation.”

“In GAN, convergence stability is improved by gradient penalization instead of clipping 35 with the cost function converging to the Nash equilibrium.”

“Turning a GAN discriminator into an energy function has been studied in the past (Kim & Bengio, 2016; Zhao et al., 2016; Dai et al., 2017) but in order to turn a GAN discriminator into an energy function, a crucial and difficult requirement is the maximization of entropy at the output of the…”

“, (Zhao et al., 2016; Salimans et al., 2016; Donahue et al., 2016; Dumoulin et al., 2016; Arjovsky et al., 2017; Hjelm et al., 2017; Li et al., 2017; Zhu et al., 2017), but a way of solving all three issues simultaneously remains elusive.”

“In future, we plan to employ other GANs to improve the performance, such as Wasserstein GAN [3], LS GAN [26], EB GANs [14].”

“Multiple researchers Radford et al. (2015); Salimans et al. (2016); Zhao et al. (2016) have shown that the adversarial interaction with the discriminator can result in a generator that produces compelling samples.”

“Similar capacity assumptions have been made in nearly all other GANs to prove its discriminative power; see, e.g., Zhao et al. (2016); Nowozin et al. (2016); Arora et al. (2017).”

“, probabilistic models [21], partial differential equation models [22], and deep networks [23]–[25]) can be reformulated in the light of “energy minimization.”

“…natural images, GAN has been actively adopted for a variety of computer vision applications, including natural image synthesis [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], image-to-image translation [12], [13], [14], [15], [16], [17], super-resolution imaging [18], [19], [20], [21],…”

“Similar to EBGAN [37], we add the Gaussian noise to the generator.”

“Many GAN variants that use different losses have been introduced (Arjovsky et al., 2017; Mao et al., 2016; Mohamed and Lakshminarayanan, 2016; Nowozin et al., 2016; Zhao et al., 2017), but the adversarial game remains the same.”

“Conditional GAN [25] and Triple-GAN [26] introduced conditions’ information to the training while EBGAN [27] and W-GAN [28] modified the objective function to stabilize the training process.”

“This consists of generating entire non-existent face images [113–115].”

“bers [30], flowers [31], birds [32], and so on.”

“Typical examples are: employing Wasserstein-1 Distance with a Gradient Penalty [2,19]; using a quantile regression loss function to implicitly push G to learn the inverse of a cumulative density function [30]; rewriting the objective function with a mean squared error form – now minimizing the χ(2)-distance [28]; or e…”

“Recently, many works [17]–[19], [23]–[25] have been proposed to improve the vanilla GAN.”

“pix2pix ), where blurred images where output as a result [34,11,35].”

“Energy-based GAN (EBGAN) [20] is a typical one in this form.”

“eloped in recent years. There are various types of GANs: with different metrics of comparing two distributions (e.g. KL divergence for the original GAN [2], Wasserstein distance for WGAN [5,6], EBGAN [7], BEGAN [8], Loss-Sensitive GAN [9]), with regularization on the loss function (e.g. WGAN-GP [5], DRAGAN [10]), with…”

“Interestingly, the contrastive points can be also generated with generative adversarial networks (GANs) [ 44, 45 ] .”

“(6)Thus, ti can be viewed as an energy function [48] we expect to minimize by updating the parameters of the reconstruction network.”

“BEGAN [5] extended EBGAN by optimizing Wasserstein distance between AE loss distributions.”

“To minimize H(pG), we adopt the pull-away term [56], [57].”