Energy-based Generative Adversarial Network

“We impose the following restrictions on EBGAN models: (i)-using learning rate 0.001 and Adam (Kingma & Ba, 2014) for both G and D ; (ii)- nLayerD represents the total number of layers combining Enc and Dec .”

“Finally, we trained EBGANs to generate high-resolution images on ImageNet (Russakovsky et al., 2015).”

“For training both EBGANs and GANs for the grid search, we use the following setting: • Batch normalization (Ioffe & Szegedy, 2015) is applied after each weight layer, except for the generator output layer and the discriminator input layer (Radford et al., 2015).”

“Generative Adversarial Networks (GAN) (Goodfellow et al., 2014) have led to significant improvements in image generation (Denton et al., 2015; Radford et al., 2015; Im et al., 2016; Salimans et al., 2016), video prediction (Mathieu et al., 2015) and a number of other domains.”

“(2010) and Rifai et al. (2011). In quest for better techniques for stabilizing GAN training, several papers was presented with helpful techniques in Salimans et al.”

“Generative Adversarial Networks (GAN) (Goodfellow et al., 2014) have led to significant improvements in image generation (Denton et al., 2015; Radford et al., 2015; Im et al., 2016; Salimans et al., 2016), video prediction (Mathieu et al., 2015) and a number of other domains.”

“We also trained a network to generate images of size 256 × 256 , on a dog-breed subset of ImageNet, using the wordNet IDs provided by Vinyals et al. (2016).”

“By adding regularizers, several techniques were addressed to resolve such problem (Vincent et al., 2010; Rifai et al., 2011; Ng; MarcAurelio Ranzato & Chopra, 2007; Kavukcuoglu et al., 2010).”

“, 2014) have led to significant improvements in image generation (Denton et al., 2015; Radford et al., 2015; Im et al., 2016; Salimans et al., 2016), video prediction (Mathieu et al., 2015) and a number of other domains. The basic idea of GAN is to simultaneously train a discriminator and a generator. The discriminato…”

“To access the capacity of generating 64× 64 RGB images on the basis of using deep convolutional models, we have trained EBGANs on the LSUN bedroom dataset (Yu et al., 2015) and the large scale face dataset CelebA under alignment (Liu et al.”

“Generative Adversarial Networks (GAN) (Goodfellow et al., 2014) have led to significant improvements in image generation (Denton et al., 2015; Radford et al., 2015; Im et al., 2016; Salimans et al., 2016), video prediction (Mathieu et al., 2015) and a number of other domains.”

“L G ] 6 M ar 2 01 7”

“Specifically, we utilize an identical architecture as reported in both papers (Rasmus et al., 2015; Pezeshki et al., 2015); namely a fully-connected network of size 784-1000-500-250-250-250 , with batch normalization and ReLU following each linear layer.”

“, 2014) have led to significant improvements in image generation (Denton et al., 2015; Radford et al., 2015; Im et al., 2016; Salimans et al., 2016), video prediction (Mathieu et al.”

“We leave the usage of upsampling or switches-unpooling (Zhao et al., 2015) to future research.”

“Specifically, we utilize an identical architecture as reported in both papers (Rasmus et al., 2015; Pezeshki et al., 2015); namely a fully-connected network of size 784-1000-500-250-250-250 , with batch normalization and ReLU following each linear layer.”

“ions, and show that our method outperforms these baselines. We provide bothPyTorchandTorchimplementations. Check out more results at ourwebsite. 2. Related work Generative Adversarial Networks (GANs) [16,63] have achieved impressive results in image generation [6, 39], image editing [66], and representation learning […”

“Fortunately, this is exactly what is done by the recently proposed Generative Adversarial Networks (GANs) [22, 12, 41, 49, 59].”

“Viewing the discriminator as an energy function, [33] used an auto-encoder architecture to improve the stability of GANs learning.”

“GANs enable a wide variety of applications such as image generation [1, 41, 60], representation learning [44], image manipulation [62], object detection [32], and video applications [37, 50, 52].”

“Arjovsky [32] proposed to compare distributions based on a Wasserstein distance rather than a KL-based divergence (DCGAN [5]) or a total-variation distance (energy-based GAN [50]).”

“Generative adversarial networks (GANs) [3] have shown remarkable results in various computer vision tasks such as image generation [1, 6, 23, 31], image translation [7, 8, 32], super-resolution imaging [13], and face image synthesis [9, 15, 25, 30].”

“of units with an energy defined for the overall network. Energy-based learning [21, 31,32] provides a unified framework for many probabilistic and non-probabilistic approaches to learning. Recent work [49] also demonstrated using energy functions to train GANs [10], where the discriminator uses energy values to differen…”

“An energy-based GAN [38] has also been proposed for more stable training behavior.”

“In EBGAN (Zhao et al., 2016) and BEGAN (Berthelot et al., 2017) (Figure 3 c), the commonly used encoder architecture for discriminator is replaced with an autoencoder architecture.”

“For the first class of stable equilibria, which we call energy solutions , the equilibrium discriminator forms an energy function for the true data distributions and might be a partial explanation for the success of autoencoder-based discrim-inators (Zhao et al., 2016; Berthelot et al., 2017).”

“…generally fall into two categories: methods that use Markov chain Monte Carlo (MCMC) based sampling (Sallans & Hinton, 2004), and methods that learn a stochastic sampling network trained to output approximate samples from the target distribution (Zhao et al., 2016; Kim & Bengio, 2016).”

“Energy-based GANs [27] also improve training stability by modeling the discriminator as an energy function, whereas the Least-Squared GAN [28] addresses the vanishing gradients problem by adopting a least-square loss function for the discriminator.”

“Energy-based Models (EBMs) [26, 48, 58, 64, 67, 68, 75, 78, 79, 120, 129, 132, 165, 170, 182, 196, 254, 281] can be viewed as one generative version of discriminators [79, 104, 131, 134], while can be learned from unlabeled input data.”

“GANs have recently been very successful [1, 4, 6, 8, 10, 35, 36, 49, 53, 54].”

“Some GAN-related works have shown visually impressive results of randomly drawing face images [31, 19, 32, 21, 17].”

“EBGAN has more stable behavior than original GANs during training.”

“[40] where they deviate from the traditional probabilistic interpretation of the discriminator model.”

“Heuristic regularizers such as batch discrimination [16] and the repelling regularizer [21] have been proposed to alleviate this problem with varying degrees of success.”

“[56], a DCGAN model [32] is trained for quantitative comparison using the public available source code on the ImageNet Dog dataset.”

“There have been many other approaches trying to blend the adversarial training of GANs with auto-encoder architectures [17, 18, 19, 20].”

“…the GAN framework has also enabled a number of successful extensions of the technique, for instance for structured prediction (Reed et al., 2016a;b; Odena et al., 2016), training energy based models (Zhao et al., 2016), and combining the GAN loss with a mutual information loss (Chen et al., 2016).”

“Energy-based GAN was proposed by Yann LeCun from Facebook in 2016 [195].”

“izing the entropy H(P G(x)) in the objective (3) is technically challenging since a GAN does not model the generator distribution explicitly. To handle the issue, we leverage the pull-away term (PT) (Zhao et al., 2017): H (P G(x)) w PT(P G(x)) = 1 M(M 1) XM i=1 X j6=i G(z i) >G(z j) kG(z i)kkG(z j)k ! 2 ; where z i…”

“function which maps to low energy values where the data falls in a high-dimensional space (e.g., where actual CelebA pictures are positioned in the image space) and high energy values everywhere else [127]. GANs offer one way to learn this energy function. The samples from the dataset indicate where the energy functio…”

“Energy-based GANs are proposed in [562], it is claimed that they have better stability.”

“, 2016], EBGAN [Zhao et al., 2016] and ALI [Dumoulin et al.”

“However, we first point that there are several recent works, such as EBGAN [7], also matching pairwise relation between samples of batch size, leading to O ( B 2 ) complexity as well.”

“To this end, there have been a number of recent methods which hypothesize a different loss function for the discriminator [17], [18], [19], [20].”

“In order to improve sample quality and address overpowered discriminators, numerous works have taken inspiration from the connection between GANs and energy models [256].”

“In this section, the different loss components and network architecture of MedGAN will be presented starting first with some preliminary information.”

“[36] introduces energy-based GAN (EBGAN), which views the discriminator as an energy function that attributes low energies to the regions near the data manifold and higher energies to other regions.”

“Similar idea is adopted by Zhao et al. (2016).”

“Based on this observation, we provide new geometric interpretations of GAN [1], f -GAN [3], EB-GAN [13], and W-GAN [7] in terms of separating hyperplanes and geometric scaling factors, and discuss their limitations.”

“In this paper, we use two standard definitions for diversity regularization – namely, (i) Determinantal Point Process (DPP) [38, 7, 43]; and (ii) Repelling regularizer (REP) [45].”

“References [22], [24], and [26] used different divergences to construct the objective function to enhance the GANs’ stability.”

“However, the core idea of representation learning with autoencoders is still used today to learn relevant statistics of images for inverse problems and can be a central component in some of the generative models described in the section “Recent Developments: Increasing the Perceptual Quality of Images Predicted by Neu…”

“Specifically, we adapt the pull-away term (PT) proposed by [25], , where N is the size of a mini-batch and x are samples.”

“Zhao, et al. [85] presented an Energy-based GANs (EBGAN) in which an autoencoder is used for D and implemented an energy function on D to assign low energies to regions near the data manifold and higher energies to other regions (see Figure 22).”

“These rules are amenable to stochastic approximation and alternating optimisation, as described by Goodfellow et al. (2014), along with many of the insights for optimisation that have been developed since (Radford et al., 2015; Salimans et al., 2016; Sønderby et al., 2016; Zhao et al., 2016).”

“For instance, Zhao et al. [2016] replace the discrimi-nator of a GAN by an AE, and Ulyanov et al. [2017] replace the decoder of an AE by a generator of a GAN.”

“There are also energy-based GAN variants, such as BEGAN [3] and EBGAN [25].”

“Independent of our work, [4] proposes to regularize EBGAN [68] with entropy maximization in order to prevent the discriminator from degenerating to uniform prediction.”

“As is widely known, various generative models (Goodfellow et al., 2014; Radford et al., 2016; Zhao et al., 2017) learn the distribution p ( x ) of the images within a specific domain, such as human faces or bedrooms.”

“From this point of view, an autoencoder architecture is used for the discriminator [44].”

“Using GANs, many researches have gained encouraging performance through architectural improvements and modification to the training scheme (Radford et al., 2015; Arjovsky and Bottou, 2017; Arjovsky et al., 2017; Berthelot et al., 2017; Zhao et al., 2016; Bi et al., 2017).”

“2) Frechet Inception Distance: The Frechet Inception Distance (FID) has emerged as a well-established metric for evaluating the effectiveness of generative models, particularly GANs, in terms of image quality and diversity [65].”

“Much work has been [54, 26, 23, 4, 15] proposed to improve GANs by stabilizing training behavior and eliminating mode collapse, via using alternative objective losses.”

“Recently, there have been a lot of focus on training adversarial generation networks to create Generative Adversarial Networks (GANs) [3, 22, 32, 24].”

“ginal GAN model [19] was proposed, many variants have appeared in the past few years, for example, to improve the quality of the generated images [12,15,25,36], or to stabilize the training procedure [7,9,20,34,36,40,57]. GANs have also been modied to generate images of a given class by conditioning on additional info…”

“Some of these GANs-based works are committed to training a better generative model, such as InfoGAN [37], Energy-based GAN [38], WGAN(-GP) [39], [40], Progressive GAN [41], E-GAN [42] and SN-GAN [43].”

“In this work, we particularly focus on EBMs trained with maximum likelihood, and expand upon the connection recognized by Kim & Bengio [12] for the case where the generator’s density can be computed.”

“However, fixed margin m phenomena used in EBGAN make it difficult to adapt to the changing dynamics of the discriminator and generator, which hurts the performance of the discriminator network in-reconstructing the real samples because the energy value of the generated example vary near the margin m .”

“The dataset with lesion masks and dataset with DR & DME grading labels may have domain shifts, and generative adversarial networks [57]–[59] will be beneficial for this task.”

“…models, GANs only provide samples rather than an analytic density form, the entropy can either optimized in the input space i . e ., H( p G ( x )) using variational inference or the feature space i . e ., H( p G ( h ( x ))) using a pull-away term (PT) [183] as an auxiliary cost for the entropy.”

“This problem also exists in the recently proposed energy-based GAN (EBGAN) (Zhao et al., 2017), as the distance metric implied by EBGAN is the Total Variance Distance (TVD), which has the same issue w.”

“While research is still underway to improve training techniques and heuristics (Salimans et al. (2016)), most approaches have focused on understanding and generalizing GANs theoretically with the aim of exploring more tractable formulations (Zhao et al. (2016); Li et al. (2015); Uehara et al.…”

“The purpose of the reconstruction is to encourage the generator to preserve the contents of the original input image [13, 147] or to adopt auto-encoder architecture for the discriminator [8, 143].”

“EBGAN[70] replaces the discriminator with an autoencoder and uses the reconstruction cost (MSE) to criticize the real and generated images.”

“To implement the adversarial structure in learning data embedding, existing works like BiGAN [22], EBGAN [23] and ALI [24] arrive at extending the original adversarial framework with exter-nal structures for the inference, which have achieved non-negligible performance in applications, such as document retrieval [41]…”

“Inspired by the recent success in Generative Adversarial Networks (GAN) [24, 39, 27, 12, 9], we propose to design the “discriminator” as the discriminator network while the regression network functions as a generative network.”

“GAN [21] and least-squares GAN [20] improved training stability by employing different training objectives.”

“We train D to minimize the marginbased reconstruction error proposed by [38], Fully unsupervised training using only the reconstruction and adversarial loss on the output image will often result in semantically meaningless latent representations.”

“Although our self-supervision strategy for D comes in the form of an auto-encoder (AE), this approach is fundamentally different from works trying to combine GAN and AE (Larsen et al., 2016; Guo et al., 2019; Zhao et al., 2016; Berthelot et al., 2017).”

“osed for both the network architecture, as in Deep Convolutional GAN [22], Stacked GAN [23] and Progressive Growing GAN [24], as well for the loss functions used to optimize it, as in Energybased GAN [25], Boundary Equilibrium GAN [26] and the Wasserstein GAN [27]. We focus on the loss function proposed in [27] which…”

“In general, many other uses of generative networks are distinct from our problem setting [13, 5, 51, 37, 30, 6, 8], as they deal primarily with generating novel images rather than changing existing ones.”

“Notable exceptions are [55,70] which utilize an encoder-decoder discriminator architecture.”

“We use a GAN (generative adversarial network) [4, 19, 22, 33] to inject realism into our synthesized images.”

“Numerous GAN-like architectures have been proposed during the last years to solve those issues to some degree [40, 23, 27, 4, 13].”

“However, despite the availability of a plethora of GAN models (Arjovsky et al., 2017; Qi, 2017; Zhao et al., 2016), their evaluation is still predominantly qualitative, very often resorting to manual inspection of the visual fidelity of generated images.”

“Many methods [44, 1, 2, 15, 33] have been attempted to improve GANs in terms of their training stability and sample variation.”

“Our design decision of having a trainable mixture model for latent space can be viewed as an algorithmic “plug-in” that can be added to almost any GAN framework including recently proposed models [24, 21] to obtain better performance on diverse data.”

“There is also prior work that attempts to introduce more structured discriminators, for instance, the energy-based GAN (EBGAN) (Zhao et al., 2016) and RankGAN (Lin et al., 2017).”

“We use a repelling regularizer [34] L div to enforce diversity among selected keyframes.”

“Boundary Equilibrium GAN (BEGAN) uses an autoencoder architecture for the discriminator that was first proposed in EBGAN [151].”

“the inherent uncertainty in futureprediction tasks. GAN training has recently been improved using the Wasserstein GAN approach of Arjovsky and Bottou [2], and the energy based approach of Zhao et al. [37]. An interesting approach using GANs for unsupervised image representation learning is proposed in [5], where the g…”

“DeConv was the deconvolutional layer [43] that was used to rescale the images to match them to the actual size.”

“EBGAN Energy Based GAN (EBGAN) [127] uses auto-encoder architecture to construct the discriminator as an energy function instead of a classifier.”

“…the original GAN work for image generation (Radford et al., 2015; Denton et al., 2015; Salimans et al., 2016; Chen et al., 2016; Zhu et al., 2016; Zhao et al., 2016), multi-stage image generation including part-based models (Im et al., 2016; Kwak & Zhang, 2016), image generation conditioned on…”

“We note that ST-GAN is amenable to any other GAN variants [21, 38, 3], and that the choice of GAN architecture is orthogonal to this work.”

“Alternative loss functions have been proposed to mitigate this problem (Mao et al., 2016; Zhao et al., 2016; Arjovsky et al., 2017).”

“…et al., 2016; Zhang et al., 2017; Lu et al., 2022; Liu et al., 2021; Zhang et al., 2022a; Wang et al., 2022e; Zamir et al., 2022) focused on improving the representation capability of IR networks for complicated degradation through well-designed backbones based on Convolutional neural networks…”

“Most of them are the dual form of f -divergence (Goodfellow et al., 2014; Mao et al., 2017; Nowozin et al., 2016), integral probability metrics (IPMs) (Zhao et al., 2016; Li et al., 2017a; Arjovsky et al., 2017; Gulrajani et al., 2017) or IPM extensions (Mroueh and Sercu, 2017; Mroueh et al., 2017).”

“). Following [4], the covariance matrices and centroids are chosen such that the distribution exhibits severely separated modes, which makes it a relatively hard task despite its 2D nature. Following [21], to study stability, we run an exhaustive grid search over a set of architectural choices and hyper-parameters, 57…”