Energy-based Generative Adversarial Network

“We impose the following restrictions on EBGAN models: (i)-using learning rate 0.001 and Adam (Kingma & Ba, 2014) for both G and D ; (ii)- nLayerD represents the total number of layers combining Enc and Dec .”

“Finally, we trained EBGANs to generate high-resolution images on ImageNet (Russakovsky et al., 2015).”

“For training both EBGANs and GANs for the grid search, we use the following setting: • Batch normalization (Ioffe & Szegedy, 2015) is applied after each weight layer, except for the generator output layer and the discriminator input layer (Radford et al., 2015).”

“Generative Adversarial Networks (GAN) (Goodfellow et al., 2014) have led to significant improvements in image generation (Denton et al., 2015; Radford et al., 2015; Im et al., 2016; Salimans et al., 2016), video prediction (Mathieu et al., 2015) and a number of other domains.”

“(2010) and Rifai et al. (2011). In quest for better techniques for stabilizing GAN training, several papers was presented with helpful techniques in Salimans et al.”

“Generative Adversarial Networks (GAN) (Goodfellow et al., 2014) have led to significant improvements in image generation (Denton et al., 2015; Radford et al., 2015; Im et al., 2016; Salimans et al., 2016), video prediction (Mathieu et al., 2015) and a number of other domains.”

“We also trained a network to generate images of size 256 × 256 , on a dog-breed subset of ImageNet, using the wordNet IDs provided by Vinyals et al. (2016).”

“By adding regularizers, several techniques were addressed to resolve such problem (Vincent et al., 2010; Rifai et al., 2011; Ng; MarcAurelio Ranzato & Chopra, 2007; Kavukcuoglu et al., 2010).”

“To access the capacity of generating 64× 64 RGB images on the basis of using deep convolutional models, we have trained EBGANs on the LSUN bedroom dataset (Yu et al., 2015) and the large scale face dataset CelebA under alignment (Liu et al.”

“, 2014) have led to significant improvements in image generation (Denton et al., 2015; Radford et al., 2015; Im et al., 2016; Salimans et al., 2016), video prediction (Mathieu et al., 2015) and a number of other domains. The basic idea of GAN is to simultaneously train a discriminator and a generator. The discriminato…”

“In the spirit of probabilistic GAN [49], we use one loss function L D to train the discriminator ( D ) and another one L G to train the generator ( G ): where ξ denotes the positive margin and the operator [ · ] + = max(0 , · ) .”

“ML includes traditional algorithms, for example, support vector machine (SVM) [16] and linear discriminant analysis (LDA) [17], but also deep learning (DL), such as convolutional neural network (CNN) [18] and generative adversarial network (GAN) [19].”

“However, EBGAN (Energy-based GAN) first proposed by [13] which extends the GANs with an auto-encoder/decoder which is pre-trained (See Figure 5 ).”

“We omit other previous methods since as a representative of state-of-the-art GAN model, WGAN-GP has been shown to rival or outperform a number of former methods, such as the original GAN [13], Energy-based generative adversarial network [41], the original WGAN with weight clipping [4], Least Squares GAN [24], Boundary…”

“Lots of works have improved the stability of GANs by modifying the learning objectives [2, 5, 34, 49, 66] and the regularization function [14, 36, 64].”

“Many research efforts are conducted to improve GAN in stablizing the training process and increasing the generated diversity [20]–[24], improving the visual quality and resolution of generated images [7], [25], introducing controllability by adding conditional label information [26]–[28], and…”

“Several studies have introduced alternative methods for training GANs [47]–[50].”

“An energy function is computed by the discriminator that works essentially as a cost function [55].”

“…VAE-based image generative models, generative adversarial networks (GANs) [1, 8, 28] have a wide range of applications such as image generation [35, 55, 56], representation learning [16, 42], image manipulation [53, 57], object detection [23], video applications [33, 45, 46], and saliency…”

“The energy-based generative adversarial network model was proposed and trained to generate high-resolution images in the work [48].”

“For example, Zhao et al. [32] proposed energy-based GAN (EBGAN) that considers the generator and discriminator as energy functions.”

“There are various types of generative models in the literature, such as VAE (Kingma & Welling, 2013), GAN (Goodfellow et al., 2014a) and reversible generative models, e.g. Real NVP (Dinh et al., 2016).”

“Some works have focused on the architecture of the original GAN approach and have provided better solutions for image generation [52]–[55].”

“As they improved, GANs also began seeing widespread success in a variety of applications, including image generation [5, 22, 33], image editing [34], and even unsupervised or semi-supervised representation learning for standard computer vision tasks [27, 7, 8].”

“EBGAN (Zhao, Mathieu, and LeCun 2017) introduces the energy-based model, in which the discriminator is considered as the energy function minimized via reconstruction errors.”

“Among them, the two prominent ones are Variational Auto-Encoders (VAE) [3, 12] and Generative Adversarial Networks (GAN) [2, 4, 23, 25, 26, 39].”

“There is a line of work [15, 8, 67, 26] that prepares a separate generator with an energy-based model.”

“In order to solve this drawback, we improve the discriminator by using the auto-encoder structure based discriminator DθD in [6].”

“There are indeed some other types of generative models that can be responsible for the generation of new wafer maps in imbalanced wafer classes, such as the original GAN [25], Energy-based GAN [26] and bidirectional GAN (BiGAN) [27].”

“To increase the diversity of the generated samples, (Energy-based GAN) EBGAN [75] views the training process as an energybased system where points near the data manifold are assigned low energy (high priority) and points from low data density are assigned high energy.”

“…to increase the diversity of generated samples by increasing generator entropy, via introducing a new loss term pull-away term (PT) first proposed in (Zhao et al., 2016): Additionally, in order for complement generator to work, the discriminator needs to have strong belief on fake-real on unlabeled…”

“The flexibility of GANs has also enabled various extensions, for instance to support structured prediction (Odena et al. 2016; Mirza and Osindero 2014), to train energy based models (Zhao et al. 2016) and combine adversarial loss with an information loss (Chen et al. 2016).”

“Inspired by [55], we adopt the Auto-Encoder to learn an energy model for translation, in which low energy is attributed to the data manifold that reflects the data distribution.”

“In this section, we will give a brief comparison between our proposed MBGAN with existing GANs, including GAN [9], EBGAN [44] and LSGAN [30].”

“Various improvements have been proposed ranging from tricks in architecture design to modifications in loss functions [28].”

“EBGAN [31] views the discriminator as an energy function, which allows to use many other architectures and loss functions in addition to the usual binary classifier with logistic output.”

“Previous attempts tried to avoid local optimum by improving adversarial learning objectives [18] [27] [4].”

“Energy-based GAN [7] and Boundary Equilibrium GAN [8] employ a pre-trained auto-encoder as the discriminator, learning to distinguish between real and generated samples via reconstruction.”

“Several techniques for stable training processes have been proposed, including DCGAN [27], energy-based GAN [28], Wasserstein GAN (WGAN) [29,30], WGAN-GP [31], BEGAN [32], and LSGAN [33].”

“mapping function from a latent space which follows a prior distribution to the data space. However, traditional GANs (DCGAN [21], LSGAN [19], BEGAN [2], WGAN [1], WGAN-GP [9], LaplacianGAN [4], EBGAN [23], among others) does not provide a means to learn the inverse mapping from the data space to the latent space. To s…”

“Several directions have attempted to stabilize the training of GAN and improve the generated sample diversity, including designing new network architectures [33, 49, 18], modifying the learning objectives and dynamics [37, 1, 27, 3, 50, 30], adding regularization methods [12, 28] and introducing heuristic tricks [36,…”

“, where actual CelebA pictures are positioned in the image space) and high energy values everywhere else [127].”

“nerative adversarial nets (GANs) [21] have made tremendous progress in this domain with different GAN models being used to generate synthetic face images with different pose [58,9,68], facial feature [52,10,60,67], age [20,3] and expression [39]. However, as pointed out by Karras et al. [33], GANs require plenty of tr…”

“Inorder to address this problem, we use a repelling regularizer proposed by Zhao et al. in their EBGAN autoencoder model [12].”

“[63] we argue in Section 5.4 that the GAN-loss acts as a regularizer preventing the encoder-generator from learning plain identity functions.”

“For the discriminator D, any deep learning model such as 1D convolutional neural networks or autoencoder can be used [35].”

“EBGAN [48] uses autoencoders as discriminators.”

“The multi-scale discriminator is optimized using only the hinge-based adversarial loss L 45 to distinguish between synthesized and real images.”

“Te generative adversarial network (GAN) [16], as a generative model, can generate new synthetic instances of data that follow rather similar, if not exactly the same, distribution of real samples through continuous confrontation between the generator and the discriminator.”

“G can be divided into two vectors and the first noise vector can be provided at the first layer, while the second noise [20] (c) BEGAN [11] / EBGAN [161] (d) Conditional GAN [93] (e) Pix2Pix [56] (f) Cycle GAN [164] and (g) LAPGAN [30].”

“In addition, Zhao et al. [51] introduced the Energy-based Generative Adversarial Networks (EBGAN) for human face synthesis, which treat the discriminator as an energy function that attributes low energies to regions near the data manifold and higher energies to other regions.”

“Several learning tricks and variants of the GAN have been proposed to overcome the stability issues of GAN, such as LapGAN [24], DCGAN [25], WGAN [26], cGAN [10], and EBGAN [32].”

“GANs have shown remarkable results in various computer vision tasks such as image generation [13][18][36][45], image translation [17][24][46], video translation [43], deblurring [27], segmentation [33], superresolution imaging [29], and face image synthesis [25][30][38].”

“We offer the following main contributions: Despite its youth, several publications ([1], [14], [16], [13]) have investigated the use of the GAN framework for sample generation and unsupervised feature learning.”

“The ultimate generator corresponds to a probability distribution p , which well approximates the training data distribution p with the existence of global optimal solution in optimizing objective (16) for p = , as proved in[45, 46].”

“There are different schools in diagnosing and addressing the problems with training GANs, that have resulted in a variety of algorithms, network architectures, training procedures, and novel objective functions [13, 14, 18, 2, 12].”

“erated images [27, 33, 19, 12] and to apply representations learned during training to discriminative, image tasks [28, 15].”

“Recently, the emergence of diffusion models has significantly contributed to the explosion of AI-generated content (AIGC) (Esser et al., 2021; Zhao, 2016; Rezende and Mohamed, 2015; Van Den Oord et al., 2016; Pa-pamakarios et al., 2017; Ramesh et al., 2021).”

“Zhao et al. [22] introduced the Energy-based GAN model (EBGAN) that takes the discriminator as an energy function that assigns low energies to the regions near the data manifold and higher energies to the others.”

“As a result, utilizing the discriminator in GANs learning process generally causes instability [7].”

“the BEGAN network is an autoencoder network based on the energy-based GAN (EBGAN) discriminator [31].”

“As for distance modeling, we adopt an energy-based discriminator [52] to estimate the distance between a transition and the transition manifold.”

“There are several efforts aimed at stabilizing and improving learning in generative adversarial networks (GANs) (Goodfellow et al., 2014; Salimans et al., 2016; Zhao et al., 2017; Arjovsky et al., 2017).”

“BEGAN [5] proposed an autoencoder based GAN, which adversarially minimizes an energy function [37] derived from Wasserstein distance [2].”

“GANs are successful implementations of deep generative models, and there are multiple variations such as WGAN [1], EBGAN [15], BEGAN [3], ACGAN [11], and DCGAN [13], which have evolved from the original GAN by altering the loss function and/or the network architecture.”

“[32] provide a similar method called repelling regularizer that punishes the cosine similarity of the feature in the same batch, forcing the features to be orthogonalized pairwise.”

“The generation model based methods start from deep generative network [32], and have received a great success with the development of generative adversarial network (GAN) [33], [34] and variational auto-encoder [35].”

“Numerous methods [26, 38] have been proposed to improve the stability of the GAN.”

“In recent years, generative adversarial networks (GANs) [2], [5], [17], [33], [34], [44] have been used as powerful tools to generate realistic images.”

“Improving GANs includes architecture modifications [8, 38, 40, 41, 53, 112], loss function design [3, 71, 119, 123] and regularization design [25, 44, 59, 65, 97].”

“1 INTRODUCTION Recent years have witnessed the development of generative adversarial networks (GANs) [5, 10, 14, 21, 22, 24, 35, 39, 42, 48, 50], which learn a function to generate high-dimension images from randomly sampled noises.”

“(2014; Zhao et al. 2017) expand the generative adversarial learning strategy to address different domains, such as image generation and style transfer tasks.”

“2) Generative adversarial networks GAN has obtained impressive results for image generation [27, 28], image editing [29], and representation learning [30].”

“and synthesizing tasks in many research fields [2], [3].”

“employed in EBGAN [49], where cosine similarity was used as a penalty to obtain orthogonal parameters.”

“Another direction is to consider the generation process as sampling from an Energy Based Model (Zhao et al., 2016).”

“The studies in [18,27] indicated that autoencoders can serve as substitutes for neural network structures in discrimination tasks.”

“The vision community is now concentrated on utilizing neural networks to encode mapping functions, as demonstrated in models such as generative adversarial networks (GANs) [9 – 12], variational autoenco-ders (VAEs) [7, 8], and autoregressive models [39].”

“GAN is a generative framework based on the deep learning models proposed in 2014 by Goodfellow et al. that can be applied to various generative tasks (Goodfellow et al. 2014; Zhao, Mathieu, and LeCun 2017).”

“Other work, such as Generative Multi-Adversarial Network (GMAN) (Durugkar et al., 2017), packing GANs (PacGAN) (Lin et al., 2017), GANs with spectrum control (Jiang et al., 2019), and energy-based GANs (Zhao et al., 2016), modifies the discriminator to achieve better stability.”

“EBGAN The architecture of the generator is similar to the one of the DCGAN generator.”

“GAN has been used in different fields, including unsupervised learning[20, 21], semi-supervised learning[22], fully supervised learning[23], and reinforcement learning[24].”

“Image Generation There are many generative networks [5, 13,38].”

“…quality of images generated by GAN, many researchers adopted various practical methods, such as modifying the loss function of discriminator network [21], [22], adding regularization term [16], [19], [23], [24], introducing attention mechanism [17], [25], and conducting progressive training [18].”

“To restore high-quality MR images from measurement, DAGAN adopted a conditional generative adversarial network (GAN) [17-19] model.”

“To overcome the problem of mode collapse, several researchers have proposed to penalize the appearance of nearduplicate images during each training batch using heuristicbased approaches [13], [21].”

“Generative Adversarial Network (GAN) [19] and its variants [3, 11, 80] have shown tremendous progress in high quality image generation.”

“…we can obtain IES operating data on various operating states such as normal operating status, fault status, and edge status, so as to study control strategies that can be used for more operating conditions (i.e., Quick response strategies can be generated in the event of a system failure).”