CoChat
Sign Up
PreprintCited by 56Generative Adversarial Networks and Image SynthesisStochastic Gradient Optimization TechniquesAdvanced Neural Network Applications

Small-GAN: Speeding Up GAN Training Using Core-sets

Open access
Sinha, Samarth, Zhang, Han, Goyal, Anirudh, Bengio, Yoshua, Larochelle, Hugo, Odena, Augustus·2019·arXiv
DOI 10.48550/arxiv.1910.13540arXiv 1910.13540
Open this paper in CoChat — free accountOpen PDF

Abstract

Recent work by Brock et al. (2018) suggests that Generative Adversarial Networks (GANs) benefit disproportionately from large mini-batch sizes. Unfortunately, using large batches is slow and expensive on conventional hardware. Thus, it would be nice if we could generate batches that were effectively large though actually small. In this work, we propose a method to do this, inspired by the use of Coreset-selection in active learning. When training a GAN, we draw a large batch of samples from the prior and then compress that batch using Coreset-selection. To create effectively large batches of 'real' images, we create a cached dataset of Inception activations of each training image, randomly project them down to a smaller dimension, and then use Coreset-selection on those projected activations at training time. We conduct experiments showing that this technique substantially reduces training time and memory usage for modern GAN variants, that it reduces the fraction of dropped modes in a synthetic dataset, and that it allows GANs to reach a new state of the art in anomaly detection.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1910.13540

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • PatchCore uses minimax facility location coreset selection to ensure approximately similar coverage of the M-coreset in patch-level feature space as compared to the original memory bank.

    “Because PatchCore uses nearest neighbour computations (next Section), we use a minimax facility location coreset selection, see e.g., [48] and [49], to ensure approximately similar coverage of the M -coreset M C in patch-level feature space as compared to the original memory bank M”

    From Towards Total Recall in Industrial Anomaly Detection (2022) · cites this paper as [49]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

Cited by 56

Research works printed but not linked: 67

  1. Martin Arjovsky, Soumith Chintala, and L´ eon Bottou. Wasserstein gan. arXiv preprint arXiv:1701.07875 , 2017. · arXiv
  2. Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. Generalization and equilibrium in generative adversarial nets (gans). In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pp. 224-232. JMLR. org, 2017.
  3. Samaneh Azadi, Catherine Olsson, Trevor Darrell, Ian Goodfellow, and Augustus Odena. Discriminator rejection sampling. arXiv preprint arXiv:1810.06758 , 2018. · arXiv
  4. Mihai B¯ adoiu, Sariel Har-Peled, and Piotr Indyk. Approximate clustering via core-sets. In Proceedings of the thiry-fourth annual ACM symposium on Theory of computing , pp. 250-257. ACM, 2002.
  5. Francisco Barahona and Fabi´ aN A Chudak. Near-optimal solutions to large-scale facility location problems. Discrete Optimization , 2(1):35-50, 2005.
  6. Marc G Bellemare, Ivo Danihelka, Will Dabney, Shakir Mohamed, Balaji Lakshminarayanan, Stephan Hoyer, and R´ emi Munos. The cramer distance as a solution to biased wasserstein gradients. arXiv preprint arXiv:1705.10743 , 2017. · arXiv
  7. Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096 , 2018. · arXiv
  8. Varun Chandola, Arindam Banerjee, and Vipin Kumar. Anomaly detection: A survey. ACM computing surveys (CSUR) , 41(3):15, 2009.
  9. Tatjana Chavdarova and Franc ¸ois Fleuret. Sgan: An alternative training of generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 9407-9415, 2018.
  10. Tatjana Chavdarova, Gauthier Gidel, Franc ¸ois Fleuret, and Simon Lacoste-Julien. Reducing noise in gan training with variance reduced extragradient. arXiv preprint arXiv:1904.08598 , 2019. · arXiv
  11. Kenneth L Clarkson. Coresets, sparse greedy approximation, and the frank-wolfe algorithm. ACM Transactions on Algorithms (TALG) , 6(4):63, 2010.
  12. Sanjoy Dasgupta and Anupam Gupta. An elementary proof of a theorem of johnson and lindenstrauss. Random Structures & Algorithms , 22(1):60-65, 2003.
  13. David L Donoho et al. High-dimensional data analysis: The curses and blessings of dimensionality. AMS math challenges lecture , 1(2000):32, 2000.
  14. Ishan Durugkar, Ian Gemp, and Sridhar Mahadevan. Generative multi-adversarial networks. arXiv preprint arXiv:1611.01673 , 2016. · arXiv
  15. Ahmet M Eskicioglu and Paul S Fisher. Image quality measures and their performance. IEEE Transactions on communications , 43(12):2959-2965, 1995.
  16. Reza Zanjirani Farahani and Masoud Hekmatfar. Facility location: concepts, models, algorithms and case studies . Springer, 2009.
  17. William Fedus, Ian Goodfellow, and Andrew M Dai. Maskgan: better text generation via filling in the . arXiv preprint arXiv:1801.07736 , 2018. · arXiv
  18. Dan Feldman, Matthew Faulkner, and Andreas Krause. Scalable training of mixture models via coresets. In Advances in neural information processing systems , pp. 2142-2150, 2011.
  19. Gauthier Gidel, Hugo Berard, Ga¨ etan Vignoud, Pascal Vincent, and Simon Lacoste-Julien. A variational inequality perspective on generative adversarial networks. arXiv preprint arXiv:1802.10551 , 2018. · arXiv
  20. Bernd Girod. What's wrong with mean-squared error? Digital images and human vision , pp. 207-220, 1993.
  21. AJ Goldman. Optimal center location in simple networks. Transportation science , 5(2):212-221, 1971.
  22. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems , pp. 2672-2680, 2014.
  23. Priya Goyal, Piotr Doll´ ar, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677 , 2017. · arXiv
  24. Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. In Advances in neural information processing systems , pp. 5767-5777, 2017.
  25. Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang. Long text generation via adversarial training with leaked information. In Thirty-Second AAAI Conference on Artificial Intelligence , 2018.
  26. Sariel Har-Peled and Akash Kushal. Smaller coresets for k-median and k-means clustering. Discrete &Computational Geometry , 37(1):3-19, 2007.
  27. Sariel Har-Peled and Soham Mazumdar. On coresets for k-means and k-median clustering. In Proceedings of the thirty-sixth annual ACM symposium on Theory of computing , pp. 291-300. ACM, 2004.
  28. Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems , pp. 6626-6637, 2017.
  29. Jonathan Huggins, Trevor Campbell, and Tamara Broderick. Coresets for scalable bayesian logistic regression. In Advances in Neural Information Processing Systems , pp. 4080-4088, 2016.
  30. Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 1125-1134, 2017.
  31. Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836 , 2016. · arXiv
  32. Rithesh Kumar, Anirudh Goyal, Aaron Courville, and Yoshua Bengio. Maximum entropy generators for energy-based models. arXiv preprint arXiv:1901.08508 , 2019. · arXiv
  33. Donghwoon Kwon, Hyunjoo Kim, Jinoh Kim, Sang C Suh, Ikkyun Kim, and Kuinam J Kim. A survey of deep learning-based network anomaly detection. Cluster Computing , pp. 1-13, 2017.
  34. Christian Ledig, Lucas Theis, Ferenc Husz´ ar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 4681-4690, 2017.
  35. Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnab´ as P´ oczos. Mmd gan: Towards deeper understanding of moment matching network. In Advances in Neural Information Processing Systems , pp. 2203-2213, 2017a.
  36. Jerry Li, Aleksander Madry, John Peebles, and Ludwig Schmidt. Towards understanding the dynamics of generative adversarial networks. arXiv preprint arXiv:1706.09884 , 2017b. · arXiv
  37. Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision , pp. 2794-2802, 2017.
  38. Lars Mescheder. On the convergence properties of gan training. arXiv preprint arXiv:1801.04406 , 1:16, 2018. · arXiv
  39. Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? arXiv preprint arXiv:1801.04406 , 2018. · arXiv
  40. Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957 , 2018. · arXiv
  41. Youssef Mroueh and Tom Sercu. Fisher gan. In Advances in Neural Information Processing Systems , pp. 2513-2523, 2017.
  42. Ben Mussay, Samson Zhou, Vladimir Braverman, and Dan Feldman. On activation function coresets for network pruning. arXiv preprint arXiv:1907.04018 , 2019. · arXiv
  43. Vaishnavh Nagarajan and J Zico Kolter. Gradient descent gan optimization is locally stable. In Advances in Neural Information Processing Systems , pp. 5585-5595, 2017.
  44. Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard E Turner. Variational continual learning. arXiv preprint arXiv:1710.10628 , 2017. · arXiv
  45. Augustus Odena. Open questions about generative adversarial networks. Distill , 2019. doi: 10. 23915/distill.00018. https://distill.pub/2019/gan-open-problems.
  46. Catherine Olsson, Surya Bhupatiraju, Tom Brown, Augustus Odena, and Ian Goodfellow. Skill rating for generative models. arXiv preprint arXiv:1808.04888 , 2018. · arXiv
  47. Jeff M Phillips. Coresets and sketches. arXiv preprint arXiv:1601.00617 , 2016. · arXiv
  48. Rameshwar Pratap and Sandeep Sen. Faster coreset construction for projective clustering via lowrank approximation. In International Workshop on Combinatorial Algorithms , pp. 336-348. Springer, 2018.
  49. Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 , 2015. · arXiv
  50. Tim Salimans, Han Zhang, Alec Radford, and Dimitris Metaxas. Improving gans using optimal transport. arXiv preprint arXiv:1803.05573 , 2018. · arXiv
  51. Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv:1708.00489 , 2017. · arXiv
  52. Christopher J Shallue, Jaehoon Lee, Joe Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E Dahl. Measuring the effects of data parallelism on neural network training. arXiv preprint arXiv:1811.03600 , 2018. · arXiv
  53. Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning. arXiv preprint arXiv:1904.00370 , 2019. · arXiv
  54. Samuel L Smith, Pieter-Jan Kindermans, Chris Ying, and Quoc V Le. Don't decay the learning rate, increase the batch size. arXiv preprint arXiv:1711.00489 , 2017. · arXiv
  55. Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In Thirty-First AAAI Conference on Artificial Intelligence , 2017.
  56. Ivor W Tsang, James T Kwok, and Pak-Ming Cheung. Core vector machines: Fast svm training on very large data sets. Journal of Machine Learning Research , 6(Apr):363-392, 2005.
  57. Ivor W Tsang, Andras Kocsor, and James T Kwok. Simpler core vector machines with enclosing balls. In Proceedings of the 24th international conference on Machine learning , pp. 911-918. ACM, 2007.
  58. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems , pp. 5998-6008, 2017.
  59. Kai Wei, Yuzong Liu, Katrin Kirchhoff, and Jeff Bilmes. Using document summarization techniques for speech data subset selection. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pp. 721-726, 2013.
  60. Laurence A Wolsey and George L Nemhauser. Integer and combinatorial optimization . John Wiley &Sons, 2014.
  61. Yongqin Xian, Tobias Lorenz, Bernt Schiele, and Zeynep Akata. Feature generating networks for zero-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 5542-5551, 2018.
  62. Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances in neural information processing systems , pp. 3320-3328, 2014.
  63. Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365 , 2015. · arXiv
  64. Houssam Zenati, Chuan Sheng Foo, Bruno Lecouat, Gaurav Manek, and Vijay Ramaseshan Chandrasekhar. Efficient gan-based anomaly detection. arXiv preprint arXiv:1802.06222 , 2018. · arXiv
  65. Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision , pp. 5907-5915, 2017.
  66. Jia-Jie Zhu and Jos´ e Bento. Generative adversarial active learning. arXiv preprint arXiv:1702.07956 , 2017. · arXiv
  67. Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision , pp. 2223-2232, 2017.

Authors
Sinha, S., Zhang, H., Goyal, A., Bengio, Y., Larochelle, H., & Odena, A.
Affiliations
  • Google (United States) (US)
  • Université de Montréal (CA)
  • Mila - Quebec Artificial Intelligence Institute (CA)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2019
Volume
1
Pages
9005-9015
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1910.13540
arXiv
1910.13540
OpenAlex
W3035353183
Cited by
56
References
0
Citations by year
2025: 7 · 2024: 16 · 2023: 7 · 2022: 9 · 2021: 10 · 2020: 7
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • International Conference on Machine Learning
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
generative adversarial networks, coreset selection, random projections, mode collapse, anomaly detection, active learning (machine learning)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review