CoChat
Sign Up
PreprintCited by 76Topic ModelingGenerative Adversarial Networks and Image SynthesisNatural Language Processing Techniques

On Accurate Evaluation of GANs for Language Generation

Open access
Semeniuta, Stanislau, Severyn, Aliaksei, Gelly, Sylvain·2018·arXiv
DOI 10.48550/arxiv.1806.04936arXiv 1806.04936
Open this paper in CoChat — free accountOpen PDF

Abstract

Generative Adversarial Networks (GANs) are a promising approach to language generation. The latest works introducing novel GAN models for language generation use n-gram based metrics for evaluation and only report single scores of the best run. In this paper, we argue that this often misrepresents the true picture and does not tell the full story, as GAN models can be extremely sensitive to the random initialization and small deviations from the best hyperparameter choice. In particular, we demonstrate that the previously used BLEU score is not sensitive to semantic deterioration of generated texts and propose alternative metrics that better capture the quality and diversity of the generated samples. We also conduct a set of experiments comparing a number of GAN models for text with a conventional Language Model (LM) and find that neither of the considered models performs convincingly better than the LM.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1806.04936

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • GAN-generated text fails to outperform generations from language models when quality and diversity are considered jointly.

    “but recent work has shown that when quality and diversity are considered jointly, GAN-generated text fails to outperform generations from language models (Caccia et al., 2018;Tevet et al., 2019;Semeniuta et al., 2018).”

    From The Curious Case of Neural Text Degeneration (2019) · cites this paper as Semeniuta et al., 2018

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 31

  1. [1][1] Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron C. Courville, and Yoshua Bengio. An actor-critic algorithm for sequence prediction. CoRR , abs/1607.07086, 2016. URL http://arxiv.org/abs/1607.07086 . - · arXiv
  2. [2][2] Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. CoRR , abs/1506.03099, 2015. URL http://arxiv.org/abs/ 1506.03099 . - · arXiv
  3. [3][3] Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR , abs/1308.3432, 2013. URL http://arxiv.org/ abs/1308.3432 . - · arXiv
  4. [4][4] David Berthelot, Tom Schumm, and Luke Metz. BEGAN: boundary equilibrium generative adversarial networks. CoRR , abs/1703.10717, 2017. URL http://arxiv.org/abs/1703.10717 . - · arXiv
  5. [5][5] Mikolaj Binkowski, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD gans. CoRR , abs/1801.01401, 2018. URL http://arxiv.org/abs/1801.01401 . - · arXiv
  6. [6][6] Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP) . Association for Computational Linguistics, 2015. -
  7. [7][7] Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Józefowicz, and Samy Bengio. Generating sentences from a continuous space. CoRR , abs/1511.06349, 2015. URL http://arxiv.org/ abs/1511.06349 . - · arXiv
  8. [8][8] Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. Universal sentence encoder. CoRR , abs/1803.11175, 2018. URL http://arxiv.org/abs/1803.11175 . - · arXiv
  9. [9][9] Ondrej Cífka, Aliaksei Severyn, Enrique Alfonseca, and Katja Filippova. Eval all, trust a few, do wrong to none: Comparing sentence generation models. CoRR , abs/1804.07972, 2018. URL http: //arxiv.org/abs/1804.07972 . - · arXiv
  10. [10][10] Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. Supervised learning of universal sentence representations from natural language inference data. CoRR , abs/1705.02364, 2017. URL http://arxiv.org/abs/1705.02364 . - · arXiv
  11. [11][11] William Fedus, Ian J. Goodfellow, and Andrew M. Dai. Maskgan: Better text generation via filling in the \_\_\_\_\_\_. CoRR , abs/1801.07736, 2018. URL http://arxiv.org/abs/1801.07736 . - · arXiv
  12. [12][12] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial networks. CoRR , abs/1406.2661, 2014. URL http://arxiv.org/abs/1406.2661 . - · arXiv
  13. [14][14] Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang. Long text generation via adversarial training with leaked information. CoRR , abs/1709.08624, 2017. URL http://arxiv.org/ abs/1709.08624 . - · arXiv
  14. [15][15] Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zhirui Zhang, and Ming Zhou. Achieving human parity on automatic chinese to english news translation. CoRR , abs/1803.05567, 2018. URL http://arxiv.org/abs/1803.05567 . - · arXiv
  15. [17][17] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. CoRR , abs/1611.01144, 2016. URL http://arxiv.org/abs/1611.01144 . - · arXiv
  16. [18][18] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. CoRR , abs/1710.10196, 2017. URL http://arxiv.org/abs/1710. 10196 . - · arXiv
  17. [19][19] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR , abs/1412.6980, 2014. URL http://arxiv.org/abs/1412.6980 . - · arXiv
  18. [20][20] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Proc. ACL workshop on Text Summarization Branches Out , page 10, 2004. URL http://research.microsoft.com/~cyl/ download/papers/WAS2004.pdf . -
  19. [21][21] Sidi Lu, Yaoming Zhu, Weinan Zhang, Jun Wang, and Yong Yu. Neural text generation: Past, present and beyond. CoRR , abs/1803.07133, 2018. URL http://arxiv.org/abs/1803.07133 . - · arXiv
  20. [22][22] Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? A large-scale study. CoRR , abs/1711.10337, 2017. URL http://arxiv.org/abs/1711.10337 . - · arXiv
  21. [23][23] Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier gans. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017 , pages 2642-2651, 2017. URL http://proceedings.mlr. press/v70/odena17a.html . -
  22. [24][24] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA. , pages 311-318, 2002. URL http://www.aclweb. org/anthology/P02-1040.pdf . -
  23. [25][25] Scott E. Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. CoRR , abs/1605.05396, 2016. URL http://arxiv.org/ abs/1605.05396 . - · arXiv
  24. [26][26] Stanislau Semeniuta, Aliaksei Severyn, and Erhardt Barth. A hybrid convolutional variational autoencoder for text generation. CoRR , abs/1702.02390, 2017. URL http://arxiv.org/abs/1702.02390 . - · arXiv
  25. [27][27] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015 , pages 1-9, 2015. doi: 10.1109/CVPR.2015.7298594. URL https://doi.org/10.1109/CVPR.2015.7298594 . - · DOI · Paper page
  26. [28][28] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA , pages 6000-6010, 2017. URL http://papers.nips.cc/paper/ 7181-attention-is-all-you-need . -
  27. [29][29] Adina Williams, Nikita Nangia, and Samuel R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. CoRR , abs/1704.05426, 2017. URL http://arxiv.org/abs/1704. 05426 . - · arXiv
  28. [30][30] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. CoRR , abs/1609.08144, 2016. URL http://arxiv.org/abs/1609.08144 . - · arXiv
  29. [31][31] Jingjing Xu, Xu Sun, Xuancheng Ren, Junyang Lin, Bingzhen Wei, and Wei Li. DP-GAN: diversitypromoting generative adversarial network for generating informative and diversified text. CoRR , abs/1802.01345, 2018. URL http://arxiv.org/abs/1802.01345 . - · arXiv
  30. [32][32] Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. Seqgan: Sequence generative adversarial nets with policy gradient. CoRR , abs/1609.05473, 2016. URL http://arxiv.org/abs/1609.05473 . - · arXiv
  31. [33][33] Junbo Jake Zhao, Yoon Kim, Kelly Zhang, Alexander M. Rush, and Yann LeCun. Adversarially regularized autoencoders for generating discrete structures. CoRR , abs/1706.04223, 2017. URL http://arxiv.org/ abs/1706.04223 . · arXiv

Cited by

Sorted by citations

Authors
Semeniuta, S., Severyn, A., & Gelly, S.
Affiliations
  • University of Lübeck (DE)
  • Google (United States) (US)
ORCID
Severyn, Aliaksei
Venue
arXiv
Publisher
arXiv
Published
2018
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1806.04936
arXiv
1806.04936
OpenAlex
W2807747378
Cited by
76
References
2
Citations by year
2025: 1 · 2024: 5 · 2023: 5 · 2022: 5 · 2021: 20 · 2020: 21 · 2019: 15 · 2018: 4
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
generative adversarial networks, language generation, GAN evaluation metrics, hyperparameter sensitivity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review