CoChat
Sign Up
Cited by 106Topic ModelingNatural Language Processing TechniquesDomain Adaptation and Few-Shot Learning

The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.

The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers

Open access
Csordás, Róbert, Irie, Kazuki, Schmidhuber, Juergen·2021·Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
DOI 10.18653/v1/2021.emnlp-main.49
Open this paper in CoChat — free accountOpen PDF

Abstract

Recently, many datasets have been proposed to test the systematic generalization ability of neural networks.The companion baseline Transformers, typically trained with default hyper-parameters from standard tasks, are shown to fail dramatically.Here we demonstrate that by revisiting model configurations as basic as scaling of embeddings, early stopping, relative positional embedding, and Universal Transformer variants, we can drastically improve the performance of Transformers on systematic generalization.We report improvements on five popular datasets: SCAN, CFQ, PCFG, COGS, and Mathematics dataset.Our models improve accuracy from 50% to 85% on the PCFG productivity split, and from 35% to 81% on COGS.On SCAN, relative positional embedding largely mitigates the EOS decision problem (Newman et al., 2020), yielding 100% accuracy on the length split with a cutoff at 26. Importantly, performance differences between these models are typically invisible on the IID data split.This calls for proper generalization validation sets for developing neural networks that generalize systematically.We publicly release the code to reproduce our results 1 .

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2021.emnlp-main.49

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • For COGS, both models were optimized for 300 epochs without early stopping, which is slightly more training than the extended amount prescribed in ref. 67 for their strong seq2seq baseline.

    “For COGS, both models were optimized for 300 epochs (also without early stopping), which is slightly more training than the extended amount prescribed in ref. 67 for their strong seq2seq baseline.”

    From Human-like systematic generalization through a meta-learning neural network (2023) · cites this paper as [67]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 48

  1. Jacob Andreas. 2020. Good-enough compositional data augmentation. In Proc. Association for Com- putational Linguistics (ACL), pages 7556-7566, Vir- tual only.
  2. Dzmitry Bahdanau, Harm de Vries, Timothy J O'Donnell, Shikhar Murty, Philippe Beaudoin, Yoshua Bengio, and Aaron Courville. 2019. CLO- SURE: Assessing systematic generalization of CLEVR models. In ViGIL workshop, NeurIPS, Van- couver, Canada.
  3. Francois Charton, Amaury Hayat, and Guillaume Lam- ple. 2021. Learning advanced mathematical compu- tations from examples. In Int. Conf. on Learning Representations (ICLR), Virtual only.
  4. Xinyun Chen, Chen Liang, Adams Wei Yu, Dawn Song, and Denny Zhou. 2020. Compositional gen- eralization via neural-symbolic stack machines. In Proc. Advances in Neural Information Processing Systems (NeurIPS), Virtual only.
  5. Róbert Csordás and Jürgen Schmidhuber. 2019. Im- proving differentiable neural computers through memory masking, de-allocation, and link distribu- tion sharpness control. In Int. Conf. on Learning Representations (ICLR), New Orleans, LA, USA.
  6. Róbert Csordás, Sjoerd van Steenkiste, and Jürgen Schmidhuber. 2021. Are neural nets modular? inspecting functional modularity through differen- tiable weight masks. In Int. Conf. on Learning Rep- resentations (ICLR), Virtual only.
  7. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Car- bonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. In Proc. Association for Computational Linguistics (ACL), pages 2978-2988, Florence, Italy.
  8. Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. 2019. Univer- sal transformers. In Int. Conf. on Learning Repre- sentations (ICLR), New Orleans, LA, USA.
  9. Jerry Fodor and Brian P McLaughlin. 1990. Con- nectionism and the problem of systematicity: Why smolensky's solution doesn't work. Cognition, 35(2):183-204.
  10. Jerry A Fodor, Zenon W Pylyshyn, et al. 1988. Connec- tionism and cognitive architecture: A critical analy- sis. Cognition, 28(1-2):3-71.
  11. Karlis Freivalds, Emils Ozolins, and Agris Sostaks. 2019. Neural shuffle-exchange networks -sequence processing in o(n log n) time. In Proc. Advances in Neural Information Processing Systems (NeurIPS), Vancouver, Canada.
  12. Daniel Furrer, Marc van Zee, Nathan Scales, and Nathanael Schärli. 2020. Compositional generaliza- tion in semantic parsing: Pre-training vs. specialized architectures. Preprint arXiv:2007.08970. · arXiv
  13. Xavier Glorot and Yoshua Bengio. 2010. Understand- ing the difficulty of training deep feedforward neu- ral networks. In Proc. Int. Conf. on Artificial Intelli- gence and Statistics (AISTATS), pages 249-256, Sar- dinia, Italy.
  14. Jonathan Gordon, David Lopez-Paz, Marco Baroni, and Diane Bouchacourt. 2020. Permutation equiv- ariant models for compositional generalization in language. In Int. Conf. on Learning Representations (ICLR), Addis Ababa, Ethiopia.
  15. Alex Graves. 2012. Sequence transduction with recur- rent neural networks. In Workshop on Representa- tion Learning, ICML, Edinburgh, Scotland.
  16. Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska- Barwinska, Sergio Gomez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John P. Agapiou, Adrià Puigdomènech Badia, Karl Moritz Hermann, Yori Zwols, Georg Ostrovski, Adam Cain, Helen King, Christopher Summerfield, Phil Blunsom, Ko- ray Kavukcuoglu, and Demis Hassabis. 2016. Hy- brid computing using a neural network with dy- namic external memory. Nature, 538(7626):471- 476.
  17. Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmid- huber. 2020. On the binding problem in artificial neural networks. Preprint arXiv:2012.05208. · arXiv
  18. Yinuo Guo, Zeqi Lin, Jian-Guang Lou, and Dongmei Zhang. 2020. Hierarchical poset decoding for com- positional generalization in language. In Proc. Ad- vances in Neural Information Processing Systems (NeurIPS), Virtual only.
  19. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proc. IEEE Int. Conf. on Computer Vision (ICCV), pages 1026-1034, Santiago, Chile.
  20. Jonathan Herzig and Jonathan Berant. 2020. Span- based semantic parsing for compositional general- ization. Preprint arXiv:2009.06040. · arXiv
  21. Jonathan Herzig, Peter Shaw, Ming-Wei Chang, Kelvin Guu, Panupong Pasupat, and Yuan Zhang. 2021. Un- locking compositional generalization in pre-trained models using intermediate representations. Preprint arXiv:2104.07478. · arXiv
  22. Xiao Shi Huang, Felipe Perez, Jimmy Ba, and Mak- sims Volkovs. 2020. Improving transformer opti- mization through better initialization. In Proc. Int. Conf. on Machine Learning (ICML), pages 4475- 4483, Virtual only.
  23. Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. 2020. Compositionality decomposed: How do neural networks generalise? Journal of Ar- tificial Intelligence Research, pages 757-795.
  24. Kazuki Irie, Imanol Schlag, Róbert Csordás, and Jür- gen Schmidhuber. 2021. Going beyond linear trans- formers with recurrent fast weight programmers. Preprint arXiv:2106.06295. · arXiv
  25. Lukasz Kaiser and Ilya Sutskever. 2016. Neural GPUs learn algorithms. In Int. Conf. on Learning Repre- sentations (ICLR), San Juan, Puerto Rico.
  26. Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, and Olivier Bousquet. 2020. Measur- ing compositional generalization: A comprehensive method on realistic data. In Int. Conf. on Learning Representations (ICLR), Addis Ababa, Ethiopia.
  27. Najoung Kim and Tal Linzen. 2020. COGS: A compo- sitional generalization challenge based on semantic interpretation. In Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLP), pages 9087-9105, Virtual only.
  28. Kris Korrel, Dieuwke Hupkes, Verna Dankers, and Elia Bruni. 2019. Transcoding compositionally: Using attention to find more generalizable solutions. In Proc. BlackboxNLP Workshop on Analyzing and In- terpreting Neural Networks for NLP, ACL, pages 1- 11, Florence, Italy.
  29. Brenden M Lake. 2019. Compositional generalization through meta sequence-to-sequence learning. In Proc. Advances in Neural Information Processing Systems (NeurIPS), pages 9788-9798, Vancouver, Canada.
  30. Brenden M. Lake and Marco Baroni. 2018. General- ization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In Proc. Int. Conf. on Machine Learning (ICML), pages 2873-2882, Stockholm, Sweden.
  31. Yuanpeng Li, Liang Zhao, Jianyu Wang, and Joel Hest- ness. 2019. Compositional generalization for primi- tive substitutions. In Proc. Conf. on Empirical Meth- ods in Natural Language Processing and Int.Joint Conf. on Natural Language Processing (EMNLP- IJCNLP), pages 4292-4301, Hong Kong, China.
  32. Adam Liska, Germán Kruszewski, and Marco Baroni. 2018. Memorize or generalize? searching for a com- positional RNN in a haystack. In AEGAP Workshop ICML, Stockholm, Sweden.
  33. Qian Liu, Shengnan An, Jian-Guang Lou, Bei Chen, Zeqi Lin, Yan Gao, Bin Zhou, Nanning Zheng, and Dongmei Zhang. 2020. Compositional generaliza- tion by learning analytical expressions. In Proc. Ad- vances in Neural Information Processing Systems (NeurIPS), Virtual only.
  34. Benjamin Newman, John Hewitt, Percy Liang, and Christopher D Manning. 2020. The eos decision and length extrapolation. In Proc. BlackboxNLP Workshop on Analyzing and Interpreting Neural Net- works for NLP, EMNLP, pages 276-291, Virtual only.
  35. Maxwell I Nye, Armando Solar-Lezama, Joshua B Tenenbaum, and Brenden M Lake. 2020. Learn- ing compositional rules via neural program synthe- sis. Preprint arXiv:2003.05562. · arXiv
  36. Santiago Ontañón, Joshua Ainslie, Vaclav Cvicek, and Zachary Fisher. 2021. Making transformers solve compositional tasks. Preprint arXiv:2108.04378. · arXiv
  37. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learn- ing library. In Proc. Advances in Neural Infor- mation Processing Systems (NeurIPS), pages 8024- 8035, Vancouver, Canada.
  38. Rebecca Roelofs. 2019. Measuring Generalization and overfitting in Machine learning. Ph.D. thesis, UC Berkeley.
  39. Jake Russin, Jason Jo, Randall C O'Reilly, and Yoshua Bengio. 2019. Compositional generalization in a deep seq2seq model by separating syntax and seman- tics. Preprint arXiv:1904.09708. · arXiv
  40. David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. 2019. Analysing mathematical rea- soning abilities of neural models. In Int. Conf. on Learning Representations (ICLR), New Orleans, LA, USA.
  41. Imanol Schlag, Paul Smolensky, Roland Fernandez, Nebojsa Jojic, Jürgen Schmidhuber, and Jianfeng Gao. 2019. Enhancing the transformer with ex- plicit relational encoding for math problem solving. Preprint arXiv:1910.06611. · arXiv
  42. Jürgen Schmidhuber. 1992. Learning to control fast- weight memories: An alternative to recurrent nets. Neural Computation, 4(1):131-139.
  43. Jürgen Schmidhuber. 2012. Self-delimiting neural net- works. Preprint arXiv:1210.0118. · arXiv
  44. Peter Shaw, Ming-Wei Chang, Panupong Pasupat, and Kristina Toutanova. 2020. Compositional general- ization and natural language variation: Can a se- mantic parsing approach handle both? Preprint arXiv:2010.12725. · arXiv
  45. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Proc. Advances in Neural Information Process- ing Systems (NIPS), pages 3104-3112, Montréal, Canada.
  46. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proc. Advances in Neural Information Processing Systems (NIPS), pages 5998-6008, Long Beach, CA, USA.
  47. Biao Zhang, Ivan Titov, and Rico Sennrich. 2019. Im- proving deep transformer with depth-scaled initial- ization and merged attention. In Proc. Conf. on Empirical Methods in Natural Language Processing and Int.Joint Conf. on Natural Language Process- ing (EMNLP-IJCNLP), pages 898-909, Hong Kong, China.
  48. Chen Zhu, Renkun Ni, Zheng Xu, Kezhi Kong, W Ronny Huang, and Tom Goldstein. 2021. Gradinit: Learning to initialize neural networks for stable and efficient training. Preprint arXiv:2102.08098. · arXiv

Cited by

Sorted by citations

Metadata

Authors
Csordás, R., Irie, K., & Schmidhuber, J.
Affiliations
University of Applied Sciences and Arts of Southern Switzerland (CH)
Corresponding author
Irie, Kazuki
Venue
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Publisher
Association for Computational Linguistics
Published
2021
Pages
619-634
DOI
10.18653/v1/2021.emnlp-main.49
OpenAlex
W3197009789
Cited by
106
References
5
Citations by year
2026: 10 · 2025: 18 · 2024: 14 · 2023: 28 · 2022: 24 · 2021: 11
FWCI
7.24
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation
  • Nvidia
  • Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung
  • H2020 European Research Council (742870)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
systematic generalization, transformer models, early stopping, COG, SCAN, hyperparameter optimization

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review