Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
PreprintCited by 0

The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.

Eureka-Moments in Transformers: Multi-Step Tasks Reveal Softmax Induced Optimization Problems

Hoffmann, David T., Schrodi, Simon, Bratulić, Jelena, Behrmann, Nadine, Fischer, Volker, Brox, Thomas·2023·arXiv
DOI 10.48550/arxiv.2310.12956arXiv 2310.12956
Open this paper in CoChat — free account

Abstract

In this work, we study rapid improvements of the training loss in transformers when being confronted with multi-step decision tasks. We found that transformers struggle to learn the intermediate task and both training and validation loss saturate for hundreds of epochs. When transformers finally learn the intermediate task, they do this rapidly and unexpectedly. We call these abrupt improvements Eureka-moments, since the transformer appears to suddenly learn a previously incomprehensible concept. We designed synthetic tasks to study the problem in detail, but the leaps in performance can be observed also for language modeling and in-context learning (ICL). We suspect that these abrupt transitions are caused by the multi-step nature of these tasks. Indeed, we find connections and show that ways to improve on the synthetic multi-step tasks can be used to improve the training of language modeling and ICL. Using the synthetic data we trace the problem back to the Softmax function in the self-attention block of transformers and show ways to alleviate the problem. These fixes reduce the required number of training steps, lead to higher likelihood to learn the intermediate task, to higher final accuracy and training becomes more robust to hyper-parameters.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2310.12956

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 43

  1. Escaping the plateau: Dynamic context length adaptation for efficient bert pretraining. 2023. URL https: //openreview.net/forum?id=urzQrPofiMM. Anonymous review.
  2. Ali, A., Touvron, H., Caron, M., Bojanowski, P., Douze, M., Joulin, A., Laptev, I., Neverova, N., Synnaeve, G., Ver- beek, J., et al. Xcit: Cross-covariance image transformers. Advances in neural information processing systems, 34: 20014-20027, 2021.
  3. Baevski, A. and Auli, M. Adaptive input representations for neural language modeling. In International Conference on Learning Representations, 2019. URL https:// openreview.net/forum?id=ByxZX20qFQ.
  4. Barak, B., Edelman, B., Goel, S., Kakade, S., Malach, E., and Zhang, C. Hidden progress in deep learning: Sgd learns parities near the computational limit. Advances in Neural Information Processing Systems, 2022.
  5. Chan, S., Santoro, A., Lampinen, A., Wang, J., Singh, A., Richemond, P., McClelland, J., and Hill, F. Data distri- butional properties drive emergent in-context learning in transformers. Advances in Neural Information Processing Systems, 35:18878-18891, 2022.
  6. Chen, A., Shwartz-Ziv, R., Cho, K., Leavitt, M. L., and Saphra, N. Sudden drops in the loss: Syntax acquisition, phase transitions, and simplicity bias in MLMs. In The Twelfth International Conference on Learning Represen- tations, 2024. URL https://openreview.net/ forum?id=MO5PiKHELW.
  7. Chen, X., Hu, Q., Li, K., Zhong, C., and Wang, G. Accu- mulated trivial attention matters in vision transformers on small datasets. In Proceedings of the IEEE/CVF Win- ter Conference on Applications of Computer Vision, pp. 3984-3992, 2023.
  8. Child, R., Gray, S., Radford, A., and Sutskever, I. Gen- erating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019. · arXiv
  9. Dehghani, M., Djolonga, J., Mustafa, B., Padlewski, P., Heek, J., Gilmer, J., Steiner, A. P., Caron, M., Geirhos, R., Alabdulmohsin, I., et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning, pp. 7480-7512. PMLR, 2023.
  10. Deshpande, A. and Narasimhan, K. Guiding attention for self-supervised learning with transformers. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4676-4686, 2020.
  11. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. In North American Chapter of the Association for Computational Linguistics, 2019.
  12. Dong, Y., Cordonnier, J.-B., and Loukas, A. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In International Conference on Machine Learning, pp. 2793-2803. PMLR, 2021.
  13. Dufter, P., Schmitt, M., and Schütze, H. Increasing learning efficiency of self-attention networks through direct posi- tion interactions, learnable temperature, and convoluted attention. In Proceedings of the 28th International Con- ference on Computational Linguistics, pp. 3630-3636, December 2020. URL https://aclanthology. org/2020.coling-main.324.
  14. Gupta, A. and Berant, J. Gmat: Global memory augmenta- tion for transformers. arXiv preprint arXiv:2006.03274, 2020. · arXiv
  15. Hassani, A., Walton, S., Shah, N., Abuduweili, A., Li, J., and Shi, H. Escaping the big data paradigm with compact transformers. arXiv preprint arXiv:2104.05704, 2021. · arXiv
  16. Huang, X. S., Perez, F., Ba, J., and Volkovs, M. Improving transformer optimization through better initialization. In International Conference on Machine Learning, pp. 4475- 4483. PMLR, 2020.
  17. Jiang, Z., Gu, J., and Pan, D. Z. Normsoftmax: Normalizing the input of softmax to accelerate and stabilize training. In 2023 IEEE International Conference on Omni-layer Intelligent Systems (COINS), pp. 1-6. IEEE, 2023.
  18. Lake, B. M., Salakhutdinov, R., and Tenenbaum, J. B. Human-level concept learning through probabilistic pro- gram induction. Science, 350(6266):1332-1338, 2015.
  19. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. · arXiv
  20. Liu, Z., Kitouni, O., Nolte, N. S., Michaud, E., Tegmark, M., and Williams, M. Towards understanding grokking: An effective theory of representation learning. In Advances in Neural Information Processing Systems, volume 35, pp. 34651-34663, 2022.
  21. Liu, Z., Michaud, E. J., and Tegmark, M. Omnigrok: Grokking beyond algorithmic data. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum? id=zDiHoIWa0q1.
  22. Loshchilov, I. and Hutter, F. Decoupled weight decay reg- ularization. In International Conference on Learning Representations, 2019. URL https://openreview. net/forum?id=Bkg6RiCqY7.
  23. Nagatsuka, K., Broni-Bediako, C., and Atsumi, M. Pre- training a bert with curriculum learning by increasing block-size of input text. In Proceedings of the Interna- tional Conference on Recent Advances in Natural Lan- guage Processing (RANLP 2021), pp. 989-996, 2021.
  24. Nanda, N., Chan, L., Liberum, T., Smith, J., and Stein- hardt, J. Progress measures for grokking via mechanistic interpretability. International Conference on Learning Representations, 2023.
  25. Noci, L., Anagnostidis, S., Biggio, L., Orvieto, A., Singh, S. P., and Lucchi, A. Signal propagation in transformers: Theoretical perspectives and the role of rank collapse. In Advances in Neural Information Processing Systems, volume 35, pp. 27198-27211, 2022.
  26. Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895, 2022. · arXiv
  27. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
  28. Reddy, G. The mechanistic basis of data dependence and abrupt learning in an in-context classification task. In The Twelfth International Conference on Learning Represen- tations, 2023.
  29. Schaeffer, R., Miranda, B., and Koyejo, S. Are emergent abilities of large language models a mirage? In Advances in Neural Information Processing Systems, volume 36, 2024.
  30. Shen, K., Guo, J., Tan, X., Tang, S., Wang, R., and Bian, J. A study on relu and softmax in transformer. arXiv preprint arXiv:2302.06461, 2023. · arXiv
  31. Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-lm: Training multi- billion parameter language models using model paral- lelism. arXiv preprint arXiv:1909.08053, 2019. · arXiv
  32. Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learn- ing Research, 2023. ISSN 2835-8856. URL https: //openreview.net/forum?id=uyTL5Bvosj.
  33. Thilak, V., Littwin, E., Zhai, S., Saremi, O., Paiss, R., and Susskind, J. M. The slingshot mechanism: An empiri- cal study of adaptive optimizers and the grokking phe- nomenon. In Has it Trained Yet? NeurIPS 2022 Work- shop, 2022.
  34. Tian, Y., Krishnan, D., and Isola, P. Contrastive multiview coding. In European Conference on Computer Vision, pp. 776-794, 2020.
  35. Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jegou, H. Training data-efficient image transform- ers & distillation through attention. In International Con- ference on Machine Learning, volume 139, pp. 10347- 10357, July 2021.
  36. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
  37. Voita, E. and Titov, I. Information-theoretic probing with minimum description length. In Webber, B., Cohn, T., He, Y., and Liu, Y. (eds.), Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Pro- cessing (EMNLP), pp. 183-196, 2020. URL https: //aclanthology.org/2020.emnlp-main.14. Wang, Q., Li, B., Xiao, T., Zhu, J., Li, C., Wong, D. F., and Chao, L. S. Learning deep transformer models for machine translation. In Korhonen, A., Traum, D., and Màrquez, L. (eds.), Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pp. 1810-1822, 2019. URL https://aclanthology. org/P19-1176.
  38. Wang, S., Liu, F., and Liu, B. Escaping the gradient vanish- ing: Periodic alternatives of softmax in attention mecha- nism. IEEE Access, 9:168749-168759, 2021.
  39. Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Met- zler, D., Chi, E. H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., and Fedus, W. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https:// openreview.net/forum?id=yzkSU5zdwD. Sur- vey Certification.
  40. Xiao, H., Rasul, K., and Vollgraf, R. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. CoRR, abs/1708.07747, 2017. URL http: //arxiv.org/abs/1708.07747. · arXiv
  41. Xiong, R., Yang, Y., He, D., Zheng, K., Zheng, S., Xing, C., Zhang, H., Lan, Y., Wang, L., and Liu, T. On layer normalization in the transformer architecture. In Inter- national Conference on Machine Learning, pp. 10524- 10533. PMLR, 2020.
  42. Zhai, S., Likhomanenko, T., Littwin, E., Busbridge, D., Ramapuram, J., Zhang, Y., Gu, J., and Susskind, J. M. Stabilizing transformer training by preventing attention entropy collapse. In International Conference on Machine Learning, pp. 40770-40803. PMLR, 2023.
  43. Zhang, L., Rao, A., and Agrawala, M. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836-3847, 2023.

Metadata

Authors
Hoffmann, D. T., Schrodi, S., Bratulić, J., Behrmann, N., Fischer, V., & Brox, T.
Venue
arXiv
Publisher
arXiv
Published
2023
DOI
10.48550/arxiv.2310.12956
arXiv
2310.12956
References
1
Type
Preprint
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.