Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 208 of the 208 references it cites were found in a registry or an index.

A Primer in BERTology: What We Know About How BERT Works

Open access
Rogers, Anna, Kovaleva, Olga, Rumshisky, Anna · 2020 · Transactions of the Association for Computational Linguistics

Abstract

Transformer-based models have pushed state of the art in many areas of NLP, but our understanding of what is behind their success is still limited. This paper is the first survey of over 150 studies of the popular BERT model. We review the current state of knowledge about how BERT works, what kind of information it learns and how it is represented, common modifications to its training objectives and architecture, the overparameterization issue, and approaches to compression. We then outline directions for future research.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1162/tacl_a_00349

Cites

208 of 208 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 182 references

Research works printed but not linked: 77

  1. Arase Yuki, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 5393
  2. Baevski Alexei, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 5360
  3. He Bai, 2020, arXiv:2004. 14996 [cs]
  4. Bouraoui Zied, 2019, arXiv:1911.12753 [cs] · arXiv
  5. Brunner Gino, 2020, International Conference on Learning Representations
  6. Clark Kevin, 2020, International Conference on Learning Representations
  7. Conneau Alexis, 2019, arXiv:1911.02116 [cs] · arXiv
  8. Cui Yiming, 2019, arXiv:1906.08101 [cs] · arXiv
  9. Da Jeff, 2019, Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing, 1
  10. Devlin Jacob, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171
  11. Elazar Yanai, 2020, arXiv:2006. 00995 [cs]
  12. Ettinger Allyson, 2019, arXiv: 1907.13528 [cs] · arXiv
  13. Fan Angela, 2019, International Conference on Learning Representations
  14. Forbes Maxwell, 2019, Proceedings of the 41st Annual Conference of the Cognitive Science Society (CogSci 2019), 7
  15. Frankle Jonathan, 2019, International Conference on Learning Representations
  16. Ganesh Prakhar, 2020, arXiv preprint arXiv:2002.11985 · arXiv
  17. Garg Siddhant, 2020, AAAI
  18. Gong Linyuan, 2019, International Conference on Machine Learning, 2337
  19. Gordon Mitchell A., 2020, arXiv preprint arXiv:2002.08307 · arXiv
  20. Goyal Saurabh, 2020, arXiv preprint arXiv:2001. 08950
  21. Hao Yaru, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4143
  22. Hinton Geoffrey, 2014, Deep Learning and Representation Learning Workshop: NIPS 2014
  23. Hoover Benjamin, 2019, arXiv:1910. 05276 [cs]
  24. Jain Sarthak, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 3543
  25. Jawahar Ganesh, 2019, 57th Annual Meeting of the Association for Computational Linguistics (ACL), Florence, Italy
  26. Jiang Haoming, 2019, arXiv preprint arXiv:1911.03437 · arXiv
  27. Jiang Zhengbao, 2019, arXiv:1911. 12543 [cs]
  28. Di Jin, 2020, AAAI 2020
  29. Kim Taeuk, 2020, ICLR 2020
  30. Kong Lingpeng, 2019, International Conference on Learning Representations
  31. Kovaleva Olga, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4356
  32. Krishna Kalpesh, 2020, ICLR 2020
  33. Kuznetsov Ilia, 2020, arXiv:2004.14999 [cs] · arXiv
  34. Lan Zhenzhong, 2020, ICLR
  35. Li Changmao, 2020, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 5709
  36. Lin Yongjie, 2019, Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 241
  37. Liu Nelson F., 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 1073
  38. Manning Christopher D., 2020, Proceedings of the National Academy of Sciences, 201907367
  39. May Chandler, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 622
  40. Thomas McCoy R., 2019, International Conference on Learning Representations
  41. Michel Paul, 2019, Advances in Neural Information Processing Systems 32 (NIPS 2019)
  42. Mikolov Tomas, 2013, Advances in Neural Information Processing Systems 26 (NIPS 2013), 3111
  43. Jiaqi Mu, 2018, International Conference on Learning Representations
  44. Pimentel Tiago, 2020, arXiv:2004. 03061 [cs]
  45. Poerner Nina, 2019, arXiv preprint arXiv: 1911.03681 · arXiv
  46. Raganato Alessandro, 2020, arXiv:2002.10260 [cs] · arXiv
  47. Richardson Kyle, 2020, AAAI 2020
  48. Richardson Kyle, 2019, arXiv:1912.13337 [cs] · arXiv
  49. Roberts Adam, 2020, arXiv preprint arXiv:2002.08910 · arXiv
  50. Rogers Anna, 2020, AAAI, 11
  51. Sanh Victor, 2019, 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing - NeurIPS 2019
  52. Serrano Sofia, 2019, arXiv:1906.03731 [cs] · arXiv
  53. Shen Sheng, 2019, arXiv preprint arXiv:1909. 05840
  54. Stickland Asa Cooper, 2019, International Conference on Machine Learning, 5986
  55. Strubell Emma, 2019, ACL 2019
  56. Sugawara Saku, 2020, AAAI
  57. Sun Siqi, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4314
  58. Talmor Alon, 2019, arXiv:1912.13283 [cs] · arXiv
  59. Tenney Ian, 2019, International Conference on Learning Representations
  60. Schijndel Marten van, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 5831
  61. Vaswani Ashish, 2017, Advances in Neural Information Processing Systems, 5998
  62. Vig Jesse, 2019, Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 63
  63. Vilares David, 2020, Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20)
  64. Voita Elena, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 4387
  65. Voita Elena, 2020, arXiv:2003.12298 [cs] · arXiv
  66. Wallace Eric, 2019, arXiv preprint arXiv:1909. 07940
  67. Warstadt Alex, 2020, Proceedings of the 42nd Annual Virtual Meeting of the Cognitive Science Society
  68. Warstadt Alex, 2019, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2870
  69. Wolf Thomas, 2020, arXiv:1910.03771 [cs] · arXiv
  70. Felix Wu, 2019, International Conference on Learning Representations
  71. Zhiyong Wu, 2020, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 4166
  72. Yin Pengcheng, 2020, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 8413
  73. Zadeh Ali Hadi, 2020, arXiv:2005.03842 [cs, stat] · arXiv
  74. Zafrir Ofir, 2019, arXiv preprint arXiv:1910.06188 · arXiv
  75. Zhang Zhuosheng, 2020, AAAI 2020
  76. Zhao Sanqiang, 2019, arXiv preprint arXiv:1909.11687 · arXiv
  77. Zhou Xuhui, 2020, AAAI 2020

Cited by

See all 1,552

Metadata

Authors
Rogers, A., Kovaleva, O., & Rumshisky, A.
Affiliations
  • University of Copenhagen (DK)
  • University of Massachusetts Lowell (US)
Venue
Transactions of the Association for Computational Linguistics
Publisher
MIT Press
Published
2020-12
Volume
8
Pages
842-866
ISSN
2307-387X (ISSN-L)
OpenAlex
W3006881356
Cited by
1,552
References
182
Citations by year
2026: 171 · 2025: 249 · 2024: 183 · 2023: 292 · 2022: 237 · 2021: 320 · 2020: 100
FWCI
93.20
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • IT University Of Copenhagen (IT University of Copenhagen) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Science Foundation (1844740)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
BERT, transformer models, pre-trained language models, model interpretability, training objectives, model architecture modification, overparameterization, model compression, learned representations

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A Primer in BERTology: What We Know About How BERT Works. https://app.cochat.ai/papers/10.1162/tacl_a_00349
The paper
Rogers, A., Kovaleva, O., & Rumshisky, A. (2020). A Primer in BERTology: What We Know About How BERT Works. Transactions of the Association for Computational Linguistics. https://doi.org/10.1162/tacl_a_00349

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.