The DOI matches its Crossref record; 56 of the 62 references it cites were found in a registry or an index, and 6 are not checked yet.
How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models
Open accessAbstract
Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, Iryna Gurevych. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“increasing inference cost and compressing effective context windows (Rust et al., 2021; Petrov et al., 2024).”
From Writing-System-Level Tokenizer Adaptation for Byte-Level BPE (2026) · cites this paper as Rust et al., 2021
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Decoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
Unsupervised Cross-lingual Representation Learning at Scale
Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.747 · Cited by 5,637
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · cited 1×
“Exponential smoothing was used when creating the subword vocabulary based on WordPieces (Wu et al., 2016) and a pretraining corpus.”
Method
Research works printed but not linked: 10
- Erkin Demirtas and Mykola Pechenizkiy. 2013. Cross- lingual polarity detection with machine translation. In Proceedings of the Second International Work- shop on Issues of Sentiment Discovery and Opinion Mining (WISDOM '13), pages 9:1-8, Chicago, USA. Association for Computing Machinery. · DOI
- Junjie Hu, Sebastian Ruder, Aditya Siddhant, Gra- ham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multi- task benchmark for evaluating cross-lingual gener- alisation. In Proceedings of the 37th International Conference on Machine Learning, pages 4411-4421, Virtual.
- PMLR.
- Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analy- sis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Hu- man Language Technologies, pages 142-150, Port- land, Oregon, USA. Association for Computational Linguistics.
- Joakim Nivre, Marie-Catherine de Marneffe, Filip Gin- ter, Yoav Goldberg, Jan Hajič, Christopher D. Man- ning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal Dependencies v1: A multilingual treebank collection. In Proceedings of the Tenth In- ternational Conference on Language Resources and Evaluation (LREC'16), pages 1659-1666, Portorož, Slovenia. European Language Resources Associa- tion (ELRA).
- Debora Nozza, Federico Bianchi, and Dirk Hovy. 2020. What the [MASK]? Making sense of language- specific BERT models. arXiv preprint.
- Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Se- bastian Ruder. 2020c. UNKs Everywhere: Adapt- ing Multilingual Language Models to New Scripts. arXiv preprint.
- Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to- text transformer. Journal of Machine Learning Re- search, 21(140):1-67.
- Bryan Wilie, Karissa Vincentio, Genta Indra Winata, Samuel Cahyawijaya, Xiaohong Li, Zhi Yuan Lim, Sidik Soleman, Rahmad Mahendra, Pascale Fung, Syafri Bahar, and Ayu Purwarianti. 2020. IndoNLU: Benchmark and resources for evaluating Indonesian natural language understanding. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Lan- guage Processing, pages 843-857, Suzhou, China. Association for Computational Linguistics.
- Yian Zhang, Alex Warstadt, Haau-Sing Li, and Samuel R. Bowman. 2020. When do you need bil- lions of words of pretraining data? arXiv preprint.
Cited by
Sorted by citations
A review of large language models and autonomous agents in chemistry
Ramos, M. C., Collison, C. J., & White, A. D. (2025) · Chemical Science · doi:10.1039/d4sc03921a · Cited by 274
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language ProcessingPreprint
Kalyan, K. S., Rajasekharan, A., & Sangeetha, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.05542 · Cited by 357
MEGA: Multilingual Evaluation of Generative AI
Ahuja, K., Diddee, H., Hada, R., Ochieng, M., Ramesh, K., Jain, P., Nambi, A. U., Ganu, T., Segal, S., Ahmed, M., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.258 · Cited by 321
The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language ModelsPreprint
Inoue, G., Alhafni, B., Baimukan, N., Bouamor, H., & Habash, N. Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.06678 · Cited by 224
Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPPreprint
Mielke, S. J., Alyafeai, Z., Salesky, E., Raffel, C., Dey, M., Gallé, M., Raja, A., Si, C., Lee, W. Y., Sagot, B., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.10508 · Cited by 328
Metadata
- Authors
- Rust, P., Pfeiffer, J., Vulić, I., Ruder, S., & Gurevych, I.
- Affiliations
- Technische Universität Darmstadt (DE)
- University of Cambridge (GB)
- Venue
- Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
- Published
- 2021
- Pages
- 3118-3135
- OpenAlex
- W3120253119
- Cited by
- 202
- References
- 57
- Citations by year
- 2026: 32 · 2025: 32 · 2024: 22 · 2023: 50 · 2022: 35 · 2021: 31
- FWCI
- 15.94
- Type
- conference paper
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- arXiv (Cornell University) (submittedVersion)
- TUbilio (Technical University of Darmstadt) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- Funders
- H2020 European Research Council (648909)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- tokenization, multilingual language models, multilingual tokenization, language model evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
