The DOI matches its Crossref record; 56 of the 62 references it cites were found in a registry or an index, and 6 are not checked yet.
How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models
Open accessAbstract
Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, Iryna Gurevych. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“increasing inference cost and compressing effective context windows (Rust et al., 2021; Petrov et al., 2024).”
From Writing-System-Level Tokenizer Adaptation for Byte-Level BPE (2026) · cites this paper as Rust et al., 2021
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Decoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
Unsupervised Cross-lingual Representation Learning at Scale
Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.747 · Cited by 5,637
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · cited 1×
“Exponential smoothing was used when creating the subword vocabulary based on WordPieces (Wu et al., 2016) and a pretraining corpus.”
Method
Research works printed but not linked: 10
- Erkin Demirtas and Mykola Pechenizkiy. 2013. Cross- lingual polarity detection with machine translation. In Proceedings of the Second International Work- shop on Issues of Sentiment Discovery and Opinion Mining (WISDOM '13), pages 9:1-8, Chicago, USA. Association for Computing Machinery. · DOI
- Junjie Hu, Sebastian Ruder, Aditya Siddhant, Gra- ham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multi- task benchmark for evaluating cross-lingual gener- alisation. In Proceedings of the 37th International Conference on Machine Learning, pages 4411-4421, Virtual.
- PMLR.
- Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analy- sis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Hu- man Language Technologies, pages 142-150, Port- land, Oregon, USA. Association for Computational Linguistics.
- Joakim Nivre, Marie-Catherine de Marneffe, Filip Gin- ter, Yoav Goldberg, Jan Hajič, Christopher D. Man- ning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal Dependencies v1: A multilingual treebank collection. In Proceedings of the Tenth In- ternational Conference on Language Resources and Evaluation (LREC'16), pages 1659-1666, Portorož, Slovenia. European Language Resources Associa- tion (ELRA).
- Debora Nozza, Federico Bianchi, and Dirk Hovy. 2020. What the [MASK]? Making sense of language- specific BERT models. arXiv preprint.
- Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Se- bastian Ruder. 2020c. UNKs Everywhere: Adapt- ing Multilingual Language Models to New Scripts. arXiv preprint.
- Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to- text transformer. Journal of Machine Learning Re- search, 21(140):1-67.
- Bryan Wilie, Karissa Vincentio, Genta Indra Winata, Samuel Cahyawijaya, Xiaohong Li, Zhi Yuan Lim, Sidik Soleman, Rahmad Mahendra, Pascale Fung, Syafri Bahar, and Ayu Purwarianti. 2020. IndoNLU: Benchmark and resources for evaluating Indonesian natural language understanding. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Lan- guage Processing, pages 843-857, Suzhou, China. Association for Computational Linguistics.
- Yian Zhang, Alex Warstadt, Haau-Sing Li, and Samuel R. Bowman. 2020. When do you need bil- lions of words of pretraining data? arXiv preprint.
Cited by
Sorted by citations
Items 201–221 of 221
Benchmarking Byte-Pair Encoding Tokenizers on Different Languages with Bits per Byte
Chowdhury, S., & Woolf, W. (2026) · Workshop on Multilinguality in the Era of Large Language Models (MeLLM) · doi:10.18653/v1/2026.mellm-1.27
Onan, A., Nasution, A. H., & Eğin, F. (2026) · Discover Computing · doi:10.1007/s10791-026-10329-0
Tokenizations for Austronesian Language Models: Study on Languages in Indonesia Archipelago
Lumbantobing, A. B., Situngkir, H., & Ringo, K. S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29532-3_13
Özmen, H. B., & Senel, F. A. (2026) · International Journal of Artificial Intelligence Tools · doi:10.1142/s021821302650017x
SiddiReddy, S. K. R., Sheshadri, S. K., Gupta, D., Paul, B., Sivabhavani, J., & Nair, P. C. (2026) · Procedia Computer Science · doi:10.1016/j.procs.2026.06.244
Memming, A. (2026) · Research Square · doi:10.21203/rs.3.rs-10486884/v1
Feature Analysis of Tokenizers in Multi-Category Text Classification
Anandita, V. V. A. F., Richasdy, D., & Gani, P. H. (2026) · International Conference on Electronics Representation and Algorithm (ICERA) · doi:10.1109/icera72709.2026.11666761
電子医療記録用のマルチモーダルディー プラーニング予測モデルの改善
Mugisha, C. (n.d.) · Institutional Repositories DataBase (IRDB) · doi:10.15016/0002000006
Knowledge-guided descriptors for multi-property prediction and candidate screening of superalloys
Wang, W., Jiang, X., & Su, Y. (2026) · Materials Today Communications · doi:10.1016/j.mtcomm.2026.116085
Qizi, M. S. T., Djamaldinova, S., & Umirzakova, S. (2026) · Electronics · doi:10.3390/electronics15184236
Sejdiu, N. P., & Sejdiu, S. (2026) · Applied Linguistics · doi:10.1093/applin/amag089
Tokenizer Impact on Multilingual to Portuguese Transfer in ModernBERT-Based Models
Accorsi, A. T., Losch, G., Gomes, C. F. A., & Griebler, D. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-39892-5_34
Evaluating IndoBERT Dense Retrieval for Indonesian Presidential Regulation Search
Christianto, A., & Madyatmadja, E. D. (2026) · International Conference on ICT for Smart Society (ICISS) · doi:10.1109/iciss71466.2026.11715370
Oluwaseyi, A. J., Osariemen, O. S., Ehijie, I. D., & Olayemi, I. M. (2026) · Asian Journal of Computer Science and Technology · doi:10.70112/ajcst-2026.15.2.4459
Semantic-Driven Context Pruning for Arabic RAG: Quality-Preserving Compression Under Token Inflation
Taresh, A. A. R., & Zghair, N. A. K. (2026) · Information · doi:10.3390/info17100990
To Slide or to Snip? LLM Fine-Tuning for Sentiment Analysis of Long Serbian Movie Reviews
Todorović, A., & Batanović, V. (2026) · International Conference on Electrical, Electronic and Computing Engineering (IcETRAN) · doi:10.1109/icetran71739.2026.11727724
Binodini, P., Naorem, M., Thokchom, C. S., Mukhopadhyay, S., Nongmeikapam, K., & Sarkar, S. (2026) · Electronics · doi:10.3390/electronics15204811
Evaluating LLM Tokenization Efficiency on Kuwaiti Dialect, MSA and Other Arabic Dialects
Alostad, H., & Almutairi, Y. (2026) · Big Data and Cognitive Computing · doi:10.3390/bdcc10100356
Cross-Lingual Question Answering in Open-Weight LLMs: English Versus Turkish Prompting
Kara, B. Z., & Kabakuş, A. T. (2026) · International Artificial Intelligence and Data Processing Symposium (IDAP) · doi:10.1109/idap71869.2026.11728095
Aksaya, H., & Gülseçen, S. (2026) · Electronics · doi:10.3390/electronics15214949
The New Perspectiva Artificialis
Hutson, J., & Plate, D. (2026) · Routledge eBooks · doi:10.4324/9781042013173
Metadata
- Authors
- Rust, P., Pfeiffer, J., Vulić, I., Ruder, S., & Gurevych, I.
- Affiliations
- Technische Universität Darmstadt (DE)
- University of Cambridge (GB)
- Venue
- Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
- Published
- 2021
- Pages
- 3118-3135
- OpenAlex
- W3120253119
- Cited by
- 202
- References
- 57
- Citations by year
- 2026: 32 · 2025: 32 · 2024: 22 · 2023: 50 · 2022: 35 · 2021: 31
- FWCI
- 15.94
- Type
- conference paper
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- arXiv (Cornell University) (submittedVersion)
- TUbilio (Technical University of Darmstadt) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- Funders
- H2020 European Research Council (648909)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- tokenization, multilingual language models, multilingual tokenization, language model evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
