ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models
Open accessAbstract
Most widely used pre-trained language models operate on sequences of tokens corresponding to word or subword units. By comparison, token-free models that operate directly on raw text (bytes or characters) have many benefits: They can process text in any language out of the box, they are more robust to noise, and they minimize technical debt by removing complex and error-prone text preprocessing pipelines. Because byte or character sequences are longer than token sequences, past work on token-free models has often introduced new model architectures designed to amortize the cost of operating directly on raw text. In this paper, we show that a standard Transformer architecture can be used with minimal modifications to process byte sequences. We characterize the trade-offs in terms of parameter count, training FLOPs, and inference speed, and show that byte-level models are competitive with their token-level counterparts. We also demonstrate that byte-level models are significantly more robust to noise and perform better on tasks that are sensitive to spelling and pronunciation. As part of our contribution, we release a new set of pre-trained byte-level Transformer models based on the T5 architecture, as well as all code and data used in our experiments.1
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In ByT5, the text is processed as UTF-8 bytes, with spans of approximately 20 bytes being masked.
“In contrast, ByT5 (Xue et al. 2022) processes the text as UTF-8 bytes, with spans of approximately 20 bytes being masked.”
From Large language models (LLMs): survey, technical frameworks, and future challenges (2024) · cites this paper as Xue et al. 2022
Character-level tokenization is advantageous when dealing with languages lacking clear word boundaries or for specific tasks like character-level language modeling.
“Character-level tokenization is advantageous when dealing with languages lacking clear word boundaries or for specific tasks like character-level language modeling (Sutskever et al. 2011; Al-Rfou et al. 2019; Xue et al. 2022).”
From Large language models (LLMs): survey, technical frameworks, and future challenges (2024) · cites this paper as Xue et al. 2022
In ByT5, the encoder is three times deeper than the decoder.
“Additionally, the encoder in ByT5 is three times deeper than the decoder.”
From Large language models (LLMs): survey, technical frameworks, and future challenges (2024) · cites this paper as Xue et al. 2022
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,759 · see citing works
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,881 · see citing works
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
Self-citationProceedings of the 25th international conference on Machine learning
Cohen, W. W., McCallum, A., & Roweis, S. T. (2008) · Cited by 5,691 · see citing works
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · see citing works
Kudo, T., & Richardson, J. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-2012 · Cited by 4,454 · see citing works
Generating Sequences With Recurrent Neural Networks
Graves, A. (2013) · arXiv · doi:10.48550/arxiv.1308.0850 · Cited by 2,894 · see citing works
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · see citing works
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer
Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., & Raffel, C. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.41 · Cited by 1,629 · see citing works
Self-citation(2018) · International Conference on Learning Representations · Cited by 1,621 · see citing works
Character-Aware Neural Language Models
Kim, Y., Jernite, Y., Sontag, D., & Rush, A. M. (2015) · arXiv · doi:10.48550/arxiv.1508.06615 · Cited by 1,530 · see citing works
ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
Sandsten, M., Brynolfsson, J., Reinhold, I., & Starkhammar, J. (2019) · doi:10.1109/icassp35589.2019 · Cited by 1,273 · see citing works
Efficient transformers: A survey
Tay, Y., Dehghani, M., Bahri, D., & Metzler, D. (2023) · ACM Computing Surveys · doi:10.1145/3530811 · Cited by 1,267 · see citing works
Efficient Transformers: A Survey
Tay, Y., Dehghani, M., Bahri, D., & Metzler, D. (2020) · arXiv · doi:10.48550/arxiv.2009.06732 · Cited by 1,267 · see citing works
Generating Text with Recurrent Neural Networks
Sutskever, I., Martens, J., & Hinton, G. E. (2011) · Cited by 1,266 · see citing works
Narayan, S., Cohen, S. B., & Lapata, M. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1206 · Cited by 1,207 · see citing works
Exploring the Limits of Language ModelingPreprint
Józefowicz, R., Vinyals, O., Schuster, M., Shazeer, N., & Wu, Y.-H. (2016) · arXiv.org · doi:10.48550/arxiv.1602.02410 · Cited by 1,202 · see citing works
Contextual String Embeddings for Sequence Labeling
Akbik, A., Blythe, D. A. J., & Vollgraf, R. (2018) · International Conference on Computational Linguistics · Cited by 1,084 · see citing works
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019) · arXiv · doi:10.48550/arxiv.1905.00537 · Cited by 1,051 · see citing works
Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates
Kudo, T. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1007 · Cited by 1,021 · see citing works
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Fedus, W., Zoph, B., & Shazeer, N. (2021) · arXiv · doi:10.48550/arxiv.2101.03961 · Cited by 1,005 · see citing works
XNLI: Evaluating Cross-lingual Sentence Representations
Conneau, A., Rinott, R., Lample, G., Williams, A., Bowman, S. R., Schwenk, H., & Stoyanov, V. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1269 · Cited by 949 · see citing works
Ambati, B. R., Deoskar, T., Johnson, M., & Steedman, M. J. (2015) · The Association for Computational Linguistics · Cited by 860 · see citing works
On the Cross-lingual Transferability of Monolingual Representations
Artetxe, M., Ruder, S., & Yogatama, D. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.421 · Cited by 618 · see citing works
Clark, J. H., Choi, E., Collins, M., Garrette, D., Kwiatkowski, T., Nikolaev, V., & Palomaki, J. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00317 · Cited by 432 · see citing works
MLQA: Evaluating Cross-lingual Extractive Question Answering
Lewis, P., Oğuz, B., Rinott, R., Riedel, S., & Schwenk, H. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.653 · Cited by 424 · see citing works
Neural Machine Translation in Linear Time
Kalchbrenner, N., Espeholt, L., Simonyan, K., Oord, A. V. D., Graves, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1610.10099 · Cited by 414 · cited 1× · see citing works
“This precedent led many to use character-level language modeling as a benchmark to evaluate neural architectures (Kalchbrenner et al., 2016; Chung et al., 2017; Ha et al., 2017; Zilly et al., 2017; Melis et al., 2018; Al-Rfou et al., 2019).”
MethodFully Character-Level Neural Machine Translation without Explicit Segmentation
Lee, J., Cho, K., & Hofmann, T. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00067 · Cited by 399 · see citing works
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., & Gardner, M. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1246 · Cited by 393 · cited 2× · see citing works
“Finally, DROP (Dua et al., 2019) is a challenging reading comprehension task that requires numerical reasoning.”
BackgroundInfluentialHierarchical Multiscale Recurrent Neural Networks
Chung, J., Ahn, S., & Bengio, Y. (2016) · arXiv · doi:10.48550/arxiv.1609.01704 · Cited by 386 · see citing works
Adaptive Computation Time for Recurrent Neural Networks
Graves, A. (2016) · arXiv · doi:10.48550/arxiv.1603.08983 · Cited by 335 · see citing works
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Zhang, J., Zhao, Y., Saleh, M., & Liu, P. X. (2020) · International Conference on Machine Learning · Cited by 325 · see citing works
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalisation
Hu, J., Ruder, S., Siddhant, A., Neubig, G., Fırat, O., & Johnson, M. (2020) · International Conference on Machine Learning · Cited by 325 · see citing works
Cross-lingual Name Tagging and Linking for 282 Languages
Pan, X., Zhang, B., May, J., Nothman, J., Knight, K. K., & Ji, H. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1178 · Cited by 313 · see citing works
A Character-level Decoder without Explicit Segmentation for Neural Machine Translation
Chung, J., Cho, K., & Bengio, Y. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1160 · Cited by 310 · see citing works
Combating Adversarial Misspellings with Robust Word Recognition
Pruthi, D., Dhingra, B., & Lipton, Z. C. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1561 · Cited by 285 · see citing works
PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification
Yang, Y., Zhang, Y. E., Tar, C., & Baldridge, J. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1382 · Cited by 231 · see citing works
Multilingual Language Processing From Bytes
Gillick, D., Brunk, C., Vinyals, O., & Subramanya, A. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1155 · Cited by 211 · see citing works
Character-based neural machine translation
Costa‐jussà, M. R., & Fonollosa, J. A. R. (2016) · RECERCAT (Consorci de Serveis Universitaris de Catalunya) · Cited by 210 · see citing works
Character-based Neural Machine TranslationPreprint
Ling, W., Trancoso, I. M., Dyer, C., & Black, A. W. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.04586 · Cited by 197 · see citing works
Canine : Pre-training an Efficient Tokenization-Free Encoder for Language Representation
Clark, J. H., Garrette, D., Turc, I., & Wieting, J. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00448 · Cited by 161 · see citing works
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
Gehrmann, S., Adewumi, T., Aggarwal, K., Ammanamanchi, P. S., Aremu, A., Bosselut, A., Chandu, K. R., Clinciu, M.-A., Das, D., Dhole, K., et al. (2021) · Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021) · doi:10.18653/v1/2021.gem-1.10 · Cited by 159 · see citing works
Self-citationBoukkouri, H. E., Ferret, O., Lavergne, T., Noji, H., Zweigenbaum, P., & Tsujii, J. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.609 · Cited by 152 · see citing works
Neural Machine Translation with Byte-Level Subwords
Wang, C., Cho, K., & Gu, J. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6451 · Cited by 147 · see citing works
Adv-BERT: BERT is not robust on misspellings! Generating nature adversarial samples on BERTPreprint
Sun, L., Hashimoto, K., Yin, W., Asai, A., Li, J., Yu, P. S., & Xiong, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.04985 · Cited by 92 · see citing works
Revisiting Character-Based Neural Machine Translation with Capacity and Compression
Cherry, C. A., Foster, G. F., Bapna, A., Fırat, O., & Macherey, W. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1461 · Cited by 91 · see citing works
SIGMORPHON 2020 Shared Task 0: Typologically Diverse Morphological Inflection
Vylomova, E., White, J. C., Salesky, E., et al. (2020) · SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology (SIGMORPHON) · doi:10.18653/v1/2020.sigmorphon-1.1 · Cited by 71 · see citing works
TWEETQA: A Social Media Focused Question Answering Dataset
Xiong, W., Wu, J., Wang, H., Kulkarni, V., Yu, M., Chang, S., Guo, X., & Wang, W. Y. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1496 · Cited by 67 · see citing works
Character-Level Language Modeling with Deeper Self-Attention
Al-Rfou, R., Choe, D., Constant, N., Guo, M., & Jones, L. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013159 · Cited by 56 · see citing works
Self-citationThe SIGMORPHON 2020 Shared Task on Multilingual Grapheme-to-Phoneme Conversion
Gorman, K., Ashby, L. F. E., Goyzueta, A., et al. (2020) · SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology (SIGMORPHON) · doi:10.18653/v1/2020.sigmorphon-1.2 · Cited by 52 · see citing works
Question Directed Graph Attention Network for Numerical Reasoning over Text
Chen, K., Xu, W., Cheng, X., Xiaochuan, Z., Zhang, Y., Song, L., Wang, T., Qi, Y., & Chu, W. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.549 · Cited by 50 · see citing works
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
Gehrmann, S., Adewumi, T., Aggarwal, K., Ammanamanchi, P. S., Anuoluwapo, A., Bosselut, A., Chandu, K. R., Clinciu, M., Das, D., Dhole, K. D., et al. (2021) · arXiv · doi:10.48550/arxiv.2102.01672 · Cited by 49 · see citing works
Self-citationLanguage ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus
Caswell, I. R., Breiner, T., van Esch, D., & Bapna, A. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.579 · Cited by 47 · see citing works
Byte-based Neural Machine Translation
Costa‐jussà, M. R., Escolano, C., & Fonollosa, J. A. R. (2017) · doi:10.18653/v1/w17-4123 · Cited by 43 · see citing works
Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitution
Garcia, X., Constant, N., Parikh, A. P., & Fırat, O. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.93 · Cited by 34 · see citing works
Self-citationBoukkouri, H. E., Ferret, O., Lavergne, T., Noji, H., Zweigenbaum, P., & Tsujii, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.10392 · Cited by 26 · see citing works
Processing South Asian Languages Written in the Latin Script: the Dakshina DatasetPreprint
Roark, B., Wolf-Sonkin, L., Kirov, C., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.01176 · Cited by 20 · see citing works
One-Size-Fits-All Multilingual Models
Peters, B., & Martins, A. F. T. (2020) · SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology (SIGMORPHON) · doi:10.18653/v1/2020.sigmorphon-1.4 · Cited by 18 · see citing works
Yu, X., Vu, N. T., & Kuhn, J. (2020) · SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology (SIGMORPHON) · doi:10.18653/v1/2020.sigmorphon-1.5 · Cited by 18 · see citing works
Training Multilingual Pre-trained Language Model with Byte-level SubwordsPreprint
Wei, V. J., Liu, Q., Guo, Y., & Jiang, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.09469 · Cited by 17 · see citing works
Bridging the Gap for Tokenizer-Free Language ModelsPreprint
Choe, D., Al‐Rfou, R., Guo, M., Lee, H., & Constant, N. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.10322 · Cited by 14 · see citing works
Self-citationWhich transformer architecture fits my data? A vocabulary bottleneck in self-attentionPreprint
Wies, N., Levine, Y., Jannai, D., & Shashua, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.03928 · Cited by 12 · see citing works
Bytes are All You Need: End-to-End Multilingual Speech Recognition and Synthesis with BytesPreprint
Li, B., Zhang, Y., Sainath, T. N., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.09021 · Cited by 9 · see citing works
Interpreting Word-Level Hidden State Behaviour of Character-Level LSTM Language Models
Hiebert, A., Peterson, C., Fyshe, A., & Mehta, N. A. (2018) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/w18-5428 · Cited by 6 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · see citing works
“In comparison, mT5 (Xue et al., 2021b) and ByT5 (Xue et al., 2021a) were trained on 6x and 1.5x as much non-English text as PaLM, respectively.”
BackgroundMethodSelf-citationLarge language models (LLMs): survey, technical frameworks, and future challenges
Kumar, P. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10888-y · Cited by 395 · see citing works
Position Information in Transformers: An Overview
Dufter, P., Schmitt, M., & Schütze, H. (2022) · Computational Linguistics · doi:10.1162/coli_a_00445 · Cited by 175 · see citing works
FairPrism: Evaluating Fairness-Related Harms in Text Generation
Fleisig, E., Amstutz, A., Atalla, C., Blodgett, S. L., Daumé III, H., Olteanu, A., Sheng, E., Vann, D., & Wallach, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.343 · Cited by 14 · see citing works
“We used ByT5 fine-tuned for hate speech detection by Narrativa, Unitary’s Detoxify model, and Detoxify’s “unbiased” version (which we refer to as “DetoxifyUnbiased”) to classify FairPrism’s AI-generated outputs as hate speech or innocuous (Xue et al., 2022; Hanu and Unitary team, 2020).”
BackgroundMethodToo Much Information: Keeping Training Simple for BabyLMs
Edman, L., & Bylinina, L. (2023) · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-babylm.8 · Cited by 1 · see citing works
- Authors
- Xue, L., Barua, A., Constant, N., Al-Rfou, R., Narang, S., Kale, M., Roberts, A., & Raffel, C.
- Affiliations
- Google (United States) (US)
- Google Research, USA. rmyeid@google.com
- Corresponding author
- Xue, Linting
- Barua, Aditya
- Constant, Noah
- Narang, Sharan
- Kale, Mihir
- Roberts, Adam
- Raffel, Colin
- ORCID
- Raffel, Colin
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2022
- Volume
- 10
- Pages
- 291-306
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W3164045210
- Cited by
- 330
- References
- 70
- Citations by year
- 2026: 48 · 2025: 62 · 2024: 39 · 2023: 112 · 2022: 46 · 2021: 23
- FWCI
- 21.30
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- character-level language models, pre-trained language models, T5, transformer models, noise robustness, parameter efficiency, training and inference efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
