Improving Transformer Models by Reordering their Sublayers
Open accessAbstract
Multilayer transformer networks consist of interleaved self-attention and feedforward sublayers.Could ordering the sublayers in a different pattern lead to better performance?We generate randomly ordered transformers and train them with the language modeling objective.We observe that some of these models are able to achieve better performance than the interleaved baseline, and that those successful variants tend to have more self-attention at the bottom and more feedforward sublayers at the top.We propose a new transformer pattern that adheres to this property, the sandwich transformer, and show that it improves perplexity on multiple word-level and character-level language modeling benchmarks, at no cost in parameters, memory, or training time.However, the sandwich reordering pattern does not guarantee performance gains across every task, as we demonstrate on machine translation models.Instead, we suggest that further exploration of task-specific sublayer reorderings is needed in order to unlock additional gains. 1
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Cumulative noise effect increases response latency and accelerates nonlinear consumption of computational and storage resources, threatening scalability and stability in long-term multi-agent system operations.
“In addition, the cumulative noise effect increases response latency and accelerates the nonlinear consumption of computational and storage resources [22], ultimately threatening both scalability and stability in long-term MAS operations [43].”
From SEDM: Scalable Self-Evolving Distributed Memory for Agents (2025) · cites this paper as [43]
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
(2014) · Cited by 33,419 · see citing works
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072 · see citing works
Layer NormalizationPreprint
Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,745 · see citing works
The Hungarian method for the assignment problem
Kuhn, H. W. (1955) · Naval Research Logistics Quarterly · doi:10.1002/nav.3800020109 · Cited by 13,120 · see citing works
Neural Architecture Search with Reinforcement LearningPreprint
Zoph, B., & Le, Q. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01578 · Cited by 5,091 · see citing works
Pointer Sentinel Mixture ModelsPreprint
Merity, S., Xiong, C., Bradbury, J., & Socher, R. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1609.07843 · Cited by 5,051 · see citing works
Transformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169 · see citing works
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., & Fidler, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.11 · Cited by 2,264 · see citing works
An Empirical Exploration of Recurrent Network Architectures
Józefowicz, R., Zaremba, W., & Sutskever, I. (2015) · Cited by 1,459 · see citing works
Scaling Neural Machine Translation
Ott, M., Edunov, S., Grangier, D., & Auli, M. (2018) · Conference on Machine Translation · doi:10.18653/v1/w18-6301 · Cited by 662 · see citing works
Using the Output Embedding to Improve Language Models
Press, O., & Wolf, L. (2017) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/e17-2025 · Cited by 642 · see citing works
Self-citationTying Word Vectors and Word Classifiers: A Loss Framework for Language ModelingPreprint
Inan, H., Khosravi, K., & Socher, R. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01462 · Cited by 314 · see citing works
Adaptive Attention Span in Transformers
Sukhbaatar, S., Grave, É., Bojanowski, P., & Joulin, A. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1032 · Cited by 286 · see citing works
Generalization through Memorization: Nearest Neighbor Language Models
Khandelwal, U., Levy, O., Jurafsky, D., Zettlemoyer, L., & Lewis, M. (2019) · arXiv · doi:10.48550/arxiv.1911.00172 · Cited by 285 · see citing works
Self-citationSo, D. R., Liang, C., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1901.11117 · Cited by 277 · see citing works
Adaptive Input Representations for Neural Language ModelingPreprint
Baevski, A., & Auli, M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.10853 · Cited by 255 · see citing works
Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of ViewPreprint
Lu, Y., Li, Z., He, D., Sun, Z., Dong, B., Qin, T., Wang, L., & Liu, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.02762 · Cited by 146 · see citing works
Transformers without Tears: Improving the Normalization of Self-AttentionPreprint
Nguyen, T. Q., & Salazar, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.05895 · Cited by 124 · see citing works
Adaptively Sparse TransformersPreprint
Correia, G. M., Niculae, V., & Martins, A. F. T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.00015 · Cited by 35 · see citing works
Improving Deep Transformer with Depth-Scaled Initialization and Merged AttentionPreprint
Zhang, B., Titov, I., & Sennrich, R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.11365 · Cited by 9 · see citing works
Flagged first, then by citations
2 of the 24 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
A primer in BERTology: What we know about how BERT works
Rogers, A., Kovaleva, O., & Rumshisky, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00349 · Cited by 1,552 · see citing works
Lin, T., Wang, Y., Liu, X., & Qiu, X. (2022) · AI Open · doi:10.1016/j.aiopen.2022.10.001 · Cited by 1,422 · see citing works
Understanding Robustness of Transformers for Image Classification
Bhojanapalli, S., Chakrabarti, A., Glasner, D., Li, D., Unterthiner, T., & Veit, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01007 · Cited by 440 · see citing works
“We then turn our attention to the action of the ViT models themselves, analyzing the evolution of information through the cascade of Transformer layers and the redundancies in their internal representations via correlation analysis and lesion studiesÐas has been done in the past for Transformers for language tasks [51…”
MethodPre-Trained Language Models and Their Applications
Wang, H., Li, J., Wu, H. R., Hovy, E. H., & Sun, Y. (2022) · Engineering · doi:10.1016/j.eng.2022.04.024 · Cited by 404 · see citing works
Transformer Feed-Forward Layers Are Key-Value Memories
Geva, M., Schuster, R., Berant, J., & Levy, O. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.446 · Cited by 353 · see citing works
Self-citationTrain Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
Press, O., Smith, N. A., & Lewis, M. (2021) · arXiv · doi:10.48550/arxiv.2108.12409 · Cited by 161 · see citing works
Self-citationStructured Pruning Learns Compact and Accurate Models
Xia, M., Zhong, Z., & Chen, D. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.107 · Cited by 136 · see citing works
E-Branchformer: Branchformer with Enhanced Merging for Speech Recognition
Kim, K., Wu, F., Peng, Y., Pan, J. S., Sridhar, P., Han, K. J., & Watanabe, S. (2023) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt54892.2023.10022656 · Cited by 107 · see citing works
Transformer Language Models without Positional Encodings Still Learn Positional Information
Haviv, A., Ram, O., Press, O., Izsak, P., & Levy, O. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.99 · Cited by 71 · see citing works
Self-citationLessons on Parameter Sharing across Layers in Transformers
Takase, S., & Kiyono, S. (2023) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2023.sustainlp-1.5 · Cited by 68 · see citing works
Zhou, K., Wang, W., Hu, T., & Deng, K. (2020) · Sensors · doi:10.3390/s20247211 · Cited by 54 · see citing works
Shortformer: Better language modeling using shorter inputs
Press, O., Smith, N. A., & Lewis, M. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.427 · Cited by 55 · see citing works
Self-citationRe-Transformer: A Self-Attention Based Model for Machine Translation
Liu, H.-I., & Chen, W. (2021) · Procedia Computer Science · doi:10.1016/j.procs.2021.05.065 · Cited by 38 · see citing works
X-Transformer: A Machine Translation Model Enhanced by the Self-Attention Mechanism
Liu, H.-I., & Chen, W.-L. (2022) · Applied Sciences · doi:10.3390/app12094502 · Cited by 32 · see citing works
Traffic demand prediction based on spatial-temporal guided multi graph Sandwich-Transformer
Wen, Y., Li, Z., Wang, X., & Xu, W. A. (2023) · Information Sciences · doi:10.1016/j.ins.2023.119269 · Cited by 33 · see citing works
Finding Skill Neurons in Pre-trained Transformer-based Language Models
Wang, X., Wen, K., Zhang, Z., Hou, L., Liu, Z., & Li, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.765 · Cited by 37 · see citing works
Finetuning Pretrained Transformers into RNNs
Kasai, J., Peng, H., Zhang, Y., Yogatama, D., Ilharco, G., Pappas, N., Mao, Y., Chen, W., & Smith, N. A. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.830 · Cited by 30 · see citing works
Self-citationNormFormer: Improved Transformer Pretraining with Extra Normalization
Shleifer, S., Weston, J., & Ott, M. (2021) · arXiv · doi:10.48550/arxiv.2110.09456 · Cited by 24 · see citing works
Multi-Teacher Distillation With Single Model for Neural Machine Translation
Liang, X., Wu, L., Li, J., Qin, T., Zhang, M., & Liu, T. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3153264 · Cited by 25 · see citing works
Character-level Representations Improve DRS-based Semantic Parsing Even in the Age of BERT
van Noord, R., Toral, A., & Bos, J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.371 · Cited by 22 · see citing works
Character-based Neural Semantic Parsing
van Noord, R. (2021) · doi:10.33612/diss.169308968 · Cited by 19 · see citing works
Shen, S., Baevski, A., Morcos, A. S., Keutzer, K., Auli, M., & Kiela, D. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.331 · Cited by 18 · see citing works
GNN-LM: Language Modeling based on Global Contexts via GNNPreprint
Meng, Y., Zong, S., Li, X., Sun, X., Zhang, T., Wu, F. Q., & Li, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.08743 · Cited by 22 · see citing works
Meta-Learning the Difference: Preparing Large Language Models for Efficient Adaptation
Hou, Z., Salazar, J., & Polovets, G. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00517 · Cited by 17 · see citing works
Less is more: Pruning BERTweet architecture in Twitter sentiment analysis
Moura, R., Carvalho, J., Plastino, A., & Paes, A. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103688 · Cited by 15 · see citing works
RankNAS: Efficient Neural Architecture Search by Pairwise Ranking
Hu, C., Wang, C., Ma, X., Xia, M., Li, Y., Xiao, T., Zhu, J., & Li, C. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.191 · Cited by 12 · see citing works
Which transformer architecture fits my data? A vocabulary bottleneck in self-attentionPreprint
Wies, N., Levine, Y., Jannai, D., & Shashua, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.03928 · Cited by 12 · see citing works
Proformer: a hybrid macaron transformer model predicts expression values from promoter sequences
Kwak, I., Kim, B.-C., Lee, J., Kang, T., Garry, D. J., Zhang, J. J., & Gong, W. (2024) · BMC Bioinformatics · doi:10.1186/s12859-024-05645-5 · Cited by 10 · see citing works
Normalized Attention Without Probability CagePreprint
Richter, O., & Wattenhofer, R. P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.09561 · Cited by 12 · see citing works
Improving ROUGE ‐1 by 6%: A novel multilingual transformer for abstractive news summarization
Kumar, S., & Solanki, A. (2024) · Concurrency and Computation Practice and Experience · doi:10.1002/cpe.8199 · Cited by 9 · see citing works
Of Non-Linearity and Commutativity in BERT
Zhao, S., Pascual, D., Brunner, G., & Wattenhofer, R. P. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533563 · Cited by 9 · see citing works
Revisiting Simple Neural Probabilistic Language Models
Sun, S., & Iyyer, M. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.407 · Cited by 9 · see citing works
Attention And Positional Encoding Are (Almost) All You Need For Shape Matching
Raganato, A., Pasi, G., & Melzi, S. (2023) · Computer Graphics Forum · doi:10.1111/cgf.14912 · Cited by 9 · see citing works
Not All Attention Is All You NeedPreprint
Wu, H., Zhao, H., & Zhang, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.04692 · Cited by 13 · see citing works
Grounded Compositional Outputs for Adaptive Language Modeling
Pappas, N., Mulcaire, P., & Smith, N. A. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.96 · Cited by 8 · see citing works
Self-citationKarim, A. A. J., Asad, K. H., & Alam, M. G. R. (2025) · PLoS ONE · doi:10.1371/journal.pone.0315829 · Cited by 8 · see citing works
Detection and Prediction of Probe Mark Damage in Wafer Testing
Chang, B. R., Tsai, H. F., & Wu, Y.-R. (2024) · Electronics · doi:10.3390/electronics13204075 · Cited by 7 · see citing works
Inspecting the concept knowledge graph encoded by modern language models
Aspillaga, C., Mendoza, M., & Soto, Á. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.263 · Cited by 7 · see citing works
Undivided Attention: Are Intermediate Layers Necessary for BERT?Preprint
Sridhar, S. N., & Sarah, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.11881 · Cited by 8 · see citing works
Pay Attention when RequiredPreprint
Mandava, S., Migacz, S., & Fit-Florea, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.04534 · Cited by 5 · see citing works
Relational Pretrained Transformers towards Democratizing Data Preparation [Vision].Preprint
Tang, N., Fan, J., Li, F., Tu, J., Du, X., Li, G., Madden, S. R., & Ouzzani, M. (2020) · arXiv (Cornell University) · Cited by 4 · see citing works
IOT: Instance-wise Layer Reordering for Transformer StructuresPreprint
Zhu, J., Wu, L., Xia, Y., Xie, S., Qin, T., Zhou, W., Li, H., & Liu, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.03457 · Cited by 4 · see citing works
Genetic Algorithm-based Transformer Architecture Design for Neural Machine Translation
Wu, J., Feng, B., & Sun, Y. (2022) · International Conference on Machine Learning and Machine Intelligence (MLMI) · doi:10.1145/3568199.3568215 · Cited by 3 · see citing works
Hierarchical Topic-Aware Contextualized Transformers
Lu, R., Chen, B., Guo, D., Wang, D., & Zhou, M. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3339344 · Cited by 3 · see citing works
A Review of Semantic Annotation in the Context of the Linked Open Data Cloud
Kadhim, K. J., & Hadi, A. S. (2024) · Ingénierie des systèmes d information · doi:10.18280/isi.290330 · Cited by 3 · see citing works
Centered Self-attention Layers
Ali, A., Galanti, T., & Wolf, L. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-95911-0_5 · Cited by 2 · see citing works
Liu, W., Sun, J., Sun, Y., & Chen, C. (2022) · Electronics · doi:10.3390/electronics11101656 · Cited by 1 · see citing works
Unravelling the semantic mysteries of transformers layer by layer
Zhang, C., Lv, J., Cao, J., Sheng, J., Song, D., & Zhang, T. (2025) · The Computer Journal · doi:10.1093/comjnl/bxaf034 · Cited by 1 · see citing works
From Transformers to the Future: An In-Depth Exploration of Modern Language Model ArchitecturesPreprint
Xu, H., Bi, Z., Tseng, H.-M., Song, X., & Feng, P. (2024) · doi:10.31219/osf.io/n8r5j · Cited by 1 · see citing works
SEDM: Scalable Self-Evolving Distributed Memory for Agents
Xu, H., Hu, J., Zhang, K., Yu, L., Tang, Y., Song, X., Duan, Y., Ai, L., & Shi, B. (2025) · arXiv · doi:10.48550/arxiv.2509.09498
“In addition, the cumulative noise effect increases response latency and accelerates the nonlinear consumption of computational and storage resources [20], ultimately threatening both scalability and stability in long-term MAS operations [35].”
LV-BERT: Exploiting Layer Variety for BERT
Yu, W., Jiang, Z., Chen, F., Hou, Q., & Feng, J. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.2
RankNAS: Efficient Neural Architecture Search by Pairwise RankingPreprint
Hu, C., Wang, C., Ma, X., Xia, M., Li, Y., Xiao, T., Zhu, J., & Li, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.07383
Analysis on the number of layers in the transformer-based model for neural machine translation
Li, D., & Luo, Z. (2022) · International Conference on Computer Science and Communication Technology (ICCSCT) · doi:10.1117/12.2661982
NarrowBERT: Accelerating Masked Language Model Pretraining and Inference
Li, H., Keung, P., Cheng, D., Kasai, J., & Smith, N. A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.146
Self-citationIterative convergent computation may not be a useful inductive bias for residual neural networksPreprint
Lippl, S., Peters, B., & Kriegeskorte, N. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.10.13.562196
CSI Reconstruction in UM-MIMO Systems Based on Improved ViT and TNT Networks
Xia, Y., & Qi, B. (2023) · International Conference on Information Systems and Computing Technology (ISCTech) · doi:10.1109/isctech60480.2023.00041
Lippl, S., Peters, B., & Kriegeskorte, N. (2024) · PLoS ONE · doi:10.1371/journal.pone.0293440
DATransformer: Deep Attention Transformer
Matsumoto, R., & Kimura, M. (2024) · International Conference on Intelligent Information Technology (ICIIT) · doi:10.1145/3654522.3654560
Semantic segmentation models with frozen weights for railway track detection
Lee, S., Kim, B., & Lee, H. (2024) · Science Progress · doi:10.1177/00368504241304204
SITH: Semantic Interpreter for Transformer Hierarchy
Zhang, C., Lv, J., Cao, J., Sheng, J., Song, D., & Zhang, T. (2024) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai62512.2024.00023
An Evolutionary Neural Architecture Search-Based Approach for Time Series Forecasting
Vu, T. H., Nguyen, T. T., & Elyan, E. (2025) · IEEE Congress on Evolutionary Computation (CEC) · doi:10.1109/cec65147.2025.11043002
Semi-Supervised Prediction of Multimodal Vehicle Trajectory Based on Transformers
Sineglazov, V. M., & Lesohorskyi, K. (2025) · Cybernetics and Systems Analysis · doi:10.1007/s10559-025-00806-7
Orthogonally Regularized Autoencoder for Dual Structure Preservation in Glove Embeddings
Nekpai, A. (2026) · AI Computer Science and Robotics Technology · doi:10.5772/acrt.20250095
Classification and benchmarking of energy forecasting methods accuracy
Abramovich, V. V., & Khanova, A. (2026) · VESTNIK OF ASTRAKHAN STATE TECHNICAL UNIVERSITY SERIES MANAGEMENT COMPUTER SCIENCE AND INFORMATICS · doi:10.24143/2072-9502-2026-1-40-52
- Authors
- Press, O., Smith, N. A., & Levy, O.
- Affiliations
- University of Washington (US)
- Venue
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Published
- 2020
- Pages
- 2996-3005
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2988945824
- Cited by
- 64
- References
- 24
- Citations by year
- 2026: 2 · 2025: 5 · 2024: 10 · 2023: 8 · 2022: 11 · 2021: 20 · 2020: 8
- FWCI
- 4.49
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- transformer models, self-attention, language modeling, perplexity, machine translation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
