Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
Open accessAbstract
There remain many open questions pertaining to the scaling behaviour of Transformer architectures. These scaling decisions and findings can be critical, as training runs often come with an associated computational cost which have both financial and/or environmental impact. The goal of this paper is to present scaling insights from pretraining and finetuning Transformers. While Kaplan et al. presents a comprehensive study of the scaling behaviour of Transformer language models, the scope is only on the upstream (pretraining) loss. Therefore, it is still unclear if these set of findings transfer to downstream task within the context of the pretrain-finetune paradigm. The key findings of this paper are as follows: (1) we show that aside from only the model size, model shape matters for downstream fine-tuning, (2) scaling protocols operate differently at different compute regions, (3) widely adopted T5-base and T5-large sizes are Pareto-inefficient. To this end, we present improved scaling protocols whereby our redesigned models achieve similar downstream fine-tuning quality while having 50\% fewer parameters and training 40\% faster compared to the widely adopted T5-base model. We publicly release over 100 pretrained checkpoints of different T5 configurations to facilitate future research and analysis.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Smaller models might have just the right capacity to capture the essential patterns without being bogged down by excessive complexity.
“A possible reason can be that these smaller models might have just the right capacity to capture the essential patterns without being bogged down by excessive complexity [71, 72].”
From Exploring the Capabilities of LLMs for Code-Change-Related Tasks (2025) · cites this paper as [72]
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Self-citationBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 21,793 · see citing works
Self-citationSequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · see citing works
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Tan, M., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1905.11946 · Cited by 17,072 · see citing works
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · see citing works
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · see citing works
Unsupervised Cross-lingual Representation Learning at Scale
Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.747 · Cited by 5,637 · see citing works
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308 · see citing works
Learning Word Vectors for Sentiment Analysis
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., & Potts, C. (2011) · Cited by 3,710 · see citing works
ViViT: A Video Vision Transformer
Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lucic, M., & Schmid, C. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00676 · Cited by 2,767 · see citing works
Self-citationRevisiting Unreasonable Effectiveness of Data in Deep Learning Era
Sun, C., Shrivastava, A., Singh, S., & Gupta, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.97 · Cited by 2,484 · see citing works
MLP-Mixer: An all-MLP Architecture for Vision
Tolstikhin, I., Houlsby, N., Kolesnikov, A., Beyer, L., Zhai, X., Unterthiner, T., Yung, J., Steiner, A., Keysers, D., Uszkoreit, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.01601 · Cited by 2,087 · see citing works
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · see citing works
(2018) · International Conference on Learning Representations · Cited by 1,621 · see citing works
Reformer: The Efficient Transformer
Kitaev, N., Kaiser, Ł., & Levskaya, A. (2020) · arXiv · doi:10.48550/arxiv.2001.04451 · Cited by 1,435 · see citing works
Efficient transformers: A survey
Tay, Y., Dehghani, M., Bahri, D., & Metzler, D. (2023) · ACM Computing Surveys · doi:10.1145/3530811 · Cited by 1,267 · see citing works
Self-citationSuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019) · arXiv · doi:10.48550/arxiv.1905.00537 · Cited by 1,051 · see citing works
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Fedus, W., Zoph, B., & Shazeer, N. (2021) · arXiv · doi:10.48550/arxiv.2101.03961 · Cited by 1,005 · see citing works
Self-citationMegatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019) · arXiv · doi:10.48550/arxiv.1909.08053 · Cited by 893 · see citing works
Rethinking Attention with Performers
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.14794 · Cited by 695 · see citing works
Ex Machina: Personal Attacks Seen at Scale
Wulczyn, E., Thain, N., & Dixon, L. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052591 · Cited by 636 · see citing works
FNet: Mixing Tokens with Fourier Transforms
Lee-Thorp, J., Ainslie, J., Eckstein, I., & Ontañón, S. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.319 · Cited by 575 · see citing works
Carbon Emissions and Large Neural Network Training
Patterson, D., Gonzalez, J., Le, Q., Liang, C., Munguia, L.-M., Rothchild, D., So, D., Texier, M., & Dean, J. (2021) · arXiv · doi:10.48550/arxiv.2104.10350 · Cited by 486 · see citing works
Mesh-TensorFlow: Deep Learning for SupercomputersPreprint
Shazeer, N., Cheng, Y.-L., Parmar, N., Tran, D., Vaswani, A., Koanantakool, P., Hawkins, P. G., Lee, H., Hong, M., Young, C., et al. (2018) · Neural Information Processing Systems · doi:10.48550/arxiv.1811.02084 · Cited by 452 · see citing works
Self-citationMulti-News: A Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model
Fabbri, A., Li, I., She, T., Li, S., & Radev, D. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1102 · Cited by 449 · see citing works
Zafrir, O., Boudoukh, G., Izsak, P., & Wasserblat, M. (2019) · 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing - NeurIPS Edition (EMC2-NIPS) · doi:10.1109/emc2-nips53020.2019.00016 · Cited by 442 · see citing works
MaskGAN: Better Text Generation via Filling in the______Preprint
Fedus, W., Goodfellow, I., & Dai, A. M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.07736 · Cited by 412 · see citing works
Self-citationPerceiver IO: A General Architecture for Structured Inputs & OutputsPreprint
Jaegle, A., Borgeaud, S., Alayrac, J.-B., Doersch, C., Ionescu, C., Ding, X. D., Koppula, S., Zoran, D., Brock, A., Shelhamer, E., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.14795 · Cited by 357 · see citing works
Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification
Borkan, D., Dixon, L., Sorensen, J. S., Thain, N., & Vasserman, L. (2019) · International World Wide Web Conference (WWW) · doi:10.1145/3308560.3317593 · Cited by 335 · see citing works
So, D. R., Liang, C., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1901.11117 · Cited by 277 · see citing works
Revisiting ResNets: Improved Training and Scaling StrategiesPreprint
Bello, I., Fedus, W., Du, X., Cubuk, E. D., Srinivas, A., Lin, T.-Y., Shlens, J., & Zoph, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.07579 · Cited by 257 · see citing works
Self-citationSynthesizer: Rethinking Self-Attention in Transformer Models
Tay, Y., Bahri, D., Metzler, D., Juan, D.-C., Zhao, Z., & Zheng, C. (2020) · arXiv · doi:10.48550/arxiv.2005.00743 · Cited by 251 · see citing works
Self-citationExplaining Neural Scaling Laws
Bahri, Y., Dyer, E., Kaplan, J., Lee, J., & Sharma, U. (2021) · arXiv · doi:10.48550/arxiv.2102.06701 · Cited by 182
Explaining neural scaling laws
Bahri, Y., Dyer, E., Kaplan, J., Lee, J., & Sharma, U. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2311878121 · Cited by 181 · see citing works
Sparse Sinkhorn AttentionPreprint
Tay, Y., Bahri, D., Liu, Y., Metzler, D., & Juan, D.-C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.11296 · Cited by 162 · see citing works
Self-citationScaling Laws for Autoregressive Generative Modeling
Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.14701 · Cited by 150 · see citing works
Train Big, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers
Li, Z., Wallace, E., Shen, S., Lin, K., Keutzer, K., Klein, D., & Gonzalez, J. E. (2020) · International Conference on Machine Learning · Cited by 125 · see citing works
Charformer: Fast Character Transformers via Gradient-based Subword Tokenization
Tay, Y., Tran, V. Q., Ruder, S., Gupta, J., Chung, H. W., Bahri, D., Qin, Z., Baumgartner, S., Yu, C., & Metzler, D. (2021) · arXiv · doi:10.48550/arxiv.2106.12672 · Cited by 92 · see citing works
Self-citationM6: A Chinese Multimodal PretrainerPreprint
Lin, J., Men, R., An, Y., Zhou, C., Ding, M., Zhang, Y., Wang, P., Wang, A., Jiang, L., Jia, X., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.00823 · Cited by 83 · see citing works
Do Transformer Modifications Transfer Across Implementations and Applications?
Narang, S., Chung, H. W., Tay, Y., Fedus, L., Févry, T., Matena, M. S., Malkan, K., Fiedel, N., Shazeer, N., Lan, Z., et al. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.465 · Cited by 79 · see citing works
Self-citationLightweight and Efficient Neural Natural Language Processing with Quaternion Networks
Tay, Y., Zhang, A., Luu, A. T., Rao, J., Zhang, S., Wang, S., Fu, J., & Hui, S. C. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1145 · Cited by 70 · see citing works
Self-citationHernandez, D., Kaplan, J., Henighan, T., & McCandlish, S. (2021) · arXiv · doi:10.48550/arxiv.2102.01293 · Cited by 51 · see citing works
Dehghani, M., Tay, Y., Gritsenko, A. A., Zhao, Z., Houlsby, N., Diaz, F., Metzler, D., & Vinyals, O. (2021) · arXiv · doi:10.48550/arxiv.2107.07002 · Cited by 48 · see citing works
Self-citationStabilizing Transformers for Reinforcement Learning
Parisotto, E., Song, F., Rae, J. W., Pascanu, R., Gülçehre, Ç., Jayakumar, S. M., Jaderberg, M., Kaufman, R. L., Clark, A., Noury, S., et al. (2020) · International Conference on Machine Learning · Cited by 46 · see citing works
Zhang, A., Tay, Y., Zhang, S., Chan, A., Luu, A. T., Hui, S. C., & Fu, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.08597 · Cited by 36 · see citing works
Self-citationAre Pre-trained Convolutions Better than Pre-trained Transformers?Preprint
Tay, Y., Dehghani, M., Gupta, J. P., Bahri, D., Aribandi, V., Qin, Z., & Metzler, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.03322 · Cited by 33 · see citing works
Self-citationETC: Encoding Long and Structured Inputs in TransformersPreprint
Ainslie, J., Ontañón, S., Alberti, C., Cvicek, V., Fisher, Z., Pham, P., Ravula, A., Sanghai, S. K., Wang, Q., & Li, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.08483 · Cited by 27 · see citing works
UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark
Lourie, N., Bras, R. J. L., Bhagavatula, C., & Choi, Y. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17590 · Cited by 19 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Reproducible Scaling Laws for Contrastive Language-Image Learning
Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., & Jitsev, J. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00276 · Cited by 673 · see citing works
Scaling Up Vision-Language Pretraining for Image Captioning
Hu, X., Gan, Z., Wang, J., Yang, Z., Liu, Z., Lu, Y., & Wang, L. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01745 · Cited by 241 · see citing works
Training Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2176 · Cited by 932 · see citing works
“Since it is typically only feasible to train these large models once, accurately estimating the best model hyperparameters for a given compute budget is critical (Tay et al., 2021).”
Should You Mask 15% in Masked Language Modeling?
Wettig, A., Gao, T., Zhong, Z., & Chen, D. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.217 · Cited by 161 · see citing works
Language Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105 · see citing works
ExT5: Towards Extreme Multi-Task Scaling for Transfer LearningPreprint
Aribandi, V., Tay, Y., Schuster, T., Rao, J., Zheng, H., Mehta, S. V., Zhuang, H., Tran, V. Q., Bahri, D., Ni, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.10952 · Cited by 94 · see citing works
Self-citationLiu, J., Capurro, D., Nguyen, A., & Verspoor, K. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104466 · Cited by 66 · see citing works
Transformers for Molecular Property Prediction: Lessons Learned from the Past Five Years
Sultan, A., Sieg, J., Mathea, M., & Volkamer, A. (2024) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.4c00747 · Cited by 67 · see citing works
Constructing and analyzing domain-specific language model for financial text mining
Suzuki, M., Sakaji, H., Hirano, M., & Izumi, K. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103194 · Cited by 63 · see citing works
A Review of Current Trends, Techniques, and Challenges in Large Language Models (LLMs)
Patil, R., & Gudivada, A. V. (2024) · doi:10.20944/preprints202402.0357.v1 · Cited by 296 · see citing works
Scaling Data-Constrained Language Models
Muennighoff, N., Rush, A., Barak, B., Scao, T. L., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., & Raffel, C. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2191 · Cited by 47 · see citing works
InstaNovo enables diffusion-powered de novo peptide sequencing in large-scale proteomics experiments
Eloff, K., Kalogeropoulos, K., Mabona, A., Morell, O., Catzel, R., Rivera‐de‐Torre, E., Jespersen, J. B., Williams, W., van Beljouw, S. P. B., Skwark, M. J., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-025-01019-5 · Cited by 47 · see citing works
The Efficiency MisnomerPreprint
Dehghani, M., Arnab, A., Beyer, L., Vaswani, A., & Tay, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.12894 · Cited by 58 · see citing works
Self-citationExploring the Limits of Large Scale Pre-trainingPreprint
Abnar, S., Dehghani, M., Neyshabur, B., & Sedghi, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.02095 · Cited by 56 · see citing works
Self-citationExploring the Capabilities of LLMs for Code-Change-Related Tasks
Fan, L., Liu, J., Liu, Z., Lo, D., Xia, X., & Li, S. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3709358 · Cited by 41 · see citing works
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?
Tay, Y., Dehghani, M., Abnar, S., Chung, H., Fedus, W., Rao, J., Narang, S., Tran, V., Yogatama, D., & Metzler, D. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.825 · Cited by 34 · see citing works
Self-citationSCENIC: A JAX Library for Computer Vision Research and Beyond
Dehghani, M., Gritsenko, A. A., Arnab, A., Minderer, M., & Tay, Y. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.02070 · Cited by 34 · see citing works
Self-citationWhat Language Model to Train if You Have One Million GPU Hours?
Le Scao, T., Wang, T., Hesslow, D., Bekman, S., Bari, M. S., Biderman, S., Elsahar, H., Muennighoff, N., Phang, J., Press, O., et al. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.54 · Cited by 33 · see citing works
Scaling Data-Constrained Language Models
Muennighoff, N., Rush, A. M., Barak, B., Scao, T. L., Piktus, A., Tazi, N., Pyysalo, S., Wolf, T., & Raffel, C. (2023) · arXiv · doi:10.48550/arxiv.2305.16264 · Cited by 49 · see citing works
Multimodal large language models for inclusive collaboration learning tasks
Lewis, A. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Student Research Workshop · doi:10.18653/v1/2022.naacl-srw.26 · Cited by 30 · see citing works
Neural Data Transformer 2: Multi-context Pretraining for Neural Spiking Activity
Ye, J., Collinger, J. L., Wehbe, L., & Gaunt, R. (2023) · neural information processing systems · doi:10.52202/075280-3523 · Cited by 34 · see citing works
GAIA-1: A Generative World Model for Autonomous Driving
Hu, A., Russell, L., Yeo, H., Murez, Z., Fedoseev, G., Kendall, A., Shotton, J., & Corrado, G. (2023) · arXiv · doi:10.48550/arxiv.2309.17080 · Cited by 28 · see citing works
“Transferring the scaling principles from large language models to the visual domain holds the potential for delivering consistent and expected performance improvements [92, 93, 43, 16, 94].”
BackgroundBioReader: a Retrieval-Enhanced Text-to-Text Transformer for Biomedical Literature
Frisoni, G., Mizutani, M., Moro, G., & Valgimigli, L. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.390 · Cited by 27 · see citing works
Transcending Scaling Laws with 0.1% Extra Compute
Tay, Y., Wei, J., Chung, H. W., Tran, V., So, D. R., Shakeri, S., Garcia, X., Zheng, S., Rao, J., Chowdhery, A., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.91 · Cited by 41 · see citing works
Self-citationTraining Compute-Optimal Protein Language ModelsPreprint
Cheng, X., Chen, B., Li, P., Gong, J., Tang, J., & Song, L. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.06.06.597716 · Cited by 21 · see citing works
Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing
Qiu, L., Shaw, P. M., Pasupat, P., Shi, T., Herzig, J., Pitler, E., Sha, F., & Toutanova, K. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.624 · Cited by 27 · see citing works
Energy and Carbon Considerations of Fine-Tuning BERT
Wang, X., Na, C., Strubell, E., Friedler, S. A., & Luccioni, A. S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.607 · Cited by 24 · see citing works
NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient FrameworkPreprint
Yao, X., Zheng, Y., Yang, X., & Yang, Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.04130 · Cited by 23 · see citing works
Ge, J., Zhang, X., Zheng, Y., Guo, K., & Liang, J. (2025) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2025.05.002 · Cited by 20 · see citing works
Chun, J., & Elkins, K. (2023) · International Journal of Digital Humanities · doi:10.1007/s42803-023-00069-8 · Cited by 19 · see citing works
Fares, I. A., & Elaziz, M. A. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113351 · Cited by 18 · see citing works
Improving Compositional Generalization with Self-Training for Data-to-Text Generation
Mehta, S. V., Rao, J., Tay, Y., Kale, M., Parikh, A. P., & Strubell, E. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.289 · Cited by 18 · see citing works
Self-citationFarhah, N. S., Adnan, M., Alqarni, A., Uddin, M. I., & Aldhyani, T. H. H. (2025) · IEEE Access · doi:10.1109/access.2025.3560863 · Cited by 15 · see citing works
Medical Reports Summarization Using Text-To-Text Transformer
Helwan, A., Azar, D., & Ozsahin, D. U. (2023) · Advances in Science and Engineering Technology International Conferences (ASET) · doi:10.1109/aset56582.2023.10180671 · Cited by 17 · see citing works
Jack and Masters of all Trades: One-Pass Learning Sets of Model Sets From Large Pre-Trained Models
Choong, H. X., Ong, Y.-S., Gupta, A., Chen, C., & Lim, R. (2023) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2023.3277769 · Cited by 16 · see citing works
Training Trajectories of Language Models Across Scales
Xia, M., Artetxe, M., Zhou, C., Lin, X. V., Pasunuru, R., Chen, D., Zettlemoyer, L., & Stoyanov, V. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.767 · Cited by 15 · see citing works
Mueller, A., & Linzen, T. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.629 · Cited by 14 · see citing works
How Predictable Are Large Language Model Capabilities? A Case Study on BIG-bench
Ye, Q., Fu, H. Y., Ren, X., & Jia, R. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.503 · Cited by 11 · see citing works
Enhancing Code Language Models for Program Repair by Curricular Fine-tuning Framework
Hao, S., Shi, X., Liu, H., & Shu, Y. (2023) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme58846.2023.00024 · Cited by 10 · see citing works
Scaling Laws Under the Microscope: Predicting Transformer Performance from Small Scale Experiments
Ivgi, M., Carmon, Y., & Berant, J. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.544 · Cited by 11 · see citing works
The Framework Tax: Disparities Between Inference Efficiency in NLP Research and Deployment
Fernandez, J., Kahn, J., Na, C., Bisk, Y., & Strubell, E. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.98 · Cited by 8 · see citing works
Cluster-Guided Label Generation in Extreme Multi-Label Classification
Jung, T., Kim, J.-K., Lee, S., & Kang, D. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.122 · Cited by 10 · see citing works
Sparse Mixers: Combining MoE and Mixing to build a more efficient BERT
Lee-Thorp, J., & Ainslie, J. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.5 · Cited by 7 · see citing works
Pre-training LLMs using human-like development data corpus
Bhardwaj, K., Shah, R. S., & Varma, S. (2023) · BabyLM Challenge · doi:10.18653/v1/2023.conll-babylm.30 · Cited by 7 · see citing works
Zhuo, Y., Zhang, E., Yu, X., Pachpande, A., Fang, W., Chen, X., Kirkpatrick, A. W., Couperus, K. S., Colombo, C. J., Tran, O., et al. (2025) · Scientific Data · doi:10.1038/s41597-025-06365-y · Cited by 5 · see citing works
Multi-agent learning for data-driven air traffic management applications
Deng, C., Choi, H.-C., Park, H., & Hwang, I. (2025) · Journal of Air Transport Management · doi:10.1016/j.jairtraman.2025.102843 · Cited by 4 · see citing works
Honey, I Shrunk the Language: Language Model Behavior at Reduced Scale.
Deshpande, V., Pechi, D., Thatte, S., Lialin, V., & Rumshisky, A. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.326 · Cited by 5 · see citing works
Generative Approach for Gender-Rewriting Task with ArabicT5
Alrowili, S., & Shanker, V. K. (2022) · Arabic Natural Language Processing Workshop (WANLP) · doi:10.18653/v1/2022.wanlp-1.55 · Cited by 4 · see citing works
Exploring the Potential of Pre-Trained Language Models of Code for Automated Program Repair
Hao, S., Shi, X., & Liu, H. (2024) · Electronics · doi:10.3390/electronics13071200 · Cited by 3 · see citing works
Leveraging large language models for spelling correction in Turkish
Turhan, C. G. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.2889 · Cited by 3 · see citing works
Ko, J., Park, S., Kim, Y., Ahn, S., Chang, D.-S., Ahn, E., & Yun, S.-Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.404 · Cited by 5 · see citing works
Location-Aware Visual Question Generation with Lightweight Models
Suwono, N. C., Chen, J., Hung, T. M., Huang, T.-H., Liao, I.-B., Li, Y., Ku, L., & Sun, S.-H. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.88 · Cited by 3 · see citing works
Next-Value Prediction of Beam Waists From Mixed Pitch Grating Using Simplified Transformer Model
Lim, Y. D., Zhao, P., Guidoni, L., Likforman, J.-P., & Tan, C. S. (2024) · IEEE photonics journal · doi:10.1109/jphot.2024.3442169 · Cited by 3 · see citing works
A Knowledge storage and semantic space alignment Method for Multi-documents dialogue generation
Zhu, M., Li, B., Weng, Y., & Xia, F. (2022) · DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering (DialDoc) · doi:10.18653/v1/2022.dialdoc-1.14 · Cited by 3 · see citing works
Fine-Tuning Strategies for Transformers
Pake, T. U., & Pachareney, U. (2025) · International Conference on Sentiment Analysis and Deep Learning (ICSADL) · doi:10.1109/icsadl65848.2025.10933255 · Cited by 2 · see citing works
Modeling and Classification of Optical Beam Profiles Using Vision Transformer
Lim, Y. D., & Tan, C. S. (2025) · Journal of Lightwave Technology · doi:10.1109/jlt.2025.3537677 · Cited by 2 · see citing works
TTIC’s Submission to WMT-SLT 23
Sandoval-Castañeda, M., Li, Y., Shi, B., Brentari, D., Livescu, K., & Shakhnarovich, G. (2023) · Conference on Machine Translation (WMT) · doi:10.18653/v1/2023.wmt-1.35 · Cited by 2 · see citing works
Adaptive Vision-Language Prompt Learners for Learning with Noisy Labels
Hu, C., Nagarajan, B., Marques, R., & Radeva, P. I. (2025) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2025.104550 · Cited by 1 · see citing works
Using Artificial Intelligence Systems in Modern Neurobehavioral Research
Apukhtin, K. V., Zolotova, A. E., Leunenko, A. O., Perfilova, V. N., & Kalueff, A. V. (2025) · Journal of Evolutionary Biochemistry and Physiology · doi:10.1134/s0022093025060080 · Cited by 1 · see citing works
Campos, D., & Zhai, C. (2023) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2023.sustainlp-1.6 · Cited by 1 · see citing works
Nikolaev, D., & Padó, S. (2023) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2023.blackboxnlp-1.11 · Cited by 4 · see citing works
Jin, S., Wang, R., Zahid, M., & Pokorny, F. T. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10802583 · Cited by 1 · see citing works
Towards AI-Assisted Inclusive Language Writing in Italian Formal Communications
Greco, S., Quatra, M. L., Cagliero, L., & Cerquitelli, T. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3729237 · Cited by 1 · see citing works
The Shared Task on Gender Rewriting
Alhafni, B., Habash, N. Y., Bouamor, H., Obeid, O., Alrowili, S., Alzeer, D. H., Alshanqiti, K. M., Elbakry, A., ElNokrashy, M., Gabr, M., et al. (2022) · Arabic Natural Language Processing Workshop (WANLP) · doi:10.18653/v1/2022.wanlp-1.10 · Cited by 2 · see citing works
Exploring the Capabilities of LLMs for Code Change Related Tasks
Fan, L., Liu, J., Liu, Z., Lo, D., Xia, X., & Li, S. (2024) · arXiv · Cited by 41 · see citing works
Multilingual Pixel Representations for Translation and Effective Cross-lingual Transfer
Salesky, E., Verma, N., Koehn, P., & Post, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.854
Generate to Understand for Representation in One Pre-training Stage
Xue, C., Zhong, X., & Liu, X. (2023) · International Conference on Information Systems and Computing Technology (ISCTech) · doi:10.1109/isctech60480.2023.00054
A Large Financial Chat ModelPreprint
Ding, F. (2024) · doi:10.33774/coe-2024-lsc94
Robustness Analysis of Maryland Watermark to Paraphrasing Attacks Under Advanced Techniques
Kumar, R. (2025) · Asian Journal of Computer Science Engineering · doi:10.22377/ajcse.v10i2.213
- Authors
- Tay, Y., Dehghani, M., Rao, J., Fedus, W., Abnar, S., Chung, H. W., Narang, S., Yogatama, D., Vaswani, A., & Metzler, D.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2021
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2109.10686
- OpenAlex
- W3199241049
- Cited by
- 63
- References
- 50
- Citations by year
- 2025: 13 · 2024: 6 · 2023: 26 · 2022: 14 · 2021: 4
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- pre-training, fine-tuning, model size, Pareto optimality, T5, parameter efficiency, training efficiency, pretrained checkpoints
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
