The DOI matches its Crossref record; 84 of the 84 references it cites were found in a registry or an index.
A Review of Current Trends, Techniques, and Challenges in Large Language Models (LLMs)
Open accessPublished version of a preprint
Abstract
Natural language processing (NLP) has significantly transformed in the last decade, especially in the field of language modeling. Large language models (LLMs) have achieved SOTA performances on natural language understanding (NLU) and natural language generation (NLG) tasks by learning language representation in self-supervised ways. This paper provides a comprehensive survey to capture the progression of advances in language models. In this paper, we examine the different aspects of language models, which started with a few million parameters but have reached the size of a trillion in a very short time. We also look at how these LLMs transitioned from task-specific to task-independent to task-and-language-independent architectures. This paper extensively discusses different pretraining objectives, benchmarks, and transfer learning methods used in LLMs. It also examines different finetuning and in-context learning techniques used in downstream tasks. Moreover, it explores how LLMs can perform well across many domains and datasets if sufficiently trained on a large and diverse dataset. Next, it discusses how, over time, the availability of cheap computational power and large datasets have improved LLM’s capabilities and raised new challenges. As part of our study, we also inspect LLMs from the perspective of scalability to see how their performance is affected by the model’s depth, width, and data size. Lastly, we provide an empirical comparison of existing trends and techniques and a comprehensive analysis of where the field of LLM currently stands.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Enriching Word Vectors with Subword Information
Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00051 · Cited by 9,645
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 6×
“Specifically, given a 10 × increase in computational budget, Kaplan et al. [97] suggest that the model size should increase 5.5 × while the number of training tokens should only increase 1.8 × .”
BackgroundMethodInfluentialIndependent component analysis: algorithms and applications
Hyvärinen, A., & Oja, E. (2000) · Neural Networks · doi:10.1016/s0893-6080(00)00026-5 · Cited by 8,705
A vector space model for automatic indexing
Salton, G., Wong, A., & Yang, C. S. (1975) · Communications of the ACM · doi:10.1145/361219.361220 · Cited by 7,521 · cited 1×
“To address these issues, statistical methods such as one-hot encoding [1], bag of words, N-grams [2], term frequency [3], and inverse document frequency [4,5] were proposed.”
MethodUnsupervised Cross-lingual Representation Learning at Scale
Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.747 · Cited by 5,637
Distributed Representations of Sentences and Documents
Le, Q. V., & Mikolov, T. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1405.4053 · Cited by 4,934
Harris, Z. S. (1954) · WORD · doi:10.1080/00437956.1954.11659520 · Cited by 4,807
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
A STATISTICAL INTERPRETATION OF TERM SPECIFICITY AND ITS APPLICATION IN RETRIEVAL
SPARCK JONES, K. (1972) · Journal of Documentation · doi:10.1108/eb026526 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · arXiv · doi:10.48550/arxiv.2104.08691 · Cited by 3,092
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Li, X. L., & Liang, P. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.353 · Cited by 2,473
A statistical approach to machine translation
Brown, P. F., Cocke, J., Pietra, S. A. D., Pietra, V. J. D., Jelinek, F., Lafferty, J., Mercer, R. L., & Roossin, P. S. (1990) · doi:10.5555/92858.92860 · Cited by 1,655
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer
Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., & Raffel, C. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.41 · Cited by 1,629
ERNIE: Enhanced Language Representation with Informative Entities
Zhang, Z., Han, X., Liu, Z., Jiang, X., Sun, M., & Liu, Q. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1139 · Cited by 1,482
Scaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441
Emergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264
Self-Instruct: Aligning Language Models with Self-Generated Instructions
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., & Hajishirzi, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.754 · Cited by 1,243
Unified Language Model Pre-training for Natural Language Understanding and Generation
Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, Y., Gao, J., Zhou, M., & Hon, H.-W. (2019) · arXiv · doi:10.48550/arxiv.1905.03197 · Cited by 1,225
Finetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101
Competition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043
Competition-Level Code Generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A. D., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.07814 · Cited by 1,043
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Fedus, W., Zoph, B., & Shazeer, N. (2021) · arXiv · doi:10.48550/arxiv.2101.03961 · Cited by 1,005
GLM: General Language Model Pretraining with Autoregressive Blank Infilling
Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., & Tang, J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.26 · Cited by 955
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., & Dean, J. (2017) · arXiv · doi:10.48550/arxiv.1701.06538 · Cited by 933
Training Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.15556 · Cited by 929
TruthfulQA: Measuring How Models Mimic Human Falsehoods
Lin, S., Hilton, J., & Evans, O. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.229 · Cited by 904 · cited 2×
“It was observed in [102] that, generally, the largest models were the least truthful, so scaling up the model size increased performance but was less promising in improving the model’s truthfulness.”
BackgroundLaMDA: Language Models for Dialog Applications
Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.08239 · Cited by 778 · cited 2×
“…- - FastGELU, Adam finetuning FLAN [44] -, 8192 1024 30 K -, Adafactor Instruction Tuning, zero-shot InstructGPT [45] 3.2 M 2048 - - RLHF LaMDA [46] - - - gated-GELU, FT T0 [47] - - - RELU, AdaFactor FT, zero-shot GPT NeoX 20B [48] 3.15 M tokens 2048 150 K steps -, AdamW with ZeRO few-shot OPT…”
BackgroundInfluentialGLaM: Efficient Scaling of Language Models with Mixture-of-ExpertsPreprint
Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A. W., Fırat, O., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.06905 · Cited by 708
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.301 · Cited by 693
KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation
Wang, X., Gao, T., Zhu, Z., Zhang, Z., Liu, Z., Li, J., & Tang, J. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00360 · Cited by 676 · cited 1×
“KEPLER [73] jointly optimized the language modeling and knowledge embedding (KEs) objective.”
MethodKnowledge Enhanced Contextual Word Representations
Peters, M. E., Neumann, M., Logan, R., Schwartz, R., Joshi, V., Singh, S., & Smith, N. A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1005 · Cited by 661
StereoSet: Measuring stereotypical bias in pretrained language models
Nadeem, M., Bethke, A., & Reddy, S. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.416 · Cited by 643
Multitask Prompted Training Enables Zero-Shot Task Generalization
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.08207 · Cited by 612
Galactica: A Large Language Model for Science
Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E., Poulton, A., Kerkez, V., & Stojnic, R. (2022) · arXiv · doi:10.48550/arxiv.2211.09085 · Cited by 605
Liu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., & Tang, J. (2024) · AI Open · doi:10.1016/j.aiopen.2023.08.012 · Cited by 596
Computer Evaluation of Indexing and Text Processing
Salton, G., & Lesk, M. E. (1968) · Journal of the ACM · doi:10.1145/321439.321441 · Cited by 580
Learned in Translation: Contextualized Word Vectors
McCann, B., Bradbury, J., Xiong, C., & Socher, R. (2017) · arXiv · doi:10.48550/arxiv.1708.00107 · Cited by 572 · cited 1×
“The dynamic embed-dings, such as C2V [15], CoVe [16], ELMo [17], ULMFiT [18], UNILM [19], etc., being context-sensitive, were able to address the polysemy aspect of words.”
BackgroundCodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X
Zheng, Q., Xia, X., Zou, X., Dong, Y., Wang, S., Xue, Y., Shen, L., Wang, Z., Wang, A., Li, Y., et al. (2023) · Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3580305.3599790 · Cited by 563
What Makes Good In-Context Examples for GPT-3?
Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., & Chen, W. (2022) · Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures · doi:10.18653/v1/2022.deelio-1.10 · Cited by 563
Learning to summarize from human feedback
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., & Christiano, P. (2020) · arXiv · doi:10.48550/arxiv.2009.01325 · Cited by 550
Few-shot Learning with Multilingual Generative Language Models
Lin, X. V., Mihaylov, T., Artetxe, M., Wang, T., Chen, S., Simig, D., Ott, M., Goyal, N., Bhosale, S., Du, J., et al. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.616 · Cited by 529
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Zhao, T. Z., Wallace, E., Feng, S., Klein, D., & Singh, S. (2021) · arXiv · doi:10.48550/arxiv.2102.09690 · Cited by 513
context2vec: Learning Generic Context Embedding with Bidirectional LSTM
Melamud, O., Goldberger, J., & Dagan, I. (2016) · Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning · doi:10.18653/v1/k16-1006 · Cited by 513
Carbon Emissions and Large Neural Network Training
Patterson, D., Gonzalez, J., Le, Q., Liang, C., Munguia, L.-M., Rothchild, D., So, D., Texier, M., & Dean, J. (2021) · arXiv · doi:10.48550/arxiv.2104.10350 · Cited by 486
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
Wang, R., Tang, D., Duan, N., Wei, Z., Huang, X., Ji, J., Cao, G., Jiang, D., & Zhou, M. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.121 · Cited by 456
CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models
Nangia, N., Vania, C., Bhalerao, R., & Bowman, S. R. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.154 · Cited by 446
Gender Bias in Coreference Resolution
Rudinger, R., Naradowsky, J., Leonard, B., & Van Durme, B. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2002 · Cited by 443 · cited 1×
“Second, it evaluates the Winogender coreference resolution dataset [103], where similar coreference accuracy across different pronouns indicates less gender bias.”
MethodGPT-NeoX-20B: An Open-Source Autoregressive Language Model
Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., et al. (2022) · Proceedings of BigScience Episode #5 -- Workshop on Challenges & Perspectives in Creating Large Language Models · doi:10.18653/v1/2022.bigscience-1.9 · Cited by 404
Crosslingual Generalization through Multitask Finetuning
Muennighoff, N., Wang, T., Sutawika, L., Roberts, A., Biderman, S., Le Scao, T., Bari, M. S., Shen, S., Yong, Z. X., Schoelkopf, H., et al. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.891 · Cited by 361
Solving Quantitative Reasoning Problems with Language Models
Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.14858 · Cited by 358 · cited 2×
“…GPT NeoX 20B [48] 3.15 M tokens 2048 150 K steps -, AdamW with ZeRO few-shot OPT [49] 2 M tokens 2048 - ReLU, AdamW few-shot, zero-shot MINERVA [50] - 1024 399 K SwiGLU, Adafactor few-shot, chain-of-thought context AlexaTM 20 B [51] 2 million tokens - - Adam Finetuning, few-shot, one-shot,…”
BackgroundMethodInfluential(2022) · doi:10.52202/068431-0278 · Cited by 358
Efficient Large Scale Language Modeling with Mixtures of Experts
Artetxe, M., Bhosale, S., Goyal, N., Mihaylov, T., Ott, M., Shleifer, S., Lin, X. V., Du, J., Iyer, S. V., Pasunuru, R., et al. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.804 · Cited by 290
GLM-130B: An Open Bilingual Pre-trained Model
Zeng, A., Liu, X., Du, Z., Wang, Z., Lai, H., Ding, M., Yang, Z., Xu, Y., Zheng, W., Xia, X., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.02414 · Cited by 281
Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation
Mosbach, M., Pimentel, T., Ravfogel, S., Klakow, D., & Elazar, Y. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.779 · Cited by 275
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
Wang, Y., Mishra, S., Alipoormolabashi, P., Kordi, Y., Mirzaei, A., Naik, A., Ashok, A., Dhanasekaran, A. S., Arunkumar, A., Stap, D., et al. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.340 · Cited by 247
Efficient Large Scale Language Modeling with Mixtures of Experts
Artetxe, M., Bhosale, S., Goyal, N., Mihaylov, T., Ott, M., Shleifer, S., Lin, X. V., Du, J., Iyer, S., Pasunuru, R., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.10684 · Cited by 223
ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
Sun, Y., Wang, S., Feng, S., Ding, S., Pang, C., Shang, J., Liu, J., Chen, X., Zhao, Y., Lu, Y., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.02137 · Cited by 186
CoLAKE: Contextualized Language and Knowledge Embedding
Sun, T., Shao, Y., Qiu, X., Guo, Q., Hu, Y., Huang, X., & Zhang, Z. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.327 · Cited by 185
Zeng, W., Ren, X., Su, T., Wang, H., Liao, Y., Wang, Z., Jiang, X., Yang, Z., Wang, K., Zhang, X., et al. (2021) · arXiv · doi:10.48550/arxiv.2104.12369 · Cited by 172 · cited 7×
“As demonstrated in [33], perplexity decreases with the increase in model capacity, training data, and computational resources.”
BackgroundMethodInfluentialKim, B., Kim, H., Lee, S., Lee, G., Kwak, D., Hyeon, J. D., Park, S., Kim, S., Kim, S., Seo, D., et al. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.274 · Cited by 170
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling
Biderman, S., Schoelkopf, H., Anthony, Q., Bradley, H., O'Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.01373 · Cited by 160
CPM-2: Large-scale cost-effective pre-trained language models
Zhang, Z., Gu, Y., Han, X., Chen, S., Xiao, C., Sun, Z., Yao, Y., Qi, F., Guan, J., Ke, P., et al. (2021) · AI Open · doi:10.1016/j.aiopen.2021.12.003 · Cited by 120
Transcending Scaling Laws with 0.1% Extra Compute
Tay, Y., Wei, J., Chung, H. W., Tran, V., So, D. R., Shakeri, S., Garcia, X., Zheng, S., Rao, J., Chowdhery, A., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.91 · Cited by 106
Athiwaratkun, B., & Wilson, A. (2017) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p17-1151 · Cited by 98
Comparing and Combining Dimension Reduction Techniques for Efficient Text Clustering
Tang, B., Shepherd, M. A., Milios, E., & Heywood, M. I. (2005) · Cited by 85
Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation
Mosbach, M., Pimentel, T., Ravfogel, S., Klakow, D., & Elazar, Y. (2023) · arXiv · doi:10.48550/arxiv.2305.16938 · Cited by 81
SaFeRDialogues: Taking Feedback Gracefully after Conversational Safety Failures
Ung, M., Xu, J., & Boureau, Y.-L. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.447 · Cited by 32
(2020) · arXiv (Cornell University)
Cited by
Multi-Modal LLMs in Agriculture: A Comprehensive Review
Sapkota, R., Qureshi, R., Hassan, S. Z., Shutske, J., Shoman, M., Sajjad, M., Dharejo, F. A., Paudel, A., Li, J., Meng, Z., et al. (2024) · doi:10.36227/techrxiv.172651082.24507804/v1 · Cited by 68
Lamaakal, I., Maleh, Y., El Makkaoui, K., Ouahbi, I., Pławiak, P., Alfarraj, O., Almousa, M., & Abd El-Latif, A. A. (2025) · Sensors · doi:10.3390/s25051318 · Cited by 60
Large language models: an overview of foundational architectures, recent trends, and a new taxonomy
Mienye, I. D., Jere, N., Obaido, G., Ogunruku, O. O., Esenogho, E., & Modisane, C. (2025) · Discover Applied Sciences · doi:10.1007/s42452-025-07668-w · Cited by 46
Kabir, S., Hossain, M. S., & Andersson, K. (2025) · Algorithms · doi:10.3390/a18090556 · Cited by 44
Galli, C., Gavrilova, A. V., & Calciolari, E. (2025) · Information · doi:10.3390/info16050378 · Cited by 36
Metadata
- Authors
- Patil, R., & Gudivada, V.
- Venue
- Applied Sciences
- Publisher
- MDPI AG
- Published
- 2024-03-01
- Volume
- 14
- Issue
- 5
- Pages
- 2074
- ISSN
- 2076-3417 (ISSN-L)
- OpenAlex
- W4392358084
- Cited by
- 290
- References
- 84
- FWCI
- 43.16
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.