The DOI matches its Crossref record; 185 of the 185 references it cites were found in a registry or an index.
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey
Open accessAbstract
Large, pre-trained language models (PLMs) such as BERT and GPT have drastically changed the Natural Language Processing (NLP) field. For numerous NLP tasks, approaches leveraging PLMs have achieved state-of-the-art performance. The key idea is to learn a generic, latent representation of language from a generic task once, then share it across disparate NLP tasks. Language modeling serves as the generic task, one with abundant self-supervised text available for extensive training. This article presents the key fundamental concepts of PLM architectures and a comprehensive view of the shift to PLM-driven NLP techniques. It surveys work applying the pre-training then fine-tuning, prompting, and text generation approaches. In addition, it discusses PLM limitations and suggested directions for future research.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“However, the field of AI is evolving rapidly, particularly in areas such as Generative AI (Brynjolfsson et al. 2023) and Large Language Models (Min et al. 2023).”
From Artificial intelligence for literature reviews: opportunities and challenges (2024) · cites this paper as Min et al. 2023
In the citing paper
“However, the open-ended nature of language means that there can be multiple valid responses to a given prompt, complicating the assessment of correctness [1].”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [1]
In the citing paper
“LLMs like Llama and GPT-4 are capable of understanding and generating human-like text, performing tasks ranging from translation to creative writing [1], [4].”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [1]
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,887
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Research works printed but not linked: 56
- Zeyuan Allen-Zhu and Yuanzhi Li. 2021. Towards understanding ensemble knowledge distillation and self-distillation in deep learning. https://arxiv.org/abs/2012.09816 · arXiv
- Asaf Amrami and Yoav Goldberg. 2019. Towards better substitution-based word sense induction. https://arxiv.org/abs/1905.12598 · arXiv
- Stephen H. Bach Victor Sanh Zheng-Xin Yong Albert Webson Colin Raffel Nihal V. Nayak Abheesht Sharma Taewoon Kim M. Saiful Bari Thibault Fevry Zaid Alyafeai Manan Dey Andrea Santilli Zhiqing Sun Srulik Ben-David Canwen Xu Gunjan Chhablani Han Wang Jason Alan Fries Maged S. Al-shaibani Shanya Sharma Urmish Thakker Khalid Almubarak Xiangru Tang Dragomir Radev Mike Tian-Jian Jiang and Alexander M. Rush. 2022. PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts. https://arxiv.org/abs/2202.01279 · arXiv
- Geoff Bacon and Terry Regier. 2019. Does BERT agree? Evaluating knowledge of structure dependence through agreement relations. https://arxiv.org/abs/1908.09892 · arXiv
- Jack Bandy and Nicholas Vincent. 2021. Addressing “Documentation Debt” in Machine Learning Research: A Retrospective Datasheet for BookCorpus. https://arxiv.org/abs/2105.05241 · arXiv
- Eyal Ben-David Nadav Oved and Roi Reichart. 2021. PADA: A prompt-based autoregressive approach for adaptation to unseen domains. https://arxiv.org/abs/2102.12206 · arXiv
- Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota.
- 10.5555/2463101 · doi:10.5555/2463101
- Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. 2017. Language modeling with gated convolutional networks. In Proceedings of the International Conference on Machine Learning. PMLR, 933–941.
- Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021. Autoregressive entity retrieval. In Proceedings of the 9th International Conference on Learning Representations (ICLR’21).
- Avia Efrat and Omer Levy. 2020. The turking test: Can language models understand instructions?https://arxiv.org/abs/2010.11982 · arXiv
- Tom Brown et al.2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 1877–1901.
- Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations.
- Steven Fincke Shantanu Agarwal Scott Miller and Elizabeth Boschee. 2021. Language model priming for cross-lingual event extraction. https://arxiv.org/abs/2109.12383 · arXiv
- Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020. Online.
- Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning. PMLR.
- Minyoung Huh Pulkit Agrawal and Alexei A. Efros. 2016. What makes ImageNet good for transfer learning?arXiv: 1608.08614. · arXiv
- Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. UNIFIEDQA: Crossing format boundaries with a single QA system. In Findings of the Association for Computational Linguistics: EMNLP 2020.
- Sawan Kumar and Partha Talukdar. 2021. Reordering examples helps during priming-based few-shot learning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. Association for Computational Linguistics, Online.
- Huiyuan Lai Antonio Toral and Malvina Nissim. 2021. Thank you BART! Rewarding pre-trained models improves formality style transfer. https://arxiv.org/abs/2105.06947 · arXiv
- Fayuan Li, Weihua Peng, Yuguang Chen, Quan Wang, Lu Pan, Yajuan Lyu, and Yong Zhu. 2020. Event extraction as multi-turn question answering. In Findings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics, Online.
- Junyi Li Tianyi Tang Wayne Xin Zhao and Ji-Rong Wen. 2021. Pretrained language models for text generation: A survey. https://arxiv.org/abs/2105.10311 · arXiv
- Jiachang Liu Dinghan Shen Yizhe Zhang Bill Dolan Lawrence Carin and Weizhu Chen. 2021. What makes good in-context examples for GPT-3?. https://arxiv.org/abs/2101.06804 · arXiv
- Pengfei Liu Weizhe Yuan Jinlan Fu Zhengbao Jiang Hiroaki Hayashi and Graham Neubig. 2021. Pre-train prompt and predict: A systematic survey of prompting methods in natural language processing. arXiv: 2107.13586. · arXiv
- Xiao Liu Yanan Zheng Zhengxiao Du Ming Ding Yujie Qian Zhilin Yang and Jie Tang. 2021. GPT understands too. https://arxiv.org/abs/2103.10385 · arXiv
- Brian MacWhinney. 2000. The CHILDES Project: Tools for Analyzing Talk. Transcription Format and Programs. Vol. 1. Psychology Press.
- Xuan-Phi Nguyen Shafiq Joty Steven C. H. Hoi and Richard Socher. 2020. Tree-structured attention with hierarchical accumulation. https://arxiv.org/abs/2002.08046 · arXiv
- Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020. Document ranking with a pretrained sequence-to-sequence model. In Findings of the Association for Computational Linguistics (EMNLP).
- Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, Rishita Anubhai, Cicero dos Santos Nogueira, Bing Xiang, and Stefano Soatto. 2021. Structured prediction as translation between augmented natural languages. In Proceedings of the 9th International Conference on Learning Representations (ICLR’21).
- Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. 2020. How context affects language models’ factual predictions. In Proceedings of AKBC.
- Raul Puri and Bryan Catanzaro. 2019. Zero-shot text classification with generative language models. https://arxiv.org/abs/1912.10165 · arXiv
- Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. OpenAI Blog (2018), 12.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.
- Evani Radiya-Dixit and Xin Wang. 2020. How fine can fine-tuning be? Learning efficient language models. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics(Proceedings of Machine Learning Research, Vol. 108), Silvia Chiappa and Roberto Calandra (Eds.). 2435–2443.
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research (2020).
- Liliang Ren, Chenkai Sun, Heng Ji, and Julia Hockenmaier. 2021. HySPA: Hybrid span generation for scalable text-to-graph extraction. In Findings of the Association for Computational Linguistics (ACL-IJCNLP’21).
- Victor Sanh Lysandre Debut Julien Chaumond and Thomas Wolf. 2020. DistilBERT a distilled version of BERT: Smaller faster cheaper and lighter. https://arxiv.org/abs/1910.01108 · arXiv
- Victor Sanh Albert Webson Colin Raffel Stephen H. Bach Lintang Sutawika Zaid Alyafeai Antoine Chaffin Arnaud Stiegler Teven Le Scao Arun Raja Manan Dey M. Saiful Bari Canwen Xu Urmish Thakker Shanya Sharma Sharma Eliza Szczechla Taewoon Kim Gunjan Chhablani Nihal Nayak Debajyoti Datta Jonathan Chang Mike Tian-Jian Jiang Han Wang Matteo Manica Sheng Shen Zheng Xin Yong Harshit Pandey Rachel Bawden Thomas Wang Trishala Neeraj Jos Rozen Abheesht Sharma Andrea Santilli Thibault Fevry Jason Alan Fries Ryan Teehan Tali Bers Stella Biderman Leo Gao Thomas Wolf and Alexander M. Rush. 2022. Multitask prompted training enables zero-shot task generalization. https://arxiv.org/abs/2110.08207 · arXiv
- Timo Schick and Hinrich Schütze. 2019. Rare words: A major problem for contextualized embeddings and how to fix it by attentive mimicking. https://arxiv.org/abs/1904.06707 · arXiv
- Timo Schick and Hinrich Schütze. 2020. Few-shot text generation with pattern-exploiting training. (2020). https://arxiv.org/abs/2012.11926 · arXiv
- Fábio Souza Rodrigo Nogueira and Roberto Lotufo. 2020. Portuguese named entity recognition using BERT-CRF. https://arxiv.org/abs/1909.10649 · arXiv
- Asa Cooper Stickland and Iain Murray. 2019. BERT and PALs: Projected attention layers for efficient adaptation in multi-task learning. In Proceedings of the 36th International Conference on Machine Learning. PMLR, 5986–5995.
- Ehsan Taher, Seyed Abbas Hoseini, and Mehrnoush Shamsfard. 2019. Beheshti-NER: Persian named entity recognition using BERT. In Proceedings of the 1st International Workshop on NLP Solutions for Under Resourced Languages (NSURL’19) co-located with ICNLSP 2019 - Short Papers. Association for Computational Linguistics, Trento, Italy, 37–42.
- Trieu H. Trinh and Quoc V. Le. 2019. A simple method for commonsense reasoning. https://arxiv.org/abs/1806.02847 · arXiv
- Maria Tsimpoukelli Jacob Menick Serkan Cabi S. M. Ali Eslami Oriol Vinyals and Felix Hill. 2021. Multimodal few-shot learning with frozen language models. https://arxiv.org/abs/2106.13884 · arXiv
- Sinong Wang Han Fang Madian Khabsa Hanzi Mao and Hao Ma. 2021. Entailment as few-shot learner. https://arxiv.org/abs/2104.14690 · arXiv
- Thomas Wang Adam Roberts Daniel Hesslow Teven Le Scao Hyung Won Chung Iz Beltagy Julien Launay and Colin Raffel. 2022. What language model architecture and pPretraining objective work bBest for zero-shot generalization?https://arxiv.org/abs/2204.05832 · arXiv
- Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: A benchmark of linguistic minimal pairs for English. In Proceedings of the Society for Computation in Linguistics 2020.
- Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. How transferable are features in deep neural networks?. Advances in Neural Information Processing Systems, Vol. 27. Curran Associates, Inc.
- Wenhao Yu Chenguang Zhu Zaitang Li Zhiting Hu Qingyun Wang Heng Ji and Meng Jiang. 2021. A survey of knowledge-enhanced text generation. https://arxiv.org/abs/2010.04389 · arXiv
- Weizhe Yuan Graham Neubig and Pengfei Liu. 2021. BARTScore: Evaluating generated text as text generation. https://arxiv.org/abs/2106.11520 · arXiv
- Karolina Zaczynska, Nils Feldhus, Robert Schwarzenberg, Aleksandra Gabryszak, and Sebastian Möller. 2020. Evaluating German transformer language models with syntactic agreement tests, In Proceedings of the 5th Swiss Text Analytics Conference and the 16th Conference on Natural Language Processing (SwissText/KONVENS’20). CoRR abs/2007.03765. · arXiv
- Tony Z. Zhao Eric Wallace Shi Feng Dan Klein and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. https://arxiv.org/abs/2102.09690 · arXiv
- Junru Zhou, Zhuosheng Zhang, Hai Zhao, and Shuailiang Zhang. 2020. LIMIT-BERT: Linguistics informed multi-task BERT. In Findings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics, Online.
- Li Zhou and Kevin Small. 2020. Multi-domain dialogue state tracking as dynamic knowledge graph enhanced question answering. https://arxiv.org/abs/1911.06192 · arXiv
- Jinhua Zhu Yingce Xia Lijun Wu Di He Tao Qin Wengang Zhou Houqiang Li and Tie-Yan Liu. 2020. Incorporating BERT into neural machine translation. https://arxiv.org/abs/2002.06823 · arXiv
Cited by
ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education
Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · doi:10.35542/osf.io/5er8f · Cited by 7,552
ChatGPT for good? On opportunities and challenges of large language models for education
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · Learning and Individual Differences · doi:10.1016/j.lindif.2023.102274 · Cited by 7,326
Unifying Large Language Models and Knowledge Graphs: A Roadmap
Pan, S., Luo, L., Wang, Y., Chen, C., Wang, J., & Wu, X. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3352100 · Cited by 1,376
A Comprehensive Overview of Large Language Models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3744746 · Cited by 1,257
Practical and ethical challenges of large language models in education: A systematic scoping review
Yan, L., Sha, L., Zhao, L., Li, Y., Martinez‐Maldonado, R., Chen, G., Li, X., Jin, Y., & Gašević, D. (2024) · British Journal of Educational Technology · doi:10.1111/bjet.13370 · Cited by 1,435
Bias and Fairness in Large Language Models: A Survey
Gallegos, I. O., Rossi, R. A., Barrow, J., Tanjim, M. M., Kim, S., Dernoncourt, F., Yu, T., Zhang, R., & Ahmed, N. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00524 · Cited by 1,245
ChatGPT for tourism: applications, benefits and risks
Carvalho, I., & Ivanov, S. (2024) · Tourism Review · doi:10.1108/tr-02-2023-0088 · Cited by 507
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., Chen, Y., et al. (2025) · Computational Linguistics · doi:10.1162/coli.a.16 · Cited by 534
Recommender Systems in the Era of Large Language Models (LLMs)
Zhao, Z., Fan, W., Li, J., Liu, Y., Mei, X., Wang, Y., Wen, Z., Wang, F., Zhao, X., Tang, J., et al. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3392335 · Cited by 481
Pre-Trained Language Models and Their Applications
Wang, H., Li, J., Wu, H., Hovy, E., & Sun, Y. (2023) · Engineering · doi:10.1016/j.eng.2022.04.024 · Cited by 385
Large language models (LLMs): survey, technical frameworks, and future challenges
Kumar, P. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10888-y · Cited by 395
Durably reducing conspiracy beliefs through dialogues with AI
Costello, T. H., Pennycook, G., & Rand, D. G. (2024) · Science · doi:10.1126/science.adq1814 · Cited by 354
The debate over understanding in AI’s large language models
Mitchell, M., & Krakauer, D. C. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2215907120 · Cited by 340
Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey
Ling, C., Zhao, X., Lu, J., Deng, C., Zheng, C., Wang, J., Chowdhury, T., Li, Y., Cui, H., Zhang, X., et al. (2025) · ACM Computing Surveys · doi:10.1145/3764579 · Cited by 464
Artificial intelligence for literature reviews: opportunities and challenges
Bolaños, F., Salatino, A., Osborne, F., & Motta, E. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10902-3 · Cited by 246
“Therefore, the first interesting research direction regards incorporating advanced NLP technologies, particularly the rapidly evolving Large Language Models (LLMs) [104].”
Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey
Wang, X., Chen, G., Qian, G., Gao, P., Wei, X.-Y., Wang, Y., Tian, Y., & Gao, W. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1410-8 · Cited by 223
Generative AI and ChatGPT in School Children’s Education: Evidence from a School Lesson
Jauhiainen, J. S., & Guerra, A. G. (2023) · Sustainability · doi:10.3390/su151814025 · Cited by 234
Deep potentials for materials science
Wen, T., Zhang, L., Wang, H., E, W., & Srolovitz, D. J. (2022) · Materials Futures · doi:10.1088/2752-5724/ac681d · Cited by 214
Friha, O., Amine Ferrag, M., Kantarci, B., Cakmak, B., Ozgun, A., & Ghoualmi-Zine, N. (2024) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2024.3456549 · Cited by 189
All in One: Multi-Task Prompting for Graph Neural Networks
Sun, X., Cheng, H., Li, J., Liu, B., & Guan, J. (2023) · Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining · doi:10.1145/3580305.3599256 · Cited by 173
Large language models in law: A survey
Lai, J., Gan, W., Wu, J., Qi, Z., & Yu, P. S. (2024) · AI Open · doi:10.1016/j.aiopen.2024.09.002 · Cited by 352
A Cookbook of Self-Supervised Learning
Balestriero, R., Ibrahim, M., Sobal, V., Morcos, A., Shekhar, S., Goldstein, T., Bordes, F., Bardes, A., Mialon, G., Tian, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.12210 · Cited by 182
Knowledge graph-based manufacturing process planning: A state-of-the-art review
Xiao, Y., Zheng, S., Shi, J., Du, X., & Hong, J. (2023) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2023.08.006 · Cited by 176
Evolutionary Computation in the Era of Large Language Model: Survey and Roadmap
Wu, X., Wu, S.-H., Wu, J., Feng, L., & Tan, K. C. (2024) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2024.3506731 · Cited by 329
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v4 · Cited by 378
NetLLM: adapting large language models for networking
Wu, D., Wang, X., Qiao, Y., Wang, Z., Jiang, J., Cui, S., & Wang, F. (2024) · Proceedings of the ACM SIGCOMM 2024 Conference · doi:10.1145/3651890.3672268 · Cited by 167
“Middle, Right : Token-based prediction with LM head fails to guarantee valid answers and produce stale responses, while NetLLM efficiently addresses these issues with the networking head module. the resource demand of each stage.”
BackgroundMethodFields, J., Chovanec, K., & Madiraju, P. (2024) · IEEE Access · doi:10.1109/access.2024.3349952 · Cited by 245
A scoping review of ChatGPT research in accounting and finance
Dong, M., Stratopoulos, T. C., & Wang, V. X. (2024) · International Journal of Accounting Information Systems · doi:10.1016/j.accinf.2024.100715 · Cited by 239
Large Language Models as Evolutionary Optimizers
Liu, S., Chen, C., Qu, X., Tang, K., & Ong, Y.-S. (2024) · IEEE Congress on Evolutionary Computation (CEC) · doi:10.1109/cec60901.2024.10611913 · Cited by 209
A Survey on the Memory Mechanism of Large Language Model based Agents
Zhang, Z., Bo, X., Ma, C., Li, R., Chen, X., Dai, Q., Zhu, J., Dong, Z., & Wen, J.-R. (2024) · arXiv · doi:10.48550/arxiv.2404.13501 · Cited by 149
Towards an understanding of large language models in software engineering tasks
Zheng, Z., Ning, K., Zhong, Q., Chen, J., Chen, W., Guo, L., Wang, W., & Wang, Y. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10602-0 · Cited by 283
Generative AI in cybersecurity: A comprehensive review of LLM applications and vulnerabilities
Ferrag, M. A., Alwahedi, F., Battah, A., Cherif, B., Mechri, A., Tihanyi, N., Bisztray, T., & Debbah, M. (2025) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2025.01.001 · Cited by 148
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.102963 · Cited by 147
Lee, J., Ahn, S., Kim, D., & Kim, D. (2024) · Automation in Construction · doi:10.1016/j.autcon.2024.105846 · Cited by 184
Large Generative AI Models for Telecom: The Next Big Thing?
Bariah, L., Zhao, Q., Zou, H., Tian, Y. T., Bader, F., & Debbah, M. A. (2024) · IEEE Communications Magazine · doi:10.1109/mcom.001.2300364 · Cited by 196
Federated Continual Learning via Knowledge Fusion: A Survey
Yang, X., Yu, H., Gao, X., Wang, H., Zhang, J., & Li, T. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3363240 · Cited by 198
Multi-purpose RNA language modelling with motif-aware pretraining and type-guided fine-tuning
Wang, N., Bian, J., Li, Y., Li, X., Mumtaz, S., Kong, L., & Xiong, H. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00836-4 · Cited by 182
Large language models for reticular chemistry
Zheng, Z., Rampal, N., Inizan, T. J., Borgs, C., Chayes, J. T., & Yaghi, O. M. (2025) · Nature Reviews Materials · doi:10.1038/s41578-025-00772-8 · Cited by 134
How understanding large language models can inform the use of ChatGPT in physics education
Polverini, G., & Gregorcic, B. (2023) · European Journal of Physics · doi:10.1088/1361-6404/ad1420 · Cited by 199
NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark
Sainz, O., Campos, J., García-Ferrero, I., Etxaniz, J., de Lacalle, O. L., & Agirre, E. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.722 · Cited by 129
Self-citationLong, S., Tan, J., Mao, B., Tang, F., Li, Y., Zhao, M., & Kato, N. (2025) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2025.3526606 · Cited by 165
McIntosh, T. R., Sušnjak, T., Liu, T., Watters, P., Xu, D., Liu, D., Nowrozy, R., & Halgamuge, M. N. (2024) · Computers & Security · doi:10.1016/j.cose.2024.103964 · Cited by 150
Lifelo, Z., Ding, J., Ning, H., Ain, Q. U., & Dhelim, S. (2024) · Electronics · doi:10.3390/electronics13244874 · Cited by 173
Leveraging large language models for generating responses to patient messages—a subjective analysis
Liu, S., McCoy, A. B., Wright, A. P., Carew, B., Genkins, J. Z., Huang, S. S., Peterson, J. F., Steitz, B. D., & Wright, A. T. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae052 · Cited by 142
Retrieval-Augmented Generation–Enabled GPT-4 for Clinical Trial Screening
Ünlü, O., Shin, J., Mailly, C. J., Oates, M., Tucci, M. R., Varugheese, M., Wagholikar, K. B., Wang, F., Scirica, B. M., Blood, A. J., et al. (2024) · NEJM AI · doi:10.1056/aioa2400181 · Cited by 132
Collaborative Large Language Model for Recommender Systems
Zhu, Y., Wu, L., Guo, Q. S., Hong, L., & Li, J. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589334.3645347 · Cited by 231
Survey of Different Large Language Model Architectures: Trends, Benchmarks, and Challenges
Shao, M., Basit, A., Karri, R., & Shafique, M. (2024) · IEEE Access · doi:10.1109/access.2024.3482107 · Cited by 176
Jiang, P., Sonne, C., Li, W., You, F., & You, S. (2024) · Engineering · doi:10.1016/j.eng.2024.04.002 · Cited by 161
Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Models
Weyssow, M., Zhou, X., Kim, K., Lo, D., & Sahraoui, H. A. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3714461 · Cited by 200
Automated Domain Modeling with Large Language Models: A Comparative Study
Chen, K., Yang, Y., Chen, B., López, J. A. H., Mussbacher, G., & Varró, D. (2023) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS) · doi:10.1109/models58315.2023.00037 · Cited by 138
BB-GeoGPT: A framework for learning a large language model for geographic information science
Zhang, Y., Wang, Z., He, Z., Li, J., Mai, G., Lin, J., Cheng, W., & Yu, W. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103808 · Cited by 134
Model Generation with LLMs: From Requirements to UML Sequence Diagrams
Ferrari, A., Abualhaija, S., & Arora, C. (2024) · IEEE International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew61692.2024.00044 · Cited by 131
Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence
McIntosh, T. R., Susnjak, T., Arachchilage, N., Liu, T., Xu, D., Watters, P., & Halgamuge, M. N. (2026) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2025.3569516 · Cited by 104
“L ARGE Language Models (LLMs), a sophisticated branch of generative Artificial Intelligence (AI), have evolved from narrow, task-specific systems to versatile models capable of few-shot learning and handling diverse tasks [1], [2].”
BackgroundInfluentialInadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence
McIntosh, T. R., Susnjak, T., Arachchilage, N., Liu, T., Watters, P., & Halgamuge, M. N. (2024) · arXiv · doi:10.48550/arxiv.2402.09880 · Cited by 104
McIntosh, T. R., Susnjak, T., Liu, T., Watters, P., Xu, D., Liu, D., & Halgamuge, M. N. (2025) · Technologies · doi:10.3390/technologies13020051 · Cited by 102
“This evolution has been driven by a relentless quest for models that more accurately reflect the nuances of human language, as well as the desire to push the boundaries of what machines can understand and generate [36], [38], [37].”
BackgroundMethodInfluentialIntegrating LLMs With ITS: Recent Advances, Potentials, Challenges, and Future Directions
Mahmud, D., Hajmohamed, H., Almentheri, S., Alqaydi, S., Aldhaheri, L., Khalil, R. A., & Saeed, N. (2025) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2025.3528116 · Cited by 138
SpecGen: Automated Generation of Formal Program Specifications via Large Language Models
Ma, L., Liu, S., Li, Y., Xie, X., & Bu, L. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00129 · Cited by 235
Mustapha, K. B. (2024) · Advanced Engineering Informatics · doi:10.1016/j.aei.2024.103066 · Cited by 117
LLMs for science: Usage for code generation and data analysis
Nejjar, M., Zacharias, L., Stiehle, F., & Weber, I. M. (2024) · Journal of Software Evolution and Process · doi:10.1002/smr.2723 · Cited by 126
Application of LLM Agents in Recruitment: A Novel Framework for Automated Resume Screening
Gan, C., Zhang, Q., & Mori, T. (2024) · Journal of Information Processing · doi:10.2197/ipsjjip.32.881 · Cited by 184
A Survey on Semantic Communication Networks: Architecture, Security, and Privacy
Guo, S., Wang, Y., Zhang, N., Su, Z., Luan, T. H., Tian, Z., & Shen, X. (2024) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2024.3516819 · Cited by 166
Applications of LLMs for Generating Cyber Security Exercise Scenarios
Mudassar Yamin, M., Hashmi, E., Ullah, M., & Katt, B. (2024) · IEEE Access · doi:10.1109/access.2024.3468914 · Cited by 91
“However, significant adoption of this method began in the latter half of the decade, with the development of the Transformer architecture [27], [28].”
MethodZero-shot information extraction from radiological reports using ChatGPT
Hu, D., Liu, B., Zhu, X., Lü, X., & Wu, N. (2023) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2023.105321 · Cited by 117
Automating Research Synthesis with Domain-Specific Large Language Model Fine-Tuning
Sušnjak, T., Hwang, P., Reyes, N. H., Barczak, A. L. C., McIntosh, T. R., & Ranathunga, S. (2025) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3715964 · Cited by 141
Diwali, A., Saeedi, K., Dashtipour, K. K., Gogate, M., Cambria, E., & Hussain, A. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3296373 · Cited by 111
Generative Knowledge Graph Construction: A Review
Ye, H., Zhang, N., Chen, H., & Chen, H. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.1 · Cited by 142
Conversational Process Modelling: State of the Art, Applications, and Implications in Practice
Klievtsova, N., Benzin, J.-V., Kampik, T., Mangler, J., & Rinderle‐Ma, S. (2023) · Lecture notes in business information processing · doi:10.1007/978-3-031-41623-1_19 · Cited by 111
Sentiment Analysis of Twitter Data Using NLP Models: A Comprehensive Review
Albladi, A., Islam, M., & Seals, C. (2025) · IEEE Access · doi:10.1109/access.2025.3541494 · Cited by 107
5%>100%: Breaking Performance Shackles of Full Fine-Tuning on Visual Recognition Tasks
Yin, D., Hu, L., Li, B., Zhang, Y., & Yang, X. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.01869 · Cited by 183
The grass is not always greener: Teacher vs. GPT-assisted written corrective feedback
Lin, S., & Crosthwaite, P. (2024) · System · doi:10.1016/j.system.2024.103529 · Cited by 86
Zhou, T., Cao, S., Zhou, S., Zhang, Y., & He, A. (2023) · System · doi:10.1016/j.system.2023.103141 · Cited by 103
“Doctor ChatGPT, Can You Help Me?” The Patient’s Perspective: Cross-Sectional Study
Armbruster, J., Bussmann, F., Rothhaas, C., Titze, N., Grützner, P. A., & Freischmidt, H. (2024) · Journal of Medical Internet Research · doi:10.2196/58831 · Cited by 102
Evaluation of LLM-based chatbots for OSINT-based Cyber Threat Awareness
Shafee, S., Bessani, A., & Ferreira, P. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125509 · Cited by 112
Augmented non-hallucinating large language models as medical information curators
Gilbert, S., Kather, J. N., & Hogan, A. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01081-0 · Cited by 90
Natural language processing in the era of large language models
Zubiaga, A. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1350306 · Cited by 81
LLMaAA: Making Large Language Models as Active Annotators
Zhang, R., Li, Y., Ma, Y., Zhou, M. Q., & Zou, L. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.872 · Cited by 114
Knowledge Editing for Large Language Models: A Survey
Wang, S., Zhu, Y., Liu, H., Zheng, Z., Chen, C., & Li, J. (2025) · ACM Computing Surveys · doi:10.1145/3698590 · Cited by 79
Harnessing Large Language Models for Text-Rich Sequential Recommendation
Zheng, Z., Chao, W., Qiu, Z., Zhu, H., & Xiong, H. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589334.3645358 · Cited by 93
Wandelt, S., Zheng, C., Wang, S., Liu, Y., & Sun, X. (2024) · Applied Sciences · doi:10.3390/app14177455 · Cited by 89
ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs
Zhuo, J., Zhang, S., Fang, X., Duan, H., Lin, D., & Chen, K. (2024) · arXiv · doi:10.48550/arxiv.2410.12405 · Cited by 78
“By training on large-scale corpora, LLMs have shown impressive capabilities in multiple tasks (Zhao et al., 2023; Min et al., 2023).”
BackgroundChow, J. C. L., & Li, K. (2024) · JMIR Bioinformatics and Biotechnology · doi:10.2196/64406 · Cited by 75
Li, W., Cao, D., Tan, R., Shi, T., Gao, Z., Ma, J., Guo, G., Hu, H., Feng, J., & Wang, L. (2023) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2023.3339798 · Cited by 76
When large language model meets optimization
Huang, S., Yang, K., Qi, S., & Wang, R. (2024) · Swarm and Evolutionary Computation · doi:10.1016/j.swevo.2024.101663 · Cited by 113
Multi-Modal LLMs in Agriculture: A Comprehensive Review
Sapkota, R., Qureshi, R., Hadi, M. U., Hassan, S. Z., Sadak, F., Shoman, M., Sajjad, M., Dharejo, F. A., Paudel, A., Li, J., et al. (2025) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2025.3612154 · Cited by 106
Agentic Large Language Models, a Survey
Plaat, A., Van Duijn, M., Van Stein, N., Preuss, M., Van der Putten, P., & Batenburg, K. J. (2025) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.18675 · Cited by 75
Thapa, S., Shiwakoti, S., Shah, S. B., Adhikari, S., Veeramani, H., Nasim, M., & Naseem, U. (2025) · Social Network Analysis and Mining · doi:10.1007/s13278-025-01428-9 · Cited by 74
Jose, S., Nguyen, K. T. P., Medjaher, K., Zemouri, R. M., Lévesque, M., & Tahan, A. S. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124603 · Cited by 80
A survey on large language models with multilingualism: recent advances and new frontiers
Huang, K., Mo, F., Zhang, X., Li, H., Li, Y., Zhang, Y., yi, W., Mao, Y., Liu, J., Xu, Y., et al. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-026-11534-5 · Cited by 145
Envisioning Information Access Systems: What Makes for Good Tools and a Healthy Web?
Shah, C. S., & Bender, E. M. (2024) · ACM Transactions on the Web · doi:10.1145/3649468 · Cited by 115
Leveraging LLMs for the Quality Assurance of Software Requirements
Lubos, S., Felfernig, A., Tran, T. N. T., Garber, D., Mansi, M. E., Erdeniz, S. P., & Le, V.-M. (2024) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re59067.2024.00046 · Cited by 84
Large language models for life cycle assessments: Opportunities, challenges, and risks
Preuss, N., Alshehri, A. S., & You, F. (2024) · Journal of Cleaner Production · doi:10.1016/j.jclepro.2024.142824 · Cited by 71
A critical review of large language models: Sensitivity, bias, and the path toward specialized AI
Hajikhani, A., & Cole, C. H. (2024) · Quantitative Science Studies · doi:10.1162/qss_a_00310 · Cited by 93
Taani, O., & Alabidi, S. (2024) · International Journal of Mathematical Education in Science and Technology · doi:10.1080/0020739x.2024.2357341 · Cited by 71
Wu, T., Lee, H., Chen, P., Lin, C., & Huang, Y. (2024) · Journal of Computer Assisted Learning · doi:10.1111/jcal.13085 · Cited by 75
He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., & Cambria, E. (2024) · doi:10.2139/ssrn.4809363 · Cited by 69
Yigit, Y., Ferrag, M. A., Ghanem, M. C., Sarker, I. H., Maglaras, L. A., Chrysoulas, C., Moradpoor, N., Tihanyi, N., & Janicke, H. (2025) · Sensors · doi:10.3390/s25061666 · Cited by 69
Javaid, S., Khalil, R. A., Saeed, N., He, B., & Alouini, M. (2024) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2024.3522103 · Cited by 92
LLMCompass: Enabling Efficient Hardware Design for Large Language Model Inference
Zhang, H., Ning, A., Prabhakar, R. B., & Wentzlaff, D. (2024) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1109/isca59077.2024.00082 · Cited by 161
Multi-Modal LLMs in Agriculture: A Comprehensive Review
Sapkota, R., Qureshi, R., Hassan, S. Z., Shutske, J., Shoman, M., Sajjad, M., Dharejo, F. A., Paudel, A., Li, J., Meng, Z., et al. (2024) · doi:10.36227/techrxiv.172651082.24507804/v1 · Cited by 68
Metadata
- Authors
- Min, B., Ross, H., Sulem, E., Veyseh, A. P. B., Nguyen, T. H., Sainz, O., Agirre, E., Heintz, I., & Roth, D.
- Affiliations
- Amazon (United States) (US)
- Harvard University (US)
- University of Pennsylvania (US)
- University of Oregon (US)
- University of the Basque Country (ES)
- Synoptic Engineering, USA
- ORCID
- Venue
- ACM Computing Surveys
- Published
- 2023-09-14
- Volume
- 56
- Issue
- 2
- Pages
- 1-40
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W4382246105
- Semantic Scholar
- c23d9d44e8bc68408cea9f305d1f24d915bc0d0d
- Cited by
- 1,387
- References
- 160
- Citations by year
- 2026: 285 · 2025: 521 · 2024: 461 · 2023: 98
- FWCI
- 128.67
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Computing Surveys (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- U.S. Department of Defense
- Office of the Director of National Intelligence
- Defense Advanced Research Projects Agency (FA8750-19-2-1004, FA8750)
- Advanced Research Projects Agency
- Intelligence Advanced Research Projects Activity (2019-19051600006)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- large pre-trained language models, BERT, GPT, transfer learning, fine-tuning, prompting, text generation, language representation learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.