Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey
Open accessThe DOI matches its Crossref record; 185 of the 185 references it cites were found in a registry or an index.
Abstract
Large, pre-trained language models (PLMs) such as BERT and GPT have drastically changed the Natural Language Processing (NLP) field. For numerous NLP tasks, approaches leveraging PLMs have achieved state-of-the-art performance. The key idea is to learn a generic, latent representation of language from a generic task once, then share it across disparate NLP tasks. Language modeling serves as the generic task, one with abundant self-supervised text available for extensive training. This article presents the key fundamental concepts of PLM architectures and a comprehensive view of the shift to PLM-driven NLP techniques. It surveys work applying the pre-training then fine-tuning, prompting, and text generation approaches. In addition, it discusses PLM limitations and suggested directions for future research.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“However, the field of AI is evolving rapidly, particularly in areas such as Generative AI (Brynjolfsson et al. 2023) and Large Language Models (Min et al. 2023).”
From Artificial intelligence for literature reviews: opportunities and challenges (2024) · cites this paper as Min et al. 2023
In the citing paper
“However, the open-ended nature of language means that there can be multiple valid responses to a given prompt, complicating the assessment of correctness [1].”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [1]
In the citing paper
“LLMs like Llama and GPT-4 are capable of understanding and generating human-like text, performing tasks ranging from translation to creative writing [1], [4].”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [1]
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,887
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Research works printed but not linked: 56
- Zeyuan Allen-Zhu and Yuanzhi Li. 2021. Towards understanding ensemble knowledge distillation and self-distillation in deep learning. https://arxiv.org/abs/2012.09816 · arXiv
- Asaf Amrami and Yoav Goldberg. 2019. Towards better substitution-based word sense induction. https://arxiv.org/abs/1905.12598 · arXiv
- Stephen H. Bach Victor Sanh Zheng-Xin Yong Albert Webson Colin Raffel Nihal V. Nayak Abheesht Sharma Taewoon Kim M. Saiful Bari Thibault Fevry Zaid Alyafeai Manan Dey Andrea Santilli Zhiqing Sun Srulik Ben-David Canwen Xu Gunjan Chhablani Han Wang Jason Alan Fries Maged S. Al-shaibani Shanya Sharma Urmish Thakker Khalid Almubarak Xiangru Tang Dragomir Radev Mike Tian-Jian Jiang and Alexander M. Rush. 2022. PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts. https://arxiv.org/abs/2202.01279 · arXiv
- Geoff Bacon and Terry Regier. 2019. Does BERT agree? Evaluating knowledge of structure dependence through agreement relations. https://arxiv.org/abs/1908.09892 · arXiv
- Jack Bandy and Nicholas Vincent. 2021. Addressing “Documentation Debt” in Machine Learning Research: A Retrospective Datasheet for BookCorpus. https://arxiv.org/abs/2105.05241 · arXiv
- Eyal Ben-David Nadav Oved and Roi Reichart. 2021. PADA: A prompt-based autoregressive approach for adaptation to unseen domains. https://arxiv.org/abs/2102.12206 · arXiv
- Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota.
- 10.5555/2463101 · doi:10.5555/2463101
- Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. 2017. Language modeling with gated convolutional networks. In Proceedings of the International Conference on Machine Learning. PMLR, 933–941.
- Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021. Autoregressive entity retrieval. In Proceedings of the 9th International Conference on Learning Representations (ICLR’21).
- Avia Efrat and Omer Levy. 2020. The turking test: Can language models understand instructions?https://arxiv.org/abs/2010.11982 · arXiv
- Tom Brown et al.2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 1877–1901.
- Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations.
- Steven Fincke Shantanu Agarwal Scott Miller and Elizabeth Boschee. 2021. Language model priming for cross-lingual event extraction. https://arxiv.org/abs/2109.12383 · arXiv
- Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020. Online.
- Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning. PMLR.
- Minyoung Huh Pulkit Agrawal and Alexei A. Efros. 2016. What makes ImageNet good for transfer learning?arXiv: 1608.08614. · arXiv
- Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. UNIFIEDQA: Crossing format boundaries with a single QA system. In Findings of the Association for Computational Linguistics: EMNLP 2020.
- Sawan Kumar and Partha Talukdar. 2021. Reordering examples helps during priming-based few-shot learning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. Association for Computational Linguistics, Online.
- Huiyuan Lai Antonio Toral and Malvina Nissim. 2021. Thank you BART! Rewarding pre-trained models improves formality style transfer. https://arxiv.org/abs/2105.06947 · arXiv
- Fayuan Li, Weihua Peng, Yuguang Chen, Quan Wang, Lu Pan, Yajuan Lyu, and Yong Zhu. 2020. Event extraction as multi-turn question answering. In Findings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics, Online.
- Junyi Li Tianyi Tang Wayne Xin Zhao and Ji-Rong Wen. 2021. Pretrained language models for text generation: A survey. https://arxiv.org/abs/2105.10311 · arXiv
- Jiachang Liu Dinghan Shen Yizhe Zhang Bill Dolan Lawrence Carin and Weizhu Chen. 2021. What makes good in-context examples for GPT-3?. https://arxiv.org/abs/2101.06804 · arXiv
- Pengfei Liu Weizhe Yuan Jinlan Fu Zhengbao Jiang Hiroaki Hayashi and Graham Neubig. 2021. Pre-train prompt and predict: A systematic survey of prompting methods in natural language processing. arXiv: 2107.13586. · arXiv
- Xiao Liu Yanan Zheng Zhengxiao Du Ming Ding Yujie Qian Zhilin Yang and Jie Tang. 2021. GPT understands too. https://arxiv.org/abs/2103.10385 · arXiv
- Brian MacWhinney. 2000. The CHILDES Project: Tools for Analyzing Talk. Transcription Format and Programs. Vol. 1. Psychology Press.
- Xuan-Phi Nguyen Shafiq Joty Steven C. H. Hoi and Richard Socher. 2020. Tree-structured attention with hierarchical accumulation. https://arxiv.org/abs/2002.08046 · arXiv
- Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020. Document ranking with a pretrained sequence-to-sequence model. In Findings of the Association for Computational Linguistics (EMNLP).
- Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, Rishita Anubhai, Cicero dos Santos Nogueira, Bing Xiang, and Stefano Soatto. 2021. Structured prediction as translation between augmented natural languages. In Proceedings of the 9th International Conference on Learning Representations (ICLR’21).
- Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. 2020. How context affects language models’ factual predictions. In Proceedings of AKBC.
- Raul Puri and Bryan Catanzaro. 2019. Zero-shot text classification with generative language models. https://arxiv.org/abs/1912.10165 · arXiv
- Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. OpenAI Blog (2018), 12.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.
- Evani Radiya-Dixit and Xin Wang. 2020. How fine can fine-tuning be? Learning efficient language models. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics(Proceedings of Machine Learning Research, Vol. 108), Silvia Chiappa and Roberto Calandra (Eds.). 2435–2443.
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research (2020).
- Liliang Ren, Chenkai Sun, Heng Ji, and Julia Hockenmaier. 2021. HySPA: Hybrid span generation for scalable text-to-graph extraction. In Findings of the Association for Computational Linguistics (ACL-IJCNLP’21).
- Victor Sanh Lysandre Debut Julien Chaumond and Thomas Wolf. 2020. DistilBERT a distilled version of BERT: Smaller faster cheaper and lighter. https://arxiv.org/abs/1910.01108 · arXiv
- Victor Sanh Albert Webson Colin Raffel Stephen H. Bach Lintang Sutawika Zaid Alyafeai Antoine Chaffin Arnaud Stiegler Teven Le Scao Arun Raja Manan Dey M. Saiful Bari Canwen Xu Urmish Thakker Shanya Sharma Sharma Eliza Szczechla Taewoon Kim Gunjan Chhablani Nihal Nayak Debajyoti Datta Jonathan Chang Mike Tian-Jian Jiang Han Wang Matteo Manica Sheng Shen Zheng Xin Yong Harshit Pandey Rachel Bawden Thomas Wang Trishala Neeraj Jos Rozen Abheesht Sharma Andrea Santilli Thibault Fevry Jason Alan Fries Ryan Teehan Tali Bers Stella Biderman Leo Gao Thomas Wolf and Alexander M. Rush. 2022. Multitask prompted training enables zero-shot task generalization. https://arxiv.org/abs/2110.08207 · arXiv
- Timo Schick and Hinrich Schütze. 2019. Rare words: A major problem for contextualized embeddings and how to fix it by attentive mimicking. https://arxiv.org/abs/1904.06707 · arXiv
- Timo Schick and Hinrich Schütze. 2020. Few-shot text generation with pattern-exploiting training. (2020). https://arxiv.org/abs/2012.11926 · arXiv
- Fábio Souza Rodrigo Nogueira and Roberto Lotufo. 2020. Portuguese named entity recognition using BERT-CRF. https://arxiv.org/abs/1909.10649 · arXiv
- Asa Cooper Stickland and Iain Murray. 2019. BERT and PALs: Projected attention layers for efficient adaptation in multi-task learning. In Proceedings of the 36th International Conference on Machine Learning. PMLR, 5986–5995.
- Ehsan Taher, Seyed Abbas Hoseini, and Mehrnoush Shamsfard. 2019. Beheshti-NER: Persian named entity recognition using BERT. In Proceedings of the 1st International Workshop on NLP Solutions for Under Resourced Languages (NSURL’19) co-located with ICNLSP 2019 - Short Papers. Association for Computational Linguistics, Trento, Italy, 37–42.
- Trieu H. Trinh and Quoc V. Le. 2019. A simple method for commonsense reasoning. https://arxiv.org/abs/1806.02847 · arXiv
- Maria Tsimpoukelli Jacob Menick Serkan Cabi S. M. Ali Eslami Oriol Vinyals and Felix Hill. 2021. Multimodal few-shot learning with frozen language models. https://arxiv.org/abs/2106.13884 · arXiv
- Sinong Wang Han Fang Madian Khabsa Hanzi Mao and Hao Ma. 2021. Entailment as few-shot learner. https://arxiv.org/abs/2104.14690 · arXiv
- Thomas Wang Adam Roberts Daniel Hesslow Teven Le Scao Hyung Won Chung Iz Beltagy Julien Launay and Colin Raffel. 2022. What language model architecture and pPretraining objective work bBest for zero-shot generalization?https://arxiv.org/abs/2204.05832 · arXiv
- Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: A benchmark of linguistic minimal pairs for English. In Proceedings of the Society for Computation in Linguistics 2020.
- Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. How transferable are features in deep neural networks?. Advances in Neural Information Processing Systems, Vol. 27. Curran Associates, Inc.
- Wenhao Yu Chenguang Zhu Zaitang Li Zhiting Hu Qingyun Wang Heng Ji and Meng Jiang. 2021. A survey of knowledge-enhanced text generation. https://arxiv.org/abs/2010.04389 · arXiv
- Weizhe Yuan Graham Neubig and Pengfei Liu. 2021. BARTScore: Evaluating generated text as text generation. https://arxiv.org/abs/2106.11520 · arXiv
- Karolina Zaczynska, Nils Feldhus, Robert Schwarzenberg, Aleksandra Gabryszak, and Sebastian Möller. 2020. Evaluating German transformer language models with syntactic agreement tests, In Proceedings of the 5th Swiss Text Analytics Conference and the 16th Conference on Natural Language Processing (SwissText/KONVENS’20). CoRR abs/2007.03765. · arXiv
- Tony Z. Zhao Eric Wallace Shi Feng Dan Klein and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. https://arxiv.org/abs/2102.09690 · arXiv
- Junru Zhou, Zhuosheng Zhang, Hai Zhao, and Shuailiang Zhang. 2020. LIMIT-BERT: Linguistics informed multi-task BERT. In Findings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics, Online.
- Li Zhou and Kevin Small. 2020. Multi-domain dialogue state tracking as dynamic knowledge graph enhanced question answering. https://arxiv.org/abs/1911.06192 · arXiv
- Jinhua Zhu Yingce Xia Lijun Wu Di He Tao Qin Wengang Zhou Houqiang Li and Tie-Yan Liu. 2020. Incorporating BERT into neural machine translation. https://arxiv.org/abs/2002.06823 · arXiv
Cited by
Items 501–600 of 1,014
Optimizing and Evaluating Pre- Trained Large Language Models for Alzheimer's Disease Detection
Casu, F., Grosso, E., Lagorio, A., & Trunfio, G. A. (2024) · Euromicro International Conference on Parallel, Distributed and Network-Based Processing (PDP) · doi:10.1109/pdp62718.2024.00046 · Cited by 16
Bi-Modal Bi-Task Emotion Recognition Based on Transformer Architecture
Song, Y., & Zhou, Q. (2024) · Applied Artificial Intelligence · doi:10.1080/08839514.2024.2356992 · Cited by 11
Posedaru, B.-Ș., Pantelimon, F.-V., Dulgheru, M.-N., & Georgescu, T.-M. (2024) · Proceedings of the ... International Conference on Business Excellence · doi:10.2478/picbe-2024-0018 · Cited by 13
Seetharam, G. U. D. E., & Yamini Satyasri, G. U. D. E. (2024) · Translational and Regulatory Sciences · doi:10.33611/trs.2024-005 · Cited by 9
Improving Industrial Question Answering Chatbots with Domain-Specific LLMs Fine-Tuning
Rosati, R., Antonini, F., Muralikrishna, N., Tonetto, F., & Mancini, A. (2024) · IEEE/ASME International Conference on Mechatronic and Embedded Systems and Applications · doi:10.1109/mesa61532.2024.10704843 · Cited by 18
DAM: A Universal Dual Attention Mechanism for Multimodal Timeseries Cryptocurrency Trend Forecasting
Fu, Y., Zhou, M., & Zhang, L. (2024) · IEEE International Conference on Metaverse Computing, Networking and Applications (MetaCom) · doi:10.1109/metacom62920.2024.00025 · Cited by 14
RTA: A reinforcement learning-based temporal knowledge graph question answering model
Zhu, Y., Ma, T., Sun, S., Rong, H., Bian, Y., & Huang, K. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128994 · Cited by 9
Impression-Aware Recommender Systems
Maurera, F. B. P., Dacrema, M. F., Castells, P., & Cremonesi, P. (2025) · ACM Transactions on Recommender Systems · doi:10.1145/3712292 · Cited by 11
Wang, Z., Cui, W., Huang, C., Zhou, Y., Zhao, Z., Yue, Y., Dong, X., & Lv, C. (2025) · Agriculture · doi:10.3390/agriculture15030305 · Cited by 10
SoK: Grouping Spam and Phishing Email Threats for Smarter Security
Saka, T., Vaniea, K. E., & Kökciyan, N. (2025) · IEEE Access · doi:10.1109/access.2025.3555157 · Cited by 14
Towards Few-Shot Identification of Morality Frames using In-Context Learning
Roy, S., Nakshatri, N. S., & Goldwasser, D. (2022) · Workshop on Natural Language Processing and Computational Social Science (NLP+CSS) · doi:10.18653/v1/2022.nlpcss-1.20 · Cited by 13
Prompting and In-Context Learning: Optimizing Prompts for Mistral LargePreprint
Higginbotham, G. Z., & Matthews, N. S. (2024) · Research Square · doi:10.21203/rs.3.rs-4430993/v1 · Cited by 15
Introduction to eye tracking in mathematics education: interpretation, potential, and challenges
Schindler, M., Шварц, А. Ю., & Lilienthal, A. J. (2025) · Educational Studies in Mathematics · doi:10.1007/s10649-025-10393-1 · Cited by 11
Multi-granularity contrastive zero-shot learning model based on attribute decomposition
Wang, Y., Wang, J., Fan, Y., Chai, Q., Zhang, H., Li, X., & Li, R. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103898 · Cited by 8
A Meta-Reinforcement Learning-Based Poisoning Attack Framework Against Federated Learning
Zhou, W., Zhang, D., Wang, H., Li, J., & Jiang, M. (2025) · IEEE Access · doi:10.1109/access.2025.3538891 · Cited by 12
Efficiency and Performance Optimization in Large Language Models through IB Fine-Tuning
Jo, A. A., Raj, E. D., & Sahoo, J. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3718096 · Cited by 9
Towards algorithmic framing analysis: expanding the scope by using LLMs
Kuang, X., Liu, J., Zhang, H., & Schweighofer, S. (2025) · Journal Of Big Data · doi:10.1186/s40537-025-01092-y · Cited by 12
Artificial intelligence-driven plant bio-genomics research: a new era
Yang, L., Wang, H., Zou, M., Chai, H., & Xia, Z. (2025) · Tropical Plants · doi:10.48130/tp-0025-0008 · Cited by 11
Hierarchical Index Retrieval-Driven Wireless Network Intent Translation With LLM
Wang, J., Guo, L., Wu, J., Yan, C., Sun, H., Zhang, L., Zhuang, Z., Qi, Q., & Liao, J. (2025) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2025.3564937 · Cited by 11
Seamless optical cloud computing across edge-metro network for generative AI
Xing, S., Sun, A., Wang, C., Wang, Y., Dong, B., Hu, J., Deng, X., Yan, A., Liu, Y., Hu, F., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-61495-6 · Cited by 14
Chandra, R., Chaudhari, A., & Rayavarapu, Y. (2025) · IEEE Access · doi:10.1109/access.2025.3585629 · Cited by 16
Fan, J., Wang, D., & Zheng, Y. (2025) · Sustainability · doi:10.3390/su17156971 · Cited by 11
Akterujjaman, S. M., de Ru, A., Jeurissen, R. J. M., & Nijhof, A. H. J. (2025) · Corporate Social Responsibility and Environmental Management · doi:10.1002/csr.70179 · Cited by 8
Wang, Z., Zhang, Y., Min, W., Guan, Q., & Yu, W. (2025) · Journal of Geography in Higher Education · doi:10.1080/03098265.2025.2549784 · Cited by 10
Leveraging ChatGPT and explainable AI for enhancing clinical decision support
Shawi, R. E., & Jamel, L. (2025) · Scientific Reports · doi:10.1038/s41598-025-22784-8 · Cited by 12
MechRAG: a multimodal large language model for mechanical engineering
Li, S., & Corney, J. (2025) · Communications Engineering · doi:10.1038/s44172-025-00517-z · Cited by 14
Time Series Language Model for descriptive caption generation
Trabelsi, M., Boyd, A., Cao, J., & Uzunalioğlu, H. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.112673 · Cited by 8
A comprehensive review on deep learning for multi-objective optimization
Ma, H., Liu, J., Li, J., & Jia, J. (2026) · Swarm and Evolutionary Computation · doi:10.1016/j.swevo.2026.102304 · Cited by 9
Zhang, Y., Huang, T., Liu, C., Miller, A. N., Yang, M., Harris, I. A., Sawaguchi, T., Miclau, T., Tian, M., Chui, C. S., et al. (2026) · Digital Health · doi:10.1177/20552076251412989 · Cited by 10
The Latest in Natural Language Generation: Trends, Tools and Applications in Industry
Skrodelis, H. K., Romānovs, A., Zeņina, N., & Gorskis, H. (2023) · IEEE Workshop on Advances in Information, Electronic and Electrical Engineering (AIEEE) · doi:10.1109/aieee58915.2023.10134841 · Cited by 15
Improvements in viral gene annotation using large language models and soft alignments
Harrigan, W., Ferrell, B. D., Wommack, K. E., Polson, S. W., Schreiber, Z. D., & Belcaid, M. (2024) · BMC Bioinformatics · doi:10.1186/s12859-024-05779-6 · Cited by 13
An Overview on Large Language Models
Dakhel, A. M., Nikanjam, A., Khomh, F., Desmarais, M. C., & Washizaki, H. (2024) · Springer eBooks · doi:10.1007/978-3-031-55642-5_1 · Cited by 15
Toward Reliable Biodiversity Information Extraction From Large Language Models
Elliott, M. J., & Fortes, J. A. B. (2024) · IEEE International Conference on eScience · doi:10.1109/e-science62913.2024.10678666 · Cited by 12
A diachronic language model for long-time span classical Chinese
Wei, Y., Li, M., Zhu, Y., Xu, Y., Li, Y., & Wu, B. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103925 · Cited by 11
What Comes After Transformers? A Selective Survey Connecting Ideas in Deep LearningGPT
Schneider, J. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-87330-0_4 · Cited by 17
AI-Driven User Story Generation
dos Santos, C. A., Bouchard, K., & Petrillo, F. D. S. (2024) · International Conference on Artificial Intelligence, Computer, Data Sciences and Applications (ACDSA) · doi:10.1109/acdsa59508.2024.10467677 · Cited by 8
Measuring the Perceived IQ of Multimodal Large Language Models Using Standardized IQ TestsPreprint
Wasilewski, E., & Jablonski, M. (2024) · doi:10.36227/techrxiv.171560572.29045385/v1 · Cited by 48
Salimi, E. A., & Hajinia, M. (2024) · Research Square · doi:10.21203/rs.3.rs-5534554/v1 · Cited by 9
Robertson, T. L., Ellis, M., Bratcher-Petersen, N., Ruidíaz, M. E., Harada, K., Toburen, D., Oberhauser, J. P., Grzenda, D., Peltier, N. E., Raza, M., et al. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.12.18.629281 · Cited by 9
Abstractive text summarization using deep learning models: a survey
Salam, M. A., Gamal, M., Hamed, H. F. A., & Sweidan, S. (2025) · International Journal of Data Science and Analytics · doi:10.1007/s41060-025-00743-w · Cited by 12
García-Méndez, S., de Arriba-Pérez, F., & Costa‐Montenegro, E. (2025) · Applied Sciences · doi:10.3390/app15126476 · Cited by 8
Transfer learning approach for identifying negative sentiment in tweets directed to football players
Mullah, N. S., Zainon, W. M. N. W., & Wahab, M. N. A. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108377 · Cited by 7
RESTLess: Enhancing State-of-the-Art REST API Fuzzing With LLMs in Cloud Service Computing
Zheng, T., Shao, J., Dai, J., Jiang, S., Chen, X., & Chang-xiang, S. (2024) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2024.3489441 · Cited by 17
Leveraging cognitive computing for advanced behavioral and emotional data insights
Du, C., & Wang, W. (2024) · International Journal of Cognitive Computing in Engineering · doi:10.1016/j.ijcce.2024.12.005 · Cited by 7
Zou, L., Li, J., Chen, H., Liang, M., Jia, K., Zhong, Y., & Chen, J. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.126660 · Cited by 7
Luo, E., Wei, L., Hao, M., Zhang, X., & Liu, Q. (2026) · Nature Communications · doi:10.1038/s41467-026-71744-x · Cited by 14
Wang, T., Bai, T., Xu, C., Zhang, E., Liu, B., Zhao, X., & Zhang, H. (2025) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2025.3567508 · Cited by 9
Emails by LLMs: A Comparison of Language in AI-Generated and Human-Written Emails
Li, W., Lai, Y., Soni, S., & Saha, K. (2025) · ACM Web Science Conference (WebSci) · doi:10.1145/3717867.3717872 · Cited by 19
IVCR-200K: A Large-Scale Multi-turn Dialogue Benchmark for Interactive Video Corpus Retrieval
Han, N., YaWen, Z., Long, S., Li, C., Yang, S., Tan, D., Dong, J., & Chen, J. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730015 · Cited by 8
Put Teacher in Student's Shoes: Cross-Distillation for Ultra-compact Model Compression Framework
Wang, M., Chu, J., Xie, S., Zang, X., Zhao, Y., Zhong, W., & Zhao, X. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737257 · Cited by 15
Cole, C. H., Hajikhani, A., Hylkilä, E., Paronen, E., & Pihkola, H. (2025) · The International Journal of Life Cycle Assessment · doi:10.1007/s11367-025-02508-w · Cited by 14
An LLM-Enabled Multi-Agent Autonomous Mechatronics Design Framework
Wang, Z., Lo, F. P.-W., Chen, Q., Zhang, Y., Chen, L., Chen, X., Yu, Z., Thompson, A. J., Yeatman, E. M., & Lo, B. P. L. (2025) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw67362.2025.00404 · Cited by 11
The Rise of Large Language Models: Evolution, Applications, and Future Directions
Rahmani, A. M., Hemmati, A., & Abbasi, S. (2025) · Engineering Reports · doi:10.1002/eng2.70368 · Cited by 9
Challenges and opportunities of automated essay scoring for low-proficient L2 English writers
Wilde, V. D., & Clercq, O. D. (2025) · Assessing Writing · doi:10.1016/j.asw.2025.100982 · Cited by 10
Is This the Best Prompt? Scoring Prompts for Arabic NLP Across LLMs
Refai, D., Al-shaibani, M. S., & Ahmad, I. (2025) · IEEE Access · doi:10.1109/access.2025.3616181 · Cited by 10
Xie, F., Chung, P., Reiss, J. D., Tjoa, E., Francesco, D. D., Phongpreecha, T., Haberkorn, W. J., Chakraborty, D., Chang, A. L., James, T. K., et al. (2025) · The Lancet Digital Health · doi:10.1016/j.landig.2025.100926 · Cited by 8
Node importance estimation leveraging LLMs for semantic augmentation in knowledge graphs
Lin, X., Zhang, T., Hou, C., Wang, J., Xue, J., & Lv, H. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114521 · Cited by 9
Domain adaptation of large language models for geotechnical applications
Fan, L., Liu, F., & Chen, C. (2026) · Solid Earth Sciences · doi:10.1016/j.sesci.2025.100285 · Cited by 9
Saleem, M., Rehman, Z. U., Ali, R. H., Akmal, U., Ijaz, A. Z., & Abbas, R. M. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2026.104613 · Cited by 8
Computing recommendations from free-form text
Eberhard, L., Popova, K., Walk, S., & Helić, D. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121268 · Cited by 11
Song, Y., & Tang, X. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127583 · Cited by 9
Leveraging Large Language Models for Tradespace Exploration
Apaza, G., & Selva, D. (2024) · Journal of Spacecraft and Rockets · doi:10.2514/1.a35834 · Cited by 7
Towards Fusing Data and Expert Knowledge for Better-Informed Digital Twins: An Initial Framework
Jungmann, M., & Lazarova‐Molnar, S. (2024) · Procedia Computer Science · doi:10.1016/j.procs.2024.06.072 · Cited by 10
Information Professionals’ Metaphorical Perceptions of Artificial Intelligence Concept
Kavak, A., & Yılmaz, E. (2024) · Journal of Library Administration · doi:10.1080/01930826.2024.2371276 · Cited by 8
DGGCCM: a hybrid neural model for legal event detection
Gong, S., & Luo, X. (2024) · Artificial Intelligence and Law · doi:10.1007/s10506-024-09418-6 · Cited by 7
Enhancing health assessments with large language models: A methodological approach
Wang, X., Zhou, Y., & Zhou, G. (2024) · Applied Psychology Health and Well-Being · doi:10.1111/aphw.12602 · Cited by 8
Shallow Learning Versus Deep Learning in Natural Language Processing Applications
Sawalha, L., & Akıncı, T. Ç. (2024) · The Springer series in applied machine learning · doi:10.1007/978-3-031-69499-8_8 · Cited by 9
Large Language Model for Low-Carbon Energy Transition: Roles and Challenges
Cheng, Y., Zhou, X., Zhao, H. Q., Gu, J., Wang, X., & Zhao, J. (2024) · Power System and Green Energy Conference (PSGEC) · doi:10.1109/psgec62376.2024.10721191 · Cited by 8
Chen, Q., Wang, Y., Wang, F., Sun, D., & Li, Q. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109615 · Cited by 15
Chatgpt and operations research: evaluation on the shortest path problem
Luzzi, M., Guerriero, F., Maratea, M., Greco, G., & Garofalo, M. (2025) · Soft Computing · doi:10.1007/s00500-025-10505-2 · Cited by 9
Tao, Y., Yang, R., Zeng, K., Yin, C., Lu, Y., Lu, W., Shi, Y., Wang, B., & Huang, B. (2025) · IEEE Access · doi:10.1109/access.2025.3549819 · Cited by 8
BanglaLem: A Transformer-based Bangla Lemmatizer with an Enhanced Dataset
Islam, M. F., Hasan, J., Islam, M. A., Dewan, P., & Rahman, M. S. (2025) · Systems and Soft Computing · doi:10.1016/j.sasc.2025.200244 · Cited by 8
Machine Learning-Based Data Quality Assessment for the Textile and Clothing Digital Product Passport
Cruz, E. F., Silva, P., Serra, S., Rodrigues, R., Alves, M., de Oliveira, J. F. G., & da Cruz, A. M. R. (2025) · Applied Sciences · doi:10.3390/app151810259 · Cited by 9
A literature review of literature reviews in pattern analysis and machine intelligence
Zhao, P., Zhang, X., Cao, J., Cheng, M.-M., Yang, J., & Li, X. (2026) · Science China Information Sciences · doi:10.1007/s11432-025-4816-6 · Cited by 13
Schepens, J. J., Woloszyn, H., Marx, N., & Gagl, B. (2023) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/gm9b6 · Cited by 10
LLaMALoop: Enhancing Information Retrieval in LLaMA with Semantic Relevance Feedback LoopPreprint
Tsai, H.-C., Kuo, C.-W., & Huang, Y.-F. (2023) · Research Square · doi:10.21203/rs.3.rs-3810235/v1 · Cited by 8
Probabilistic Inference Layer Integration in Mistral LLM for Accurate Information RetrievalPreprint
Wang, B., Wang, S., & Ouyang, Q. (2024) · Research Square · doi:10.21203/rs.3.rs-3828707/v1 · Cited by 8
AI as an accelerator for defining new problems that transcends boundaries
Obafemi-Ajayi, T., Jennings, S., Zhang, Y., Liu, K. L., Peckham, J., & Moore, J. H. (2025) · BioData Mining · doi:10.1186/s13040-025-00429-w · Cited by 8
Helpful or Harmful? Exploring the Efficacy of Large Language Models for Online Grooming Prevention
Prosser, E., & Edwards, M. (2024) · European Interdisciplinary Cybersecurity Conference · doi:10.1145/3655693.3655694 · Cited by 6
“Large Language Models (LLMs) [25, 34], sometimes referred to as Pre-trained Language Models (PLMs) [14, 21, 27], are an advanced form of Language Model (LM) [2, 10, 32] that train deep learning algorithms on massive amounts of data, with up to billions of parameters, allowing for exceptional…”
BackgroundNi, W., & Li, J. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1856854 · Cited by 6
An Adaptive Language-Agnostic Pruning Method for Greener Language Models for Code
Saad, M., López, J. A. H., Chen, B., Varró, D., & Sharma, T. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3715773 · Cited by 20
Seekg: Sentiment analysis for E-Learning evaluation incorporating knowledge graphs
Yi, W., Huang, X., Kuzmin, S. A., Герасимов, И. В., & Luo, Y. (2025) · Education and Information Technologies · doi:10.1007/s10639-025-13456-1 · Cited by 7
de Souza, A. F., Verri, F. A. N., Campos, P. H. D. S., & Balestrassi, P. P. (2025) · The International Journal of Advanced Manufacturing Technology · doi:10.1007/s00170-025-15701-w · Cited by 7
EHMQA-GPT: A Knowledge Augmented Large Language Model for Personalized Elderly Health Management
Lin, S., Duan, Y., Zhou, T., Liu, X., & Wang, J. (2025) · Information · doi:10.3390/info16060467 · Cited by 8
Yang, Q., Gan, L., Mao, Q., Xu, Y., Shu, Y., & Liu, C. (2025) · Ocean Engineering · doi:10.1016/j.oceaneng.2025.121963 · Cited by 8
GPT Applications for Construction Safety: A Use Case Analysis
Katooziani, A., Jeelani, I., & Gheisari, M. (2025) · Buildings · doi:10.3390/buildings15142410 · Cited by 12
Nonlinear Neural Dynamics and Classification Accuracy in Reservoir Computing
Metzner, C., Schilling, A., Maier, A., & Krauß, P. (2025) · Neural Computation · doi:10.1162/neco_a_01770 · Cited by 13
Tsang, Y. P., Wu, C. H., Wang, Y., & Ip, W. H. (2025) · International Journal on Semantic Web and Information Systems · doi:10.4018/ijswis.385572 · Cited by 6
Automatic Evaluation of Programming Tasks Supported by Language Models
Karsa, Z. I., & Goldschmidt, B. (2025) · IEEE Access · doi:10.1109/access.2025.3601448 · Cited by 7
Advancing text adversarial example generation using large language models
Madrueño, N., Fernández-Isabel, A., Fernández, R. R., & de Diego, I. M. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114361 · Cited by 12
Analysis of Large Language Models for Company Annual Reports Based on Retrieval-Augmented Generation
Mokashi, A., Puthuparambil, B., Daniel, C., & Hanne, T. (2025) · Information · doi:10.3390/info16090786 · Cited by 7
Intent-Based Autonomous Network Framework Guided by Large Language Model
Guo, L., Zhang, L., Wang, J., Wu, J., Yan, Y., Sun, H., He, B., Qi, Q., & Liao, J. (2025) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2025.3610906 · Cited by 9
JPPO: Joint Power and Prompt Optimization for Accelerated Large Language Model Services
You, F., Du, H., Huang, K., & Jamalipour, A. (2025) · International Conference on Communications · doi:10.1109/icc52391.2025.11161529 · Cited by 6
Hou, J., Wang, H., Li, T. K., & Dong, E. (2025) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2025.105368 · Cited by 9
TFMR-RAG: a knowledge base for traffic facility management regulations
Gan, L., Yan, C., Wang, T., & Zhang, W.-B. (2025) · Transportmetrica A: Transport Science · doi:10.1080/23249935.2025.2581158 · Cited by 7
scKGBERT: a knowledge-enhanced foundation model for single-cell transcriptomics
Li, Y. E., Qiao, G., Du, H., Gao, X., & Wang, G. (2025) · Genome biology · doi:10.1186/s13059-025-03862-6 · Cited by 6
GLoRA: a novel parameter-efficient fine-tuning framework for GIS large language models
Zhang, Y., Li, J., Zhang, W., Wang, Z., Zeng, Z., Luo, W., Guan, Q., Lin, J., & Yu, W. (2025) · International Journal of Geographical Information Systems · doi:10.1080/13658816.2025.2591830 · Cited by 7
Assessing Political Inclination of Bangla Language Models
Thapa, S., Maratha, A., Hasib, K. M., Nasim, M., & Naseem, U. (2023) · Workshop on Bangla Language Processing (BLP) · doi:10.18653/v1/2023.banglalp-1.8 · Cited by 22
Verma, D., Mumm, O., & Carlow, V. M. (2026) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2026.100277 · Cited by 16
Uncertainty Estimation for Complex Text Detection in Spanish
Abreu-Cardenas, M., Calderón-Ramírez, S., & Solís, M. (2023) · IEEE International Conference on BioInspired Processing (BIP) · doi:10.1109/bip60195.2023.10379278 · Cited by 6
Metadata
- Authors
- Min, B., Ross, H., Sulem, E., Veyseh, A. P. B., Nguyen, T. H., Sainz, O., Agirre, E., Heintz, I., & Roth, D.
- Affiliations
- Amazon (United States) (US)
- Harvard University (US)
- University of Pennsylvania (US)
- University of Oregon (US)
- University of the Basque Country (ES)
- Synoptic Engineering, USA
- ORCID
- Venue
- ACM Computing Surveys
- Published
- 2023-09-14
- Volume
- 56
- Issue
- 2
- Pages
- 1-40
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W4382246105
- Semantic Scholar
- c23d9d44e8bc68408cea9f305d1f24d915bc0d0d
- Cited by
- 1,387
- References
- 160
- Citations by year
- 2026: 285 · 2025: 521 · 2024: 461 · 2023: 98
- FWCI
- 128.67
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Computing Surveys (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- U.S. Department of Defense
- Office of the Director of National Intelligence
- Defense Advanced Research Projects Agency (FA8750-19-2-1004, FA8750)
- Advanced Research Projects Agency
- Intelligence Advanced Research Projects Activity (2019-19051600006)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- large pre-trained language models, BERT, GPT, transfer learning, fine-tuning, prompting, text generation, language representation learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.