Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey
Open accessThe DOI matches its Crossref record; 185 of the 185 references it cites were found in a registry or an index.
Abstract
Large, pre-trained language models (PLMs) such as BERT and GPT have drastically changed the Natural Language Processing (NLP) field. For numerous NLP tasks, approaches leveraging PLMs have achieved state-of-the-art performance. The key idea is to learn a generic, latent representation of language from a generic task once, then share it across disparate NLP tasks. Language modeling serves as the generic task, one with abundant self-supervised text available for extensive training. This article presents the key fundamental concepts of PLM architectures and a comprehensive view of the shift to PLM-driven NLP techniques. It surveys work applying the pre-training then fine-tuning, prompting, and text generation approaches. In addition, it discusses PLM limitations and suggested directions for future research.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“However, the field of AI is evolving rapidly, particularly in areas such as Generative AI (Brynjolfsson et al. 2023) and Large Language Models (Min et al. 2023).”
From Artificial intelligence for literature reviews: opportunities and challenges (2024) · cites this paper as Min et al. 2023
In the citing paper
“However, the open-ended nature of language means that there can be multiple valid responses to a given prompt, complicating the assessment of correctness [1].”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [1]
In the citing paper
“LLMs like Llama and GPT-4 are capable of understanding and generating human-like text, performing tasks ranging from translation to creative writing [1], [4].”
From Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence (2026) · cites this paper as [1]
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,887
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Research works printed but not linked: 56
- Zeyuan Allen-Zhu and Yuanzhi Li. 2021. Towards understanding ensemble knowledge distillation and self-distillation in deep learning. https://arxiv.org/abs/2012.09816 · arXiv
- Asaf Amrami and Yoav Goldberg. 2019. Towards better substitution-based word sense induction. https://arxiv.org/abs/1905.12598 · arXiv
- Stephen H. Bach Victor Sanh Zheng-Xin Yong Albert Webson Colin Raffel Nihal V. Nayak Abheesht Sharma Taewoon Kim M. Saiful Bari Thibault Fevry Zaid Alyafeai Manan Dey Andrea Santilli Zhiqing Sun Srulik Ben-David Canwen Xu Gunjan Chhablani Han Wang Jason Alan Fries Maged S. Al-shaibani Shanya Sharma Urmish Thakker Khalid Almubarak Xiangru Tang Dragomir Radev Mike Tian-Jian Jiang and Alexander M. Rush. 2022. PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts. https://arxiv.org/abs/2202.01279 · arXiv
- Geoff Bacon and Terry Regier. 2019. Does BERT agree? Evaluating knowledge of structure dependence through agreement relations. https://arxiv.org/abs/1908.09892 · arXiv
- Jack Bandy and Nicholas Vincent. 2021. Addressing “Documentation Debt” in Machine Learning Research: A Retrospective Datasheet for BookCorpus. https://arxiv.org/abs/2105.05241 · arXiv
- Eyal Ben-David Nadav Oved and Roi Reichart. 2021. PADA: A prompt-based autoregressive approach for adaptation to unseen domains. https://arxiv.org/abs/2102.12206 · arXiv
- Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota.
- 10.5555/2463101 · doi:10.5555/2463101
- Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. 2017. Language modeling with gated convolutional networks. In Proceedings of the International Conference on Machine Learning. PMLR, 933–941.
- Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021. Autoregressive entity retrieval. In Proceedings of the 9th International Conference on Learning Representations (ICLR’21).
- Avia Efrat and Omer Levy. 2020. The turking test: Can language models understand instructions?https://arxiv.org/abs/2010.11982 · arXiv
- Tom Brown et al.2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 1877–1901.
- Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations.
- Steven Fincke Shantanu Agarwal Scott Miller and Elizabeth Boschee. 2021. Language model priming for cross-lingual event extraction. https://arxiv.org/abs/2109.12383 · arXiv
- Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020. Online.
- Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning. PMLR.
- Minyoung Huh Pulkit Agrawal and Alexei A. Efros. 2016. What makes ImageNet good for transfer learning?arXiv: 1608.08614. · arXiv
- Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. UNIFIEDQA: Crossing format boundaries with a single QA system. In Findings of the Association for Computational Linguistics: EMNLP 2020.
- Sawan Kumar and Partha Talukdar. 2021. Reordering examples helps during priming-based few-shot learning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. Association for Computational Linguistics, Online.
- Huiyuan Lai Antonio Toral and Malvina Nissim. 2021. Thank you BART! Rewarding pre-trained models improves formality style transfer. https://arxiv.org/abs/2105.06947 · arXiv
- Fayuan Li, Weihua Peng, Yuguang Chen, Quan Wang, Lu Pan, Yajuan Lyu, and Yong Zhu. 2020. Event extraction as multi-turn question answering. In Findings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics, Online.
- Junyi Li Tianyi Tang Wayne Xin Zhao and Ji-Rong Wen. 2021. Pretrained language models for text generation: A survey. https://arxiv.org/abs/2105.10311 · arXiv
- Jiachang Liu Dinghan Shen Yizhe Zhang Bill Dolan Lawrence Carin and Weizhu Chen. 2021. What makes good in-context examples for GPT-3?. https://arxiv.org/abs/2101.06804 · arXiv
- Pengfei Liu Weizhe Yuan Jinlan Fu Zhengbao Jiang Hiroaki Hayashi and Graham Neubig. 2021. Pre-train prompt and predict: A systematic survey of prompting methods in natural language processing. arXiv: 2107.13586. · arXiv
- Xiao Liu Yanan Zheng Zhengxiao Du Ming Ding Yujie Qian Zhilin Yang and Jie Tang. 2021. GPT understands too. https://arxiv.org/abs/2103.10385 · arXiv
- Brian MacWhinney. 2000. The CHILDES Project: Tools for Analyzing Talk. Transcription Format and Programs. Vol. 1. Psychology Press.
- Xuan-Phi Nguyen Shafiq Joty Steven C. H. Hoi and Richard Socher. 2020. Tree-structured attention with hierarchical accumulation. https://arxiv.org/abs/2002.08046 · arXiv
- Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020. Document ranking with a pretrained sequence-to-sequence model. In Findings of the Association for Computational Linguistics (EMNLP).
- Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, Rishita Anubhai, Cicero dos Santos Nogueira, Bing Xiang, and Stefano Soatto. 2021. Structured prediction as translation between augmented natural languages. In Proceedings of the 9th International Conference on Learning Representations (ICLR’21).
- Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. 2020. How context affects language models’ factual predictions. In Proceedings of AKBC.
- Raul Puri and Bryan Catanzaro. 2019. Zero-shot text classification with generative language models. https://arxiv.org/abs/1912.10165 · arXiv
- Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. OpenAI Blog (2018), 12.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.
- Evani Radiya-Dixit and Xin Wang. 2020. How fine can fine-tuning be? Learning efficient language models. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics(Proceedings of Machine Learning Research, Vol. 108), Silvia Chiappa and Roberto Calandra (Eds.). 2435–2443.
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research (2020).
- Liliang Ren, Chenkai Sun, Heng Ji, and Julia Hockenmaier. 2021. HySPA: Hybrid span generation for scalable text-to-graph extraction. In Findings of the Association for Computational Linguistics (ACL-IJCNLP’21).
- Victor Sanh Lysandre Debut Julien Chaumond and Thomas Wolf. 2020. DistilBERT a distilled version of BERT: Smaller faster cheaper and lighter. https://arxiv.org/abs/1910.01108 · arXiv
- Victor Sanh Albert Webson Colin Raffel Stephen H. Bach Lintang Sutawika Zaid Alyafeai Antoine Chaffin Arnaud Stiegler Teven Le Scao Arun Raja Manan Dey M. Saiful Bari Canwen Xu Urmish Thakker Shanya Sharma Sharma Eliza Szczechla Taewoon Kim Gunjan Chhablani Nihal Nayak Debajyoti Datta Jonathan Chang Mike Tian-Jian Jiang Han Wang Matteo Manica Sheng Shen Zheng Xin Yong Harshit Pandey Rachel Bawden Thomas Wang Trishala Neeraj Jos Rozen Abheesht Sharma Andrea Santilli Thibault Fevry Jason Alan Fries Ryan Teehan Tali Bers Stella Biderman Leo Gao Thomas Wolf and Alexander M. Rush. 2022. Multitask prompted training enables zero-shot task generalization. https://arxiv.org/abs/2110.08207 · arXiv
- Timo Schick and Hinrich Schütze. 2019. Rare words: A major problem for contextualized embeddings and how to fix it by attentive mimicking. https://arxiv.org/abs/1904.06707 · arXiv
- Timo Schick and Hinrich Schütze. 2020. Few-shot text generation with pattern-exploiting training. (2020). https://arxiv.org/abs/2012.11926 · arXiv
- Fábio Souza Rodrigo Nogueira and Roberto Lotufo. 2020. Portuguese named entity recognition using BERT-CRF. https://arxiv.org/abs/1909.10649 · arXiv
- Asa Cooper Stickland and Iain Murray. 2019. BERT and PALs: Projected attention layers for efficient adaptation in multi-task learning. In Proceedings of the 36th International Conference on Machine Learning. PMLR, 5986–5995.
- Ehsan Taher, Seyed Abbas Hoseini, and Mehrnoush Shamsfard. 2019. Beheshti-NER: Persian named entity recognition using BERT. In Proceedings of the 1st International Workshop on NLP Solutions for Under Resourced Languages (NSURL’19) co-located with ICNLSP 2019 - Short Papers. Association for Computational Linguistics, Trento, Italy, 37–42.
- Trieu H. Trinh and Quoc V. Le. 2019. A simple method for commonsense reasoning. https://arxiv.org/abs/1806.02847 · arXiv
- Maria Tsimpoukelli Jacob Menick Serkan Cabi S. M. Ali Eslami Oriol Vinyals and Felix Hill. 2021. Multimodal few-shot learning with frozen language models. https://arxiv.org/abs/2106.13884 · arXiv
- Sinong Wang Han Fang Madian Khabsa Hanzi Mao and Hao Ma. 2021. Entailment as few-shot learner. https://arxiv.org/abs/2104.14690 · arXiv
- Thomas Wang Adam Roberts Daniel Hesslow Teven Le Scao Hyung Won Chung Iz Beltagy Julien Launay and Colin Raffel. 2022. What language model architecture and pPretraining objective work bBest for zero-shot generalization?https://arxiv.org/abs/2204.05832 · arXiv
- Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: A benchmark of linguistic minimal pairs for English. In Proceedings of the Society for Computation in Linguistics 2020.
- Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. How transferable are features in deep neural networks?. Advances in Neural Information Processing Systems, Vol. 27. Curran Associates, Inc.
- Wenhao Yu Chenguang Zhu Zaitang Li Zhiting Hu Qingyun Wang Heng Ji and Meng Jiang. 2021. A survey of knowledge-enhanced text generation. https://arxiv.org/abs/2010.04389 · arXiv
- Weizhe Yuan Graham Neubig and Pengfei Liu. 2021. BARTScore: Evaluating generated text as text generation. https://arxiv.org/abs/2106.11520 · arXiv
- Karolina Zaczynska, Nils Feldhus, Robert Schwarzenberg, Aleksandra Gabryszak, and Sebastian Möller. 2020. Evaluating German transformer language models with syntactic agreement tests, In Proceedings of the 5th Swiss Text Analytics Conference and the 16th Conference on Natural Language Processing (SwissText/KONVENS’20). CoRR abs/2007.03765. · arXiv
- Tony Z. Zhao Eric Wallace Shi Feng Dan Klein and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. https://arxiv.org/abs/2102.09690 · arXiv
- Junru Zhou, Zhuosheng Zhang, Hai Zhao, and Shuailiang Zhang. 2020. LIMIT-BERT: Linguistics informed multi-task BERT. In Findings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics, Online.
- Li Zhou and Kevin Small. 2020. Multi-domain dialogue state tracking as dynamic knowledge graph enhanced question answering. https://arxiv.org/abs/1911.06192 · arXiv
- Jinhua Zhu Yingce Xia Lijun Wu Di He Tao Qin Wengang Zhou Houqiang Li and Tie-Yan Liu. 2020. Incorporating BERT into neural machine translation. https://arxiv.org/abs/2002.06823 · arXiv
Cited by
Items 201–300 of 1,014
An Investigation of Applying Large Language Models to Spoken Language Learning
Gao, Y., Nuchged, B., Li, Y., & Peng, L. (2023) · Applied Sciences · doi:10.3390/app14010224 · Cited by 41
Knowledge graph construction for heart failure using large language models with prompt engineering
Xu, T., Gu, Y., Xue, M., Gu, R., Li, B., & Gu, X. (2024) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2024.1389475 · Cited by 40
Wang, J., Karatzoglou, A., Arapakis, I., & Jose, J. M. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657767 · Cited by 58
Large Language Models for Conducting Advanced Text Analytics Information Systems Research
Ampel, B., Yang, C.-H., Hu, J. L., & Chen, H. (2024) · ACM Transactions on Management Information Systems · doi:10.1145/3682069 · Cited by 41
Sun, Z., Hong, W., Xue, C., & Dong, N. (2024) · The Science of The Total Environment · doi:10.1016/j.scitotenv.2024.175971 · Cited by 38
Vision-language model-driven scene understanding and robotic object manipulation
Liu, S., Zhang, J., Gao, R. X., Wang, X. V., & Wang, L. (2024) · IEEE International Conference on Automation Science and Engineering (CASE) · doi:10.1109/case59546.2024.10711845 · Cited by 57
Multiknowledge and LLM-Inspired Heterogeneous Graph Neural Network for Fake News Detection
Xie, B., Ma, X., Xue, S., Beheshti, A., Yang, J. Y., Fan, H., & Wu, J. X. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3488191 · Cited by 40
Quantum mixed-state self-attention network
Fu, C., Zhao, Q., Feng, L., Chen, C., Lin, Y., & Lin, J. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107123 · Cited by 54
Evolutionary Reinforcement Learning: A Systematic Review and Future Directions
Lin, Y., Lin, F., Cai, G., Chen, H., Zou, L., Liu, Y., & Wu, P. (2025) · Mathematics · doi:10.3390/math13050833 · Cited by 45
AutoProSAM: Automated Prompting SAM for 3D Multi-Organ Segmentation
Li, C., Sultan, R. I., Khanduri, P., Qiang, Y., Chetty, I. J., & Zhu, D. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00352 · Cited by 68
Ono, K., & Morita, A. (2024) · doi:10.36227/techrxiv.170956672.21573677/v1 · Cited by 38
Taneja, K. K., Vashishtha, J., & Ratnoo, S. (2024) · Procedia Computer Science · doi:10.1016/j.procs.2024.04.220 · Cited by 32
iSign: A Benchmark for Indian Sign Language Processing
Joshi, A., Mohanty, R., Kanakanti, M., Mangla, A., Choudhary, S., Barbate, M., & Modi, A. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.643 · Cited by 53
Torkamaan, H., Steinert, S., Pera, M. S., Kudina, O., Freire, S. K., Verma, H., Kelly, S., Sekwenz, M.-T., Yang, J., van Nunen, K., et al. (2024) · Behaviour and Information Technology · doi:10.1080/0144929x.2024.2431068 · Cited by 43
Fayyazi, R., Taghdimi, R., & Yang, S. J. (2024) · Annual Computer Security Applications Conference Workshops (ACSAC Workshops) · doi:10.1109/acsacw65225.2024.00036 · Cited by 54
GAugLLM: Improving Graph Contrastive Learning for Text-Attributed Graphs with Large Language Models
Fang, Y., Fan, D., Zha, D., & Tan, Q. (2024) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3637528.3672035 · Cited by 74
Use of SNOMED CT in Large Language Models: Scoping Review
Chang, E., & Sung, S. (2024) · JMIR Medical Informatics · doi:10.2196/62924 · Cited by 50
Rui, J., Xu, Y., Cai, C., & Li, X. (2024) · Tourism Management · doi:10.1016/j.tourman.2024.105115 · Cited by 31
The Frontier of Data Erasure: A Survey on Machine Unlearning for Large Language Models
Qu, Y., Ding, M., Sun, N., Thilakarathna, K., Zhu, T., & Niyato, D. T. (2025) · Computer · doi:10.1109/mc.2024.3405397 · Cited by 56
Gadekallu, T. R., Yenduri, G., Kaluri, R., Rajput, D. S., Lakshmanna, K., Fang, K., Chen, J., & Wang, W. W. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.2400 · Cited by 36
Cheng, H., Gong, Z., & Wang, C. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113174 · Cited by 41
A large language model for multimodal identification of crop diseases and pests
Wang, Y.-Q., Wang, F.-H., Chen, W.-B., Lv, B., Liu, M., Kong, X., Zhao, C., & Pan, Z. (2025) · Scientific Reports · doi:10.1038/s41598-025-01908-0 · Cited by 41
Cerchione, R., Liccardo, G., & Passaro, R. (2025) · Journal of Innovation & Knowledge · doi:10.1016/j.jik.2025.100866 · Cited by 44
Textual Entailment for Event Argument Extraction: Zero- and Few-Shot with Multi-Source Learning
Sainz, O., González-Dios, I., de Lacalle, O. L., Min, B., & Agirre, E. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.187 · Cited by 57
Self-citationDurably reducing conspiracy beliefs through dialogues with AI
Costello, T. H., Pennycook, G., & Rand, D. G. (2024) · doi:10.31234/osf.io/xcwdn · Cited by 28
Collective predictive coding hypothesis: symbol emergence as decentralized Bayesian inference
Taniguchi, T. (2024) · Frontiers in Robotics and AI · doi:10.3389/frobt.2024.1353870 · Cited by 28
A Comprehensive Survey of Fake Text Detection on Misinformation and LM-Generated Texts
Kwon, S.-C., & Jang, B. (2025) · IEEE Access · doi:10.1109/access.2025.3538805 · Cited by 37
Research on Sentiment Analysis of Online Public Opinion Based on RoBERTa–BiLSTM–Attention Model
Deng, J., & Liu, Y. (2025) · Applied Sciences · doi:10.3390/app15042148 · Cited by 39
A Survey of Source Code Representations for Machine Learning-Based Cybersecurity Tasks
Casey, B., Santos, J. C. S., & Perry, G. (2025) · ACM Computing Surveys · doi:10.1145/3721977 · Cited by 35
The Mutators Reloaded: Fuzzing Compilers with Large Language Model Generated Mutation Operators
Ou, X., Li, C., Jiang, Y., & Xu, C. (2024) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/3622781.3674171 · Cited by 58
Evaluation of Large Language Models as Solution Generators in Complex Optimization
Huang, B., Wu, X., Zhou, Y. F., Wu, J., Feng, L., Cheng, R., & Tan, K. C. (2025) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2025.3580520 · Cited by 59
Prompting Frameworks for Large Language Models: A Survey
Liu, X., Wang, J., Sun, J., Yuan, X., Dong, G., Di, P., Wang, W., & Wang, D. (2023) · arXiv · doi:10.48550/arxiv.2311.12785 · Cited by 28
Panchendrarajan, R., & Zubiaga, A. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124097 · Cited by 37
Ding, N., Yuan, Z., Ma, Z., Wu, Y., & Yin, L. (2024) · Molecules · doi:10.3390/molecules29153512 · Cited by 30
Bonfigli, A., Bacco, L., Merone, M., & Dell’Orletta⋄, F. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.103003 · Cited by 36
ActuaryGPT: applications of large language models to insurance and actuarial work
Balona, C. (2024) · British Actuarial Journal · doi:10.1017/s1357321724000102 · Cited by 41
Xie, T., Kuang, Y., Tang, Y., Liao, J., & Yang, Y. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.126371 · Cited by 39
Aloudat, M. Z., Aboumadi, A., Soliman, A., Al-Mohammed, H. A., Al-Ali, M. S., Mahgoub, A., Barhamgi, M., & Yaacoub, E. E. (2025) · IEEE Access · doi:10.1109/access.2025.3555753 · Cited by 46
Rocca, R., Tamagnone, N., Fekih, S., Contla, X., & Rekabsaz, N. (2023) · Frontiers in Big Data · doi:10.3389/fdata.2023.1082787 · Cited by 46
Revisiting Pre-trained Language Models and their Evaluation for Arabic Natural Language Processing
Ghaddar, A., Wu, Y., Bagga, S., Rashid, A., Bibi, K., Rezagholizadeh, M., Xing, C., Wang, Y., Duan, X., Wang, Z., et al. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.205 · Cited by 39
Parallel Dual-Branch Fusion Network for Epileptic Seizure Prediction
Ma, H., Wu, Y., Tang, Y., Chen, R., Xu, T., & Zhang, W. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108565 · Cited by 29
Multi-Agent Collaboration Mechanisms: A Survey of LLMs
Tran, K.-T., Dao, D., Nguyen, M.-D., Pham, Q.-V., O'Sullivan, B., & Nguyen, H. D. (2025) · arXiv · doi:10.48550/arxiv.2501.06322 · Cited by 680
“Moreover, the geopolitical and ethical implications of LLM development have become the limelight for policymakers, especially concerning the computational power required and potential misuse by malicious actors [76, 86].”
SiamQuality: a ConvNet-based foundation model for photoplethysmography signals
Ding, C., Guo, Z., Chen, Z., Lee, R. J., Rudin, C. D., & Hu, X. (2024) · Physiological Measurement · doi:10.1088/1361-6579/ad6747 · Cited by 51
FoodSky: A food-oriented large language model that can pass the chef and dietetic examinations
Zhou, P., Min, W., Fu, C., Jin, Y., Huang, M., Li, X., Mei, S., & Jiang, S. (2025) · Patterns · doi:10.1016/j.patter.2025.101234 · Cited by 43
Graph-Aware Language Model Pre-Training on a Large Graph Corpus Can Help Multiple Graph Applications
Xie, H., Zheng, D., Ma, J., Zhang, H., Ioannidis, V. N., Song, X., Ping, Q., Wang, S., Yang, C., Xu, Y. T., et al. (2023) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3580305.3599833 · Cited by 57
Towards human-AI collaborative urban science research enabled by pre-trained large language models
Fu, J., Han, H., Su, X., & Fan, C. H. (2024) · Urban Informatics · doi:10.1007/s44212-024-00042-y · Cited by 36
A Map of Exploring Human Interaction Patterns with LLM: Insights into Collaboration and Creativity
Li, J., Li, J., & Su, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-60615-1_5 · Cited by 44
Nashaat, M., & Miller, J. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3428324 · Cited by 47
Testing prompt engineering methods for knowledge extraction from text
Polat, F., Tiddi, I., & Groth, P. (2025) · Semantic Web · doi:10.3233/sw-243719 · Cited by 59
Analyzing and Mitigating Cultural Hallucinations of Commercial Language Models in TurkishPreprint
Boztemir, Y., & Çalışkan, N. (2024) · doi:10.36227/techrxiv.171504678.80895811/v1 · Cited by 26
“A Comparative Analysis of Automatic Speech Recognition Errors in Small Group Classroom Discourse.”
Cao, J., Ganesh, A., Cai, J., Southwell, R., Perkoff, E. M., Regan, M., Kann, K., Martin, J. H., Palmer, M., & D'Mello, S. (2023) · Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization · doi:10.1145/3565472.3595606 · Cited by 25
Zahid, I. A., Joudar, S. S., Albahri, A. S., Albahri, O. S., Alamoodi, A. H., Santamaría, J., & Alzubaidi, L. (2024) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2024.200431 · Cited by 38
Ivanisenko, T., Сергеевич, Д. П., & Иванисенко, В. А. (2024) · International Journal of Molecular Sciences · doi:10.3390/ijms252111811 · Cited by 40
FELLAS: Enhancing Federated Sequential Recommendation with LLM as External Services
Yuan, W., Yang, C., Ye, G., Chen, T., Nguyen, Q. V. H., & Yin, H. (2025) · ACM Transactions on Information Systems · doi:10.1145/3709138 · Cited by 48
Joffe, I., Felobes, G., Elgouhari, Y., Talebi-Kalaleh, M., Mei, Q., & Chui, Y. H. (2025) · Journal of Computing in Civil Engineering · doi:10.1061/jccee5.cpeng-6037 · Cited by 34
Automatic detection of Feature Envy and Data Class code smells using machine learning
Škipina, M., Сливка, Ј., Luburić, N., & Kovačević, A. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122855 · Cited by 31
Network intrusion detection system by learning jointly from tabular and text‐based features
Düzgün, B., Çayır, A., Ünal, U., & Dağ, H. (2023) · Expert Systems · doi:10.1111/exsy.13518 · Cited by 30
Ma, W., Liu, Y., Yi, Q., Liu, X., Xing, W., Zhao, R., Liu, H., & Li, R. (2025) · PLoS ONE · doi:10.1371/journal.pone.0317839 · Cited by 21
Fayyazi, R., Taghdimi, R., & Yang, S. J. (2024) · arXiv · doi:10.48550/arxiv.2401.00280 · Cited by 25
“5 model 2 , has led to substantial progress in NLP tasks (Min et al. 2023; Zhao et al. 2023).”
BackgroundCan LLMs Master Math? Investigating Large Language Models on Math Stack Exchange
Satpute, A., Gießing, N., Greiner-Petter, A., Schubotz, M., Teschke, O., Aizawa, A., & Gipp, B. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657945 · Cited by 38
A Large Language Model for Determining Partial Tripping of Distributed Energy Resources
Zhao, T., Yogarathnam, A., & Yue, M. (2024) · IEEE Transactions on Smart Grid · doi:10.1109/tsg.2024.3453649 · Cited by 26
Unleashing the Retrieval Potential of Large Language Models in Conversational Recommender Systems
Yang, T., & Chen, L. (2024) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3640457.3688146 · Cited by 50
PTF-FSR: A Parameter Transmission-Free Federated Sequential Recommender System
Yuan, W., Yang, C., Qu, L., Nguyen, Q. V. H., Ye, G., & Yin, H. (2024) · ACM Transactions on Information Systems · doi:10.1145/3708344 · Cited by 36
Liang, X., Wang, Z., & Liu, J. (2025) · Advanced Engineering Informatics · doi:10.1016/j.aei.2025.103265 · Cited by 24
Detecting mental disorder on social media: A ChatGPT-augmented explainable approach
Belcastro, L., Cantini, R., Marozzo, F., Talia, D., & Trunfio, P. (2025) · Online Social Networks and Media · doi:10.1016/j.osnem.2025.100321 · Cited by 31
Song, Y., Zhang, Y., Zhou, A., Shi, Y., Shen, S., Tang, X., Li, J., Zhang, M., & Wang, D. (2025) · IEEE Communications Magazine · doi:10.1109/mcom.001.2400342 · Cited by 28
Coal Mine Accident Risk Analysis with Large Language Models and Bayesian Networks
Du, G., & Chen, A. (2025) · Sustainability · doi:10.3390/su17051896 · Cited by 26
Kim, H., Wang, N., Xia, Q., Huang, J., Yazdanbakhsh, A., & Kim, N. S. (2025) · Proceedings - International Symposium on Computer Architecture/Proceedings · doi:10.1145/3695053.3731092 · Cited by 42
Wang, T., Zheng, L., Lv, H., Zhou, C., Shen, Y., Qiu, Q., Li, Y., Li, P., & Wang, G. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.119216 · Cited by 26
López, J. A. H., Weyssow, M., Cuadrado, J. S., & Sahraoui, H. A. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556900 · Cited by 55
N.A., A. A. T. (2024) · Future Business Journal · doi:10.1186/s43093-024-00368-8 · Cited by 31
Wong, S. M., Leung, H.-F., & Wong, K. Y. (2024) · Research Square · doi:10.21203/rs.3.rs-4063228/v1 · Cited by 60
Gamouh, H., Novotný, M., & Hoksza, D. (2025) · Bioinformatics · doi:10.1093/bioinformatics/btaf431 · Cited by 23
Ahmed, A., Hou, M., Xi, R., Zeng, X., & Shah, S. A. (2024) · Companion Proceedings of the ACM Web Conference 2024 · doi:10.1145/3589335.3651904 · Cited by 22
Yu, J. (2025) · Transport Policy · doi:10.1016/j.tranpol.2025.05.030 · Cited by 22
The role of Large Language Models in addressing IoT challenges: A systematic literature review
Vito, G. D., Palomba, F., & Ferrucci, F. (2025) · Future Generation Computer Systems · doi:10.1016/j.future.2025.107829 · Cited by 30
In-Memory Associative Processors: Tutorial, Potential, and Challenges
Fouda, M. E., Yantır, H. E., Eltawil, A. M., & Kurdahi, F. J. (2022) · IEEE Transactions on Circuits & Systems II Express Briefs · doi:10.1109/tcsii.2022.3170468 · Cited by 28
Opportunities and challenges in the application of large artificial intelligence models in radiology
Pan, L., Zhao, Z., Lu, Y., Tang, K., Fu, L., Liang, Q., & Peng, S. (2024) · Meta-Radiology · doi:10.1016/j.metrad.2024.100080 · Cited by 26
Zhang, Z., Zhu, Y., Zhang, Q., Cui, N., & Shang, Y. (2024) · Energy · doi:10.1016/j.energy.2024.133993 · Cited by 24
On Large Language Models in National Security Applications
Caballero, W. N., & Jenkins, P. R. (2025) · Stat · doi:10.1002/sta4.70057 · Cited by 51
Towards Message Brokers for Generative AI: Survey, Challenges, and Opportunities
Saleh, A., Morabito, R., Dustdar, S., Tarkoma, S., Pirttikangas, S., & Lovén, L. (2025) · ACM Computing Surveys · doi:10.1145/3742891 · Cited by 37
A Comparative Analysis to Evaluate Bias and Fairness Across Large Language Models with BenchmarksPreprint
陳文意, & 黃兆明 (2024) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/mc762 · Cited by 26
Multimodal Artificial Intelligence in Medicine
Judge, C. S., Krewer, F., O'Donnell, M. J., Kiely, L., Sexton, D. J., Taylor, G. W., Skorburg, J. A., & Tripp, B. (2024) · Kidney360 · doi:10.34067/kid.0000000000000556 · Cited by 28
Natural Language Processing for Digital Health in the Era of Large Language Models
Sarker, A., Zhang, R., Wang, Y., Xiao, Y., Das, S., Schutte, D., Oniani, D., Xie, Q., & Xu, H. (2024) · Yearbook of Medical Informatics · doi:10.1055/s-0044-1800750 · Cited by 39
Fairness and social bias quantification in Large Language Models for sentiment analysis
Radaideh, M. I., Kwon, O. H., & Radaideh, M. I. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113569 · Cited by 37
Large Language Models for Psychological Assessment: A Comprehensive Overview
Brickman, J., Gupta, M., & Oltmanns, J. R. (2025) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459251343582 · Cited by 48
Alrayzah, A., Alsolami, F., & Saleh, M. (2023) · PeerJ Computer Science · doi:10.7717/peerj-cs.1633 · Cited by 28
The Impact of Example Selection in Few-Shot Prompting on Automated Essay Scoring Using GPT Models
Yoshida, L. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-64315-6_5 · Cited by 39
Taking It Easy: Off-the-Shelf Versus Fine-Tuned Supervised Modeling of Performance Appraisal Text
Speer, A. B., Perrotta, J., & Kordsmeyer, T. L. (2024) · Organizational Research Methods · doi:10.1177/10944281241271249 · Cited by 25
Using Generative Text Models to Create Qualitative Codebooks for Student Evaluations of Teaching
Katz, A., Gerhardt, M., & Soledad, M. M. (2024) · International Journal of Qualitative Methods · doi:10.1177/16094069241293283 · Cited by 26
A survey on machine unlearning: Techniques and new emerged privacy risks
Liu, H., Xiong, P., Zhu, T., & Yu, P. S. (2025) · Journal of Information Security and Applications · doi:10.1016/j.jisa.2025.104010 · Cited by 45
Procedural knowledge management in Industry 5.0: Challenges and opportunities for knowledge graphs
Celino, I., Carriero, V. A., Azzini, A., Baroni, I., & Scrocca, M. (2024) · Journal of Web Semantics · doi:10.1016/j.websem.2024.100850 · Cited by 23
Reducing hallucinations of large language models via hierarchical semantic piece
Liu, Y., Yang, Q., Tang, J.-W., Guo, T., Chen, W., Pan, L., Sai, X., Gao, X., Li, Z., Liu, J., et al. (2025) · Complex & Intelligent Systems · doi:10.1007/s40747-025-01833-9 · Cited by 26
Cui, H., Zheng, P., Ren, M., & Yan, Y. (2025) · Advanced Engineering Informatics · doi:10.1016/j.aei.2025.104017 · Cited by 24
Using an AI-based avatar for interviewer training at Children’s Advocacy Centers: Proof of Concept
Baugerud, G. A., Johnson, M. S., Dianiska, R. E., Røed, R. K., Powell, M. B., Lamb, M. E., Hassan, S. Z., Sabet, S. S., Hicks, S. A., Salehi, P., et al. (2024) · Child Maltreatment · doi:10.1177/10775595241263017 · Cited by 23
Tc-llama 2: fine-tuning LLM for technology and commercialization applications
Yeom, J., Lee, H., Byun, H., Kim, Y., Byun, J., Choi, Y., Kim, S., & Song, K. (2024) · Journal Of Big Data · doi:10.1186/s40537-024-00963-0 · Cited by 23
Wang, H., Qiu, X., & Tan, X. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05802-6 · Cited by 27
MMPolymer: A Multimodal Multitask Pretraining Framework for Polymer Property Prediction
Wang, F., Guo, W., Cheng, M., Yuan, S., Xu, H., & Gao, Z. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679684 · Cited by 26
Wang, S., Ouyang, Q., & Wang, B. (2024) · Research Square · doi:10.21203/rs.3.rs-3987793/v1 · Cited by 63
Yang, C. T.-Y., & Chen, H. H.-J. (2025) · Computer Assisted Language Learning · doi:10.1080/09588221.2025.2453205 · Cited by 26
Metadata
- Authors
- Min, B., Ross, H., Sulem, E., Veyseh, A. P. B., Nguyen, T. H., Sainz, O., Agirre, E., Heintz, I., & Roth, D.
- Affiliations
- Amazon (United States) (US)
- Harvard University (US)
- University of Pennsylvania (US)
- University of Oregon (US)
- University of the Basque Country (ES)
- Synoptic Engineering, USA
- ORCID
- Venue
- ACM Computing Surveys
- Published
- 2023-09-14
- Volume
- 56
- Issue
- 2
- Pages
- 1-40
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W4382246105
- Semantic Scholar
- c23d9d44e8bc68408cea9f305d1f24d915bc0d0d
- Cited by
- 1,387
- References
- 160
- Citations by year
- 2026: 285 · 2025: 521 · 2024: 461 · 2023: 98
- FWCI
- 128.67
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Computing Surveys (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- U.S. Department of Defense
- Office of the Director of National Intelligence
- Defense Advanced Research Projects Agency (FA8750-19-2-1004, FA8750)
- Advanced Research Projects Agency
- Intelligence Advanced Research Projects Activity (2019-19051600006)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- large pre-trained language models, BERT, GPT, transfer learning, fine-tuning, prompting, text generation, language representation learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.