Language Model Behavior: A Comprehensive Survey
Open accessThe DOI matches its Crossref record; 428 of the 428 references it cites were found in a registry or an index.
Abstract
Transformer language models have received widespread public attention, yet their generated text is often surprising even to NLP researchers. In this survey, we discuss over 250 recent studies of English language model behavior before task-specific fine-tuning. Language models possess basic capabilities in syntax, semantics, pragmatics, world knowledge, and reasoning, but these capabilities are sensitive to specific inputs and surface features. Despite dramatic increases in generated text quality as models scale to hundreds of billions of parameters, the models are still prone to unfactual responses, commonsense errors, memorized text, and social biases. Many of these weaknesses can be framed as over-generalizations or under-generalizations of learned patterns in text. We synthesize recent results to highlight what is currently known about large language model capabilities, thus providing a resource for applied work and for research in adjacent fields that use language models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“So far, quantitative comparisons have looked mainly at basic grammar and syntax (5) or features such as word choice, punctuation, sentence length, and so on, finding evidence of some differences between human- and LLM-written text (10–14).”
From Do LLMs write like humans? Variation in grammatical and rhetorical styles (2025) · cites this paper as [5]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 3×
“Transformer language models have revolutionized the field of natural language processing (NLP) since their introduction in 2018 (Radford et al. 2018; Devlin et al. 2019).”
BackgroundMethodLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Grice, H. P. (1975) · Speech Acts · doi:10.1163/9789004368811_003 · Cited by 23,221
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,887
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“increases, the number of pre-training tokens should increase roughly proportionally (Kaplan et al. 2020; Hoffmann et al. 2022).”
BackgroundOn the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Research works printed but not linked: 28
- Aher, 2022, Using large language models to simulate multiple humans, ArXiv
- Broyde, 2021, Build a medical sentence matching application using BERT and Amazon SageMaker, AWS Machine Learning Blog
- Jigsaw, 2017, Perspective API, Google Jigsaw
- Johnson, 2022, A.I. is mastering language. Should we trust what it says?, The New York Times
- Jones, 2022, Distributional semantics still can’t account for affordances, 482
- Kabbara, 2022, Investigating the performance of transformer-based NLI models on presuppositional inferences, 779
- Kalo, 2022, KAMEL: Knowledge analysis with multitoken entities in language models
- Kavumba, 2020, Balanced COPA: Countering superficial cues in causal reasoning, Association for Natural Language Processing, 1105
- Krawczyk, 2023, Bard is getting better at logic and reasoning, The Keyword: Google Blog
- Lakretz, 2022, Can transformers process recursive nested constructions, like humans?, 3226
- Lee, 2023, What are large language models used for?, NVIDIA Blog
- Li, 2022, Counterfactual reasoning: Do language models need world knowledge for causal inference?
- Lieber, 2021, Jurassic-1: Technical details and evaluation, White Paper. AI21 Labs
- Mahowald, 2022, Experimentally measuring the redundancy of grammatical cues in transitive clauses, ArXiv
- Nayak, 2019, Understanding searches better than ever before, The Keyword: Google Blog
- OpenAI, 2022, ChatGPT: Optimizing language models for dialogue, OpenAI Blog
- OpenAI, 2023, GPT-4 technical report, OpenAI
- OpenAI, 2023, Model index for researchers, OpenAI
- Radford, 2018, Improving language understanding by generative pre-training, OpenAI
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI
- Rae, 2021, Scaling language models: Methods, analysis & insights from training gopher, ArXiv
- Reif, 2019, Visualizing and measuring the geometry of BERT, 8594
- Shi, 2021, What Transformers might know about the physical world: T5 and the origins of knowledge, 2218
- Stevenson, 2022, Putting GPT-3’s creativity to the (alternative uses) test, 164
- Tejada, 2021, A study of BERT’s processing of negations to determine sentiment, 47
- Thewsey, 2021, Bring structure to diverse documents with Amazon Textract and transformer-based models on Amazon SageMaker, AWS Machine Learning Blog
- Trott, 2023, In cautious defense of LLM-ology, Blog Post
- Valmeekam, 2022, Large language models still can’t plan (a benchmark for LLMs on planning and reasoning about change)
Cited by
A survey on large language model based autonomous agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40231-1 · Cited by 1,941
“[14] also discusses the capabilities and limitations of LLMs in various downstream tasks.”
BackgroundA Survey on Large Language Model based Autonomous Agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.11432 · Cited by 1,925
“[14] also discusses the capabilities and limitations of LLMs in various downstream tasks.”
BackgroundSequence modeling and design from molecular to genome scale with Evo
Nguyen, E., Poli, M., Durrant, M. G., Kang, B., Katrekar, D., Li, D. B., Bartie, L. J., Thomas, A. W., King, S. H., Brixi, G., et al. (2024) · Science · doi:10.1126/science.ado9336 · Cited by 580
Yang, L., Chen, H., Li, Z., Ding, X., & Wu, X. (2023) · arXiv · doi:10.48550/arxiv.2306.11489 · Cited by 239
“Furthermore, they have limited capabilities in inferring relationships between actions and events [65].”
BackgroundYang, L., Chen, H., Li, Z., Ding, X., & Wu, X. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3360454 · Cited by 238
“Furthermore, they have limited capabilities in inferring relationships between actions and events [65].”
BackgroundBojić, L., Zagovora, O., Zelenkauskaitė, A., Vuković, V., Čabarkapa, M., Jerković, S. V., & Jovančević, A. (2025) · Scientific Reports · doi:10.1038/s41598-025-96508-3 · Cited by 173
“Chang & Bergen 28 investigated the use of GPT-3 for sentiment classification and found that it performed competitively with fine-tuned models on specific datasets.”
Chengke, W., Ding, W., Jin, Q., Jiang, J., Jiang, R., Xiao, Q., Liao, L., & Li, X. (2025) · Advanced Engineering Informatics · doi:10.1016/j.aei.2025.103158 · Cited by 120
Transitioning from MLOps to LLMOps: Navigating the Unique Challenges of Large Language Models
Pahune, S. A., & Akhtar, Z. (2025) · Information · doi:10.3390/info16020087 · Cited by 116
Generative design of bacteriophages with genome language models
King, S. B., Driscoll, C. L., Li, D. B., Guo, D., Merchant, A. T., Brixi, G., Wilkinson, M. E., & Hie, B. (2026) · Science · doi:10.1126/science.aec2657 · Cited by 120
People cannot distinguish GPT-4 from a human in a Turing test
Jones, C. R., Rathi, I., Taylor, S., & Bergen, B. K. (2025) · Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3715275.3732108 · Cited by 76
“Large Language Models like GPT-4 [ While these advances offer some potential advantages—allowing systems to play valuable societal roles which have until now been the sole preserve of people—they also bring attendant risks [11, 18, 23, 38].”
Self-citationPeople cannot distinguish GPT-4 from a human in a Turing test
Jones, C. R., & Bergen, B. K. (2024) · arXiv · doi:10.48550/arxiv.2405.08007 · Cited by 76
Self-citationHagendorff, T., Dasgupta, I., Binz, M., Chan, S. C. Y., Lampinen, A., Wang, J. X., Akata, Z., & Schulz, E. (2023) · arXiv · doi:10.48550/arxiv.2303.13988 · Cited by 68
Inductive reasoning in humans and large language models
Han, S. J., Ransom, K. J., Perfors, A., & Kemp, C. (2024) · Cognitive Systems Research · doi:10.1016/j.cogsys.2023.101155 · Cited by 50
Do LLMs write like humans? Variation in grammatical and rhetorical styles
Reinhart, A., Markey, B., Laudenbach, M., Pantusen, K., Yurko, R., Weinberg, G., & Brown, D. W. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2422455122 · Cited by 107
“Each human text was split into two chunks of roughly 500 words; the first chunk was used to prompt an LLM to create text that was compared to the second human chunk.”
Peykani, P., Ramezanlou, F., Tănăsescu, C. R., & Ghanidel, S. (2025) · Applied Sciences · doi:10.3390/app15148103 · Cited by 84
(Ir)rationality and cognitive biases in large language models
Macmillan-Scott, O., & Musolesi, M. (2024) · Royal Society Open Science · doi:10.1098/rsos.240255 · Cited by 47
“As we will see, these models still exhibit biases and produce information that is not factual [1].”
Can large language models help augment English psycholinguistic datasets?
Trott, S. (2024) · Behavior Research Methods · doi:10.3758/s13428-024-02337-z · Cited by 85
Several Categories of Large Language Models (LLMs): A Short Survey
Pahune, S. A., & Chandrasekharan, M. (2023) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2023.54677 · Cited by 58
“Transformer language models have received widespread public attention, yet their generated text is often surprising even to NLP researchers[4,7].”
BackgroundAchieving Peak Performance for Large Language Models: A Systematic Review
Rostam, Z. R. K., Szénási, S., & Kertész, G. (2024) · IEEE Access · doi:10.1109/access.2024.3424945 · Cited by 65
“• Transformer language models refer to those models that leverage the capabilities of a deep learning architecture called Transformer to process and understand human language [46], [47].”
BackgroundPrivacy preserving large language models: ChatGPT case study based vision and framework
Ullah, I., Hassan, N., Gill, S. S., Suleiman, B., Ahanger, T. A., Shah, Z., Qadir, J., & Kanhere, S. S. (2024) · IET Blockchain · doi:10.1049/blc2.12091 · Cited by 46
Andersson, M., & McIntyre, D. (2025) · Journal of Pragmatics · doi:10.1016/j.pragma.2025.02.001 · Cited by 45
Advances in Pre-trained Language Models for Domain-Specific Text Classification: A Systematic Review
Rostam, Z. R. K., & Kertész, G. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3763002 · Cited by 42
“This helps to identify the strengths and weaknesses of various models in specific contexts.”
Large Language Models in Urban Planning: A Systematic Review and Conceptual Framework
Liu, K., Yiğitcanlar, T., Mehmood, R., Corchado, J. M., & Fu, X. (2025) · Journal of Urban Technology · doi:10.1080/10630732.2025.2556551 · Cited by 27
Wang, S., Ouyang, Q., & Wang, B. (2024) · Research Square · doi:10.21203/rs.3.rs-3987793/v1 · Cited by 76
A benchmark for evaluating Arabic contextualized word embedding models
Elnagar, A., Yagi, S., Mansour, Y., Lulu, L., & Fareh, S. I. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103452 · Cited by 19
Equipping Llama with Google Query API for Improved Accuracy and Reduced HallucinationPreprint
Bae, Y. S., Kim, H. R., & Kim, J. H. (2024) · Research Square · doi:10.21203/rs.3.rs-4014474/v1 · Cited by 61
Tschisgale, P., Maus, H., Kieser, F., Kroehs, B., Petersen, S., & Wulff, P. (2025) · Physical Review Physics Education Research · doi:10.1103/6fmx-bsnl · Cited by 37
“…inference: Given an input prompt (typically a user-provided question or instruction), they generate text—one token 1 at a time—by sampling the next token from a probability distribution over the model’s vocabulary, conditioned on the input prompt and all previously generated tokens [53].”
Chatbots With Attitude: Enhancing Chatbot Interactions Through Dynamic Personality Infusion
Kovačević, N., Boschung, T., Holz, C., Groß, M. H., & Wampfler, R. (2024) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3640794.3665543 · Cited by 31
Fine-Tuning Large Language Models for Scientific Text Classification: A Comparative Study
Rostam, Z. R. K., & Kertész, G. (2024) · IEEE International Symposium on Logistics and Industrial Informatics (LINDI) · doi:10.1109/lindi63813.2024.10820432 · Cited by 36
Development of Linguistic Competence in English for Specific Purposes Through ChatGPT: A Case Study
Enamorado, M. Á. J. (2025) · Journal of language and Education · doi:10.17323/jle.2025.23745 · Cited by 21
iDesignGPT enhances conceptual design via large language model agentic workflows
Liu, S., Shen, Y., Zhang, Y., Hou, Z., Wang, X., Luo, J., & Zhang, Z. (2026) · Nature Communications · doi:10.1038/s41467-026-68672-1 · Cited by 30
Humans Learn Language from Situated Communicative Interactions. What about Machines?
Beuls, K., & Eecke, P. V. (2024) · Computational Linguistics · doi:10.1162/coli_a_00534 · Cited by 37
“While pragmatic inferences play a central role in human communication, they are notoriously difficult to draw for LLMs (Chang and Bergen 2024; Ruis et al. 2023).”
Holtzman, A., West, P., & Zettlemoyer, L. (2025) · Journal of Social Computing · doi:10.23919/jsc.2025.0009 · Cited by 38
Prompt-based fine-tuning with multilingual transformers for language-independent sentiment analysis
Ullah, F., Faizullah, S., Khan, I., Alghamdi, T. G., Syed, T. A., Alkhodre, A. B., Ayub, M. S., & Karim, A. (2025) · Scientific Reports · doi:10.1038/s41598-025-03559-7 · Cited by 12
“These models use a transformer architecture with self-attention mechanisms to allow them to capture sentence context in both directions 2,3,13,19,27 .”
The Factuality of Large Language Models in the Legal Domain
Hamdani, R. E., Bonald, T., Malliaros, F. D., Holzenberger, N., & Suchanek, F. M. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679961 · Cited by 25
“This natural language interface democratizes access to knowledge, especially in domains where users may not master querying structured KBs [18].”
Chen, J., Wu, M., Liu, Q., & Zhang, Y. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104336 · Cited by 19
Mine, Y., Iwamoto, Y., Okazaki, S., Nishimura, T., Tabata, E., Takeda, S., Peng, T., Nomura, R., Kakimoto, N., & MURAYAMA, T. (2025) · International Journal of Paediatric Dentistry · doi:10.1111/ipd.70029 · Cited by 12
“Furthermore, the low reading consistency, especially in Claude 3.5 Sonnet, indicates that LLMs generate outputs through probabilistic inference [26].”
InfluentialFrom Form(s) to Meaning: Probing the Semantic Depths of Language Models Using Multisense Consistency
Ohmer, X. I., Bruni, E., & Hupkes, D. (2024) · Computational Linguistics · doi:10.1162/coli_a_00529 · Cited by 36
“Chang and Bergen (2023) suggest that many weaknesses of LLMs, including form-dependencies, can be framed as under-and over-generalization errors.”
Open-Ethical AI: Advancements in Open-Source Human-Centric Neural Language Models
Sicari, S., Moreno, J. F. C., Rizzardi, A., & Coen‐Porisini, A. (2024) · ACM Computing Surveys · doi:10.1145/3703454 · Cited by 17
Do Multimodal Large Language Models and Humans Ground Language Similarly?
Jones, C. R., Bergen, B. K., & Trott, S. (2024) · Computational Linguistics · doi:10.1162/coli_a_00531 · Cited by 23
Self-citationLarge Language Models as Few-Shot Defect Detectors for Additive Manufacturing
Fang, Q., Xiong, G., Wang, F., Shen, Z., Dong, X., & Wang, F. (2024) · China Automation Congress (CAC) · doi:10.1109/cac63892.2024.10865554 · Cited by 10
“Furthermore, an LLM typically exhibits random behavior, i.e., its answers always varies even with the same input query [15].”
Jones, C. R., Trott, S., & Bergen, B. K. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00674 · Cited by 35
Self-citationWhat Drives Performance in Multilingual Language Models?
Nezhad, S. B., & Agrawal, A. (2024) · Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial) · doi:10.18653/v1/2024.vardial-1.2 · Cited by 26
“We systematically study the multilingual models under an important NLP task – text classification (Chang and Bergen, 2023 • Figure 4 in Appendix B illustrates the distribution of language families within the SIB-200 dataset.”
MethodUTILIZING RETRIEVAL-AUGMENTED GENERATION IN LARGE LANGUAGE MODELS TO ENHANCE INDONESIAN LANGUAGE NLP
Tohir, H., Merlina, N., & Haris, M. (2024) · JITK (Jurnal Ilmu Pengetahuan dan Teknologi Komputer) · doi:10.33480/jitk.v10i2.5916 · Cited by 11
Auxiliary task demands mask the capabilities of smaller language models
Hu, J., & Frank, M. C. (2024) · arXiv · doi:10.48550/arxiv.2404.02418 · Cited by 8
“Reasoning abilities have been of great recent interest in LM evaluation (e.g., Kojima et al., 2022; Webb et al., 2023; Chang & Bergen, 2024), and the test sets we used feature well-motivated answer option sets that enable forced choice evaluation.”
Using ChatGPT with Confidence for Biodiversity-Related Information Tasks
Elliott, M. J., & Fortes, J. A. B. (2023) · Biodiversity Information Science and Standards · doi:10.3897/biss.7.112926 · Cited by 7
WiseEDA: LLMs in RF Circuit Design
Jin, H., Wang, J., Sheng, J., Wu, Y., Chen, J., Wang, Y., & Liu, J. (2025) · Microelectronics Journal · doi:10.1016/j.mejo.2025.106607 · Cited by 11
Khademi, S., Palmer, C. J., Javed, M., Dimaguila, G. L., Clothier, H. J., Buttery, J. P., & Black, J. (2024) · JMIR AI · doi:10.2196/54449 · Cited by 7
“We chose transformers for the training data evaluation because with their capacity to take account of language structure, they were more sensitive to textual information compared to the other standard neural networks we were evaluating [42].”
MethodNeural Generative Models and the Parallel Architecture of Language: A Critical Review and Outlook
Rambelli, G., Chersoni, E., Testa, D., Blache, P., & Lenci, A. (2024) · Topics in Cognitive Science · doi:10.1111/tops.12733 · Cited by 8
“…different from the one that foundation models seem to be able to acquire from linguistic data and represent in their continuous embeddings, as shown by their differences from human semantic processing (Chang & Bergen, 2024; Kauf et al., 2024; Kauf et al., 2023; Pedinotti et al., 2021).”
Evaluating Company-specific Biases in Financial Sentiment Analysis using Large Language Models
Nakagawa, K., Hirano, M., & Fujimoto, Y. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10826008 · Cited by 15
“This common knowledge, which includes prior knowledge about named entities, is heavily reliant on the training data [34].”
BackgroundComparison of Large Language Model with Aphasia
Watanabe, T., Inoue, K., Kuniyoshi, Y., Nakajima, K., & Aihara, K. (2025) · Advanced Science · doi:10.1002/advs.202414016 · Cited by 6
Opinion paper: generative AI and the future of scientometrics
Lepori, B., Andersen, J. P., & Donnay, K. (2026) · Scientometrics · doi:10.1007/s11192-026-05667-1 · Cited by 9
“…time: regularities in the structure of token sequences corresponds to language syntax whereas patterns of association between token sequences enable semantic interpretation (Chang & Bergen, 2024) and pragmatics are captured through long-range dependencies between token sequences (Ma et al., 2025).”
Lehtonen, E., Gröndahl, T., & Nordhoff, S. (2025) · IEEE Access · doi:10.1109/access.2025.3546565 · Cited by 5
“It thus seems that understanding negation does not automatically emerge from the pre-training of LLMs, in contrast to many other linguistic capacities [32].”
Shades of zero: Distinguishing impossibility from inconceivability
Hu, J., Sosa, F., & Ullman, T. D. (2025) · Journal of Memory and Language · doi:10.1016/j.jml.2025.104640 · Cited by 11
Collaborative Growth: When Large Language Models Meet Sociolinguistics
Nguyen, D. (2025) · Language and Linguistics Compass · doi:10.1111/lnc3.70010 · Cited by 17
Mapping the learning curves of deep learning networks
Jiang, Y., & Dale, R. A. C. (2025) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1012286 · Cited by 5
Natural Language Processing RELIES on Linguistics
Opitz, J., Wein, S., & Schneider, N. (2025) · Computational Linguistics · doi:10.1162/coli_a_00560 · Cited by 30
“Explanations and binding can also adopt an outside, behavioral view (Beguš et al., 2023; Behzad et al., 2023; Chang and Bergen, 2024).”
Lodoen, S., & Orchard, A. (2025) · IEEE International Symposium on Ethics in Engineering, Science, and Technology (ETHICS) · doi:10.1109/ethics65148.2025.11098367 · Cited by 6
“If procedures reveal “the logics that structure behavior” in “cultural, social, and historical systems” [14, pp. 7–8], what does this shift indicate about the role of LLMs in the production and consumption of information?”
Nair, A. R., Gupta, D., Paul, B., & Sivabhavani, J. (2025) · IEEE Access · doi:10.1109/access.2025.3648480 · Cited by 4
“The rapid progress in NLP has been significantly driven by the emergence of LLMs, which have reshaped various applications, particularly MT [38].”
Exemplification in ChatGPT and student argumentative writing: A local grammar analysis
Jiang, F. K., & Su, H. (2026) · Linguistics and Education · doi:10.1016/j.linged.2026.101533 · Cited by 4
Shao, Z., Gao, F., Zhu, Z., Luo, C., Xing, H., Yu, Z., Zheng, Q., Yan, M., & Bu, J. (2025) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2025.emnlp-main.1574 · Cited by 8
Hong, X., Ryzhova, M., Biondi, D. A., & Demberg, V. (2024) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2024.starsem-1.34 · Cited by 7
Large Language Models in Psycholinguistic Studies
Günther, F., & Cassani, G. (2025) · Elsevier eBooks · doi:10.1016/b978-0-323-95504-1.00749-3 · Cited by 6
Evidence Against Syntactic Encapsulation in Large Language Models
McGee, T. A., Zhang, Y., & Blank, I. (2026) · Cognitive Science · doi:10.1111/cogs.70187 · Cited by 4
“Whether LLMs have conceptual understanding and “common sense” knowledge is an ongoing debate (e.g., Chang & Bergen, 2024; Mahowald et al., 2024; Pavlick, 2022).”
Trusting the black box: Adapting a multidimensional measure of trust in generative AI
Alon, L., & Levkovich, I. (2026) · Computers in Human Behavior Artificial Humans · doi:10.1016/j.chbah.2026.100295 · Cited by 9
Jin, C. (., Kim, J., & Im, J. (2026) · Tourism Management · doi:10.1016/j.tourman.2026.105468 · Cited by 3
Shang, Y., Pallarès-Barberà, M., & Romagosa, F. (2024) · Current Issues in Tourism · doi:10.1080/13683500.2024.2416024 · Cited by 6
“This may involve leveraging recent advancements in natural language processing, such as BERT or GPT models, which have shown promise in understanding the nuances of language (Chang & Bergen, 2024).”
Developing Multi-Agent LLM Applications Through Continuous Human-LLM Co-Programming
Song, H., Goknil, A., Jiang, X., Melum, E., Joe, H., Gazzotti, C., Frascolla, V., Videsjorden, A. N., & Nguyen, P. (2025) · 2025 IEEE/ACM 4th International Conference on AI Engineering – Software Engineering for AI (CAIN) · doi:10.1109/cain66642.2025.00013 · Cited by 3
Selecting Shots for Demographic Fairness in Few-Shot Learning with Large Language Models
Aguirre, C. A., Sasse, K., Cachola, I. A., & Dredze, M. H. (2024) · Workshop on NLP for Positive Impact (NLP4PI) · doi:10.18653/v1/2024.nlp4pi-1.4 · Cited by 11
Goto, T., Ono, K., & Morita, A. (2024) · doi:10.36227/techrxiv.170975411.12532297/v1 · Cited by 3
Apidianaki, M., Fourtassi, A., & Padó, S. (2024) · Computational Linguistics · doi:10.1162/coli_e_00539 · Cited by 5
Sung, H., & Shin, G. (2025) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3767330 · Cited by 3
SGNet: Sequence Grouping Network via Vision-Language Model for Text-Guided Video Summarization
Yao, J., Zhang, J., & Li, Z. (2025) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2025.3626259 · Cited by 2
LI, C., & Yuan, X. (2026) · Journal of Pragmatics · doi:10.1016/j.pragma.2026.01.008 · Cited by 3
Can ChatGPT reciprocate impoliteness? The AI moral dilemma
Tantucci, V., & Culpeper, J. (2026) · Journal of Pragmatics · doi:10.1016/j.pragma.2026.03.008 · Cited by 5
Papakostas, C. (2026) · Teaching Theology & Religion · doi:10.1111/teth.70034 · Cited by 4
Do Language Models Know Who Did What to Whom?
Denning, J. M., Guo, X., Snefjella, B., & Blank, I. (2026) · Open Mind · doi:10.1162/opmi.a.365 · Cited by 4
Sieker, J., & Zarrieß, S. (2023) · Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/2023.blackboxnlp-1.14 · Cited by 2
“…in evaluating whether language models adhere to the MP! principle, and fine-tuned models do not necessarily reflect the linguistic properties of language models in general (cf. Ettinger (2020) or Chang and Bergen (2023)), we center our investigation on the predictions of these base models.”
BackgroundModelling child comprehension: A case of suffixal passive construction in Korean
Shin, G., & Mun, S. (2024) · Computer Speech & Language · doi:10.1016/j.csl.2024.101701 · Cited by 4
Introduction: Explainability, AI literacy, and language development
Shin, G.-H., & Parde, N. (2024) · Computer Speech & Language · doi:10.1016/j.csl.2024.101766 · Cited by 2
Linguistic traces of stochastic empathy in language models
Kleinberg, B., Zegers, J., Festor, J., Vida, S., Präsent, J., Loconte, R., & Peereboom, S. (2024) · arXiv · doi:10.48550/arxiv.2410.01675 · Cited by 1
“Large language models (LLMs) have been among the landmark technical achievements of the past decade [1–3].”
A distinctive meaning makes a sentence memorable
Clark, T. H., Tuckute, G., Medina, B., & Fedorenko, E. G. (2025) · Journal of Memory and Language · doi:10.1016/j.jml.2025.104700 · Cited by 3
Foundation language models through the lens of manufacturing
Aghaei, S., & Ansari, F. (2026) · Production & Manufacturing Research · doi:10.1080/21693277.2026.2632468 · Cited by 3
AeroTag-RAG: A structured intelligent question-answering system for civil aviation services
Zhao, H., (50563), L. Z., & Deng, W. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.133158 · Cited by 1
Economidou‐Kogetsidis, M., & Lai, S. (2026) · Journal of Pragmatics · doi:10.1016/j.pragma.2026.05.009 · Cited by 2
Generalization of large models: Methods, limitations and applications
Bu, K., Liu, Y., Wang, W., & Cao, Z. (2026) · Applied Soft Computing · doi:10.1016/j.asoc.2026.115864 · Cited by 1
Synthesizing Public Opinions with LLMs: Role Creation, Impacts, and the Future of Edemorcacy
Karanjai, R., Shor, B., Austin, A., Kennedy, R., Lu, Y., Xu, L., & Shi, W. L. (2025) · International Conference on eDemocracy & eGovernment · doi:10.1109/icedeg65568.2025.11081685 · Cited by 3
“When applying linguistic rules and world knowledge, LLMs can be influenced by specific examples and phrasing, leading to response variations based on subtle changes in prompt formulation [29].”
InfluentialSnakes and ladders in the evolution of language
Fields, C., & Levin, M. G. (2024) · Physics of Life Reviews · doi:10.1016/j.plrev.2024.04.007 · Cited by 1
“The impressive abilities of large language models (LLMs; [20]), which are statistical models of samples of human language use, pose an additional challenge to the idea of a single, innate universal grammar, and hence to the idea of a single origin of language.”
BackgroundStudies in conversational AI : multilingual capabilities, world knowledge, and evaluation strategies
Bruyn, M. D. (2024) · doi:10.63028/10067/2023030151162165141 · Cited by 1
Artificial Intelligence and Language
Lenci, A., & Vestrucci, A. (2024) · Elsevier eBooks · doi:10.1016/b978-0-323-95504-1.00241-6 · Cited by 1
Braberman, V. A., Bonomo, F., Charalambous, Y., Colonna, J. G., Cordeiro, L. C., & de Freitas, R. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5624902 · Cited by 8
The Synergistic Integration of Access Control Management and Large Language Model Agents: A SurveyPreprint
Periti, F., & Saha, S. (2026) · doi:10.36227/techrxiv.177075184.43798334/v1 · Cited by 1
Robustness of the random language model
Lalegani, F., & DeGiuli, E. (2024) · Physical review. E · doi:10.1103/physreve.109.054313 · Cited by 5
An abstract linguistic space for transmitting information from one mind to another
Tuckute, G., & Fedorenko, E. G. (2024) · Neuron · doi:10.1016/j.neuron.2024.09.005
Fodor, J., Murawski, C., & Suzuki, S. (2025) · eLife · doi:10.7554/elife.108442.1 · Cited by 3
Fodor, J., Murawski, C., & Suzuki, S. (2026) · eLife · doi:10.7554/elife.108442 · Cited by 1
Category-Theoretic Approach to Distributed Information Flow
Fields, C., & Glazebrook, J. F. (2025) · Synthesis lectures on engineering, science, and technology · doi:10.1007/978-3-031-97263-8_3
Expertise and Information Seeking in the Age of Generative AI: New Procedures, New Problematics
Lodoen, S., & Orchard, A. (2026) · IEEE Transactions on Technology and Society · doi:10.1109/tts.2026.3690975
Bateman, J. A. (2025) · Elsevier eBooks · doi:10.1016/b978-0-323-95504-1.01235-7
Metadata
- Authors
- Chang, T. A., & Bergen, B. K.
- Affiliations
- University of California San Diego (US)
- ORCID
- Bergen, Benjamin K.
- Venue
- Computational Linguistics
- Publisher
- MIT Press
- Published
- 2024
- Volume
- 50
- Issue
- 1
- Pages
- 293-350
- ISSN
- 0891-2017 (ISSN-L)
- 1530-9312
- OpenAlex
- W4388691863
- Cited by
- 105
- References
- 379
- Citations by year
- 2026: 33 · 2025: 37 · 2024: 30
- FWCI
- 9.89
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Computational Linguistics (publishedVersion, cc-by-nc-nd)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- Medical Library Association
- University of California, San Diego
- Harvard Data Science Initiative, Harvard University
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- large language models, transformer models, syntax, semantics, pragmatics, world knowledge, reasoning, factual errors, memorization, social bias, overgeneralization, undergeneralization, model scaling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.