Language Model Behavior: A Comprehensive Survey
Open accessThe DOI matches its Crossref record; 428 of the 428 references it cites were found in a registry or an index.
Abstract
Transformer language models have received widespread public attention, yet their generated text is often surprising even to NLP researchers. In this survey, we discuss over 250 recent studies of English language model behavior before task-specific fine-tuning. Language models possess basic capabilities in syntax, semantics, pragmatics, world knowledge, and reasoning, but these capabilities are sensitive to specific inputs and surface features. Despite dramatic increases in generated text quality as models scale to hundreds of billions of parameters, the models are still prone to unfactual responses, commonsense errors, memorized text, and social biases. Many of these weaknesses can be framed as over-generalizations or under-generalizations of learned patterns in text. We synthesize recent results to highlight what is currently known about large language model capabilities, thus providing a resource for applied work and for research in adjacent fields that use language models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“So far, quantitative comparisons have looked mainly at basic grammar and syntax (5) or features such as word choice, punctuation, sentence length, and so on, finding evidence of some differences between human- and LLM-written text (10–14).”
From Do LLMs write like humans? Variation in grammatical and rhetorical styles (2025) · cites this paper as [5]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 3×
“Transformer language models have revolutionized the field of natural language processing (NLP) since their introduction in 2018 (Radford et al. 2018; Devlin et al. 2019).”
BackgroundMethodLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Grice, H. P. (1975) · Speech Acts · doi:10.1163/9789004368811_003 · Cited by 23,221
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,281
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,887
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“increases, the number of pre-training tokens should increase roughly proportionally (Kaplan et al. 2020; Hoffmann et al. 2022).”
BackgroundOn the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Research works printed but not linked: 28
- Aher, 2022, Using large language models to simulate multiple humans, ArXiv
- Broyde, 2021, Build a medical sentence matching application using BERT and Amazon SageMaker, AWS Machine Learning Blog
- Jigsaw, 2017, Perspective API, Google Jigsaw
- Johnson, 2022, A.I. is mastering language. Should we trust what it says?, The New York Times
- Jones, 2022, Distributional semantics still can’t account for affordances, 482
- Kabbara, 2022, Investigating the performance of transformer-based NLI models on presuppositional inferences, 779
- Kalo, 2022, KAMEL: Knowledge analysis with multitoken entities in language models
- Kavumba, 2020, Balanced COPA: Countering superficial cues in causal reasoning, Association for Natural Language Processing, 1105
- Krawczyk, 2023, Bard is getting better at logic and reasoning, The Keyword: Google Blog
- Lakretz, 2022, Can transformers process recursive nested constructions, like humans?, 3226
- Lee, 2023, What are large language models used for?, NVIDIA Blog
- Li, 2022, Counterfactual reasoning: Do language models need world knowledge for causal inference?
- Lieber, 2021, Jurassic-1: Technical details and evaluation, White Paper. AI21 Labs
- Mahowald, 2022, Experimentally measuring the redundancy of grammatical cues in transitive clauses, ArXiv
- Nayak, 2019, Understanding searches better than ever before, The Keyword: Google Blog
- OpenAI, 2022, ChatGPT: Optimizing language models for dialogue, OpenAI Blog
- OpenAI, 2023, GPT-4 technical report, OpenAI
- OpenAI, 2023, Model index for researchers, OpenAI
- Radford, 2018, Improving language understanding by generative pre-training, OpenAI
- Radford, 2019, Language models are unsupervised multitask learners, OpenAI
- Rae, 2021, Scaling language models: Methods, analysis & insights from training gopher, ArXiv
- Reif, 2019, Visualizing and measuring the geometry of BERT, 8594
- Shi, 2021, What Transformers might know about the physical world: T5 and the origins of knowledge, 2218
- Stevenson, 2022, Putting GPT-3’s creativity to the (alternative uses) test, 164
- Tejada, 2021, A study of BERT’s processing of negations to determine sentiment, 47
- Thewsey, 2021, Bring structure to diverse documents with Amazon Textract and transformer-based models on Amazon SageMaker, AWS Machine Learning Blog
- Trott, 2023, In cautious defense of LLM-ology, Blog Post
- Valmeekam, 2022, Large language models still can’t plan (a benchmark for LLMs on planning and reasoning about change)
Cited by
Items 101–136 of 136
Väkevä, L., & Partti, H. (2025) · Springer eBooks · doi:10.1007/978-3-031-99804-1_9 · Cited by 4
Three Tiers of Computation in Transformers and in Brain Architectures
Graham, E., & Granger, R. (2026) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3844612 · Cited by 2
Injection of Linguistic Knowledge into Transformer-Based Models for Cross-Lingual Transfer
López-Otal, M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-99554-5_38
Huang, S., Wang, J., Deng, H., & Liu, X. (2026) · Communications in computer and information science · doi:10.1007/978-981-95-3483-8_27
EFL Students’ Bilingual Community: A Sociolinguistic Study of Language Choice and Its Implications
Susilawati, E., & Bunau, E. (2025) · Indonesian Journal of EFL and Linguistics · doi:10.21462/ijefl.v10i2.1014
“…of EESP were not only competent to communicate with more than one language, but they also shared the same knowledge of appropriateness, occurrence, and feasibility as controls to their social behavior, including their language behavior in communication (Chang & Bergen, 2024; Kuznietsova, 2024).”
Evidence for Daily and Weekly Periodic Variability in GPT-4o PerformancePreprint
Tschisgale, P., & Wulff, P. (2026) · Research Square · doi:10.21203/rs.3.rs-8869261/v1
Providers of relief in distress: RAG-based LLMs as situation and intent-aware assistants
Nazar, A. M., Norman, B., Northway, H., Toutoungi, A., Zatkalik, E., Carlson, G., Sabado, E., Shawa, H., & Selim, M. Y. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1712596
“LLMs enable open-domain, conversational interaction grounded in context and semantics (Chang and Bergen, 2024; Celik and Eltawil, 2024).”
Structural priming in humans and large language models
Sinclair, A., Klimovich‐Gray, A., Jumelet, J., Adamian, N., & Konopka, A. E. (2026) · Journal of Memory and Language · doi:10.1016/j.jml.2025.104713 · Cited by 3
Beyond Verdicts: Evaluating Language Model Moral Competence
Snoswell, A. J., Kilov, D., & Lazar, S. (2026) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v40i44.41131
産業技術総合研究所 (2025) · Institutional Repositories DataBase (IRDB) · doi:10.50886/0002003354
Aligned Probing: Relating Toxic Behavior and Model Internals
Waldis, A., Gautam, V., Lauscher, A., Klakow, D., & Gurevych, I. (2026) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl.a.613 · Cited by 5
Domain Adaptation of Named Entity Recognition for Plant Health Monitoring
Borovikova, M. (2024) · theses.fr (ABES) · doi:10.70675/aa08ea7az594ez48adz814bz17ff1279b349
Huang, J., Xiao, Y., Wu, L., Cheng, X., Zheng, W., Hao, Q., He, M., & Zhao, H. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792261
“For instance, for the same task, using different foundation models can lead to significant performance gaps [2].”
Минимизация ошибок нейросетевых моделей в библиотечно-издательской сфере: подходы и их эффективность
Рзянкин, И. С. (2025) · Sovremennye informacionnye tehnologii i IT-obrazovanie. · doi:10.25559/sitito.021.202502.272-286
Reasoning in Large Language Models Across Multilingual, Long-Context, and Symbolic Settings
Nezhad, S. B. (2026) · PDXScholar (Portland State University) · doi:10.15760/etd.4098
Rote Memorization or Intelligence: An Assessment of Inferential Reasoning in Large Language Models
Mehmood, R., Rehman, E., & Habib, M. (2026) · International Journal of Computational Intelligence Systems · doi:10.1007/s44196-026-01262-7 · Cited by 1
“Developed primarily on transformer architectures and internet-scale datasets [ 17 , 24 ], these models have followed a scaling-centric trajectory [ 33 ] to achieve emergent capabilities [ 9 ].”
Acquisition of syntactic knowledge by monolingual Korean children
Shin, G. (2026) · Trends in language acquisition research · doi:10.1075/tilar.36.05shi
Sieker, J., & Zarrieß, S. (2026) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2026.findings-acl.372 · Cited by 4
Özmen, H. B., & Senel, F. A. (2026) · International Journal of Artificial Intelligence Tools · doi:10.1142/s021821302650017x
Khieovongphachanh, V. (2026) · Vālasān vithanyāsāt mahāvithanyālai Suphānuvong. · doi:10.69692/sujmrd120350
Nair, A. R., & Gupta, D. (2026) · Procedia Computer Science · doi:10.1016/j.procs.2026.06.107
Syahidi, A. A., Nuchitprasitchai, S., & Viriyapant, K. (2026) · International Joint Conference on Computer Science and Software Engineering · doi:10.1109/jcsse68839.2026.11596890
“Autoregressive LLMs without grounding [15] Prone to factual inconsistency and unsupported outputs.”
The End of Theory in Social Science? AI and Sleepwalking into Epistemic SurrenderPreprint
Smith, A. N., & Humphries, C. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6807338
Shin, G., Kwon, N., & Mun, S. (2026) · Humanities and Social Sciences Communications · doi:10.1057/s41599-026-08299-1
Curfman, E. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7237410
Liu, M., Zheng, D., Su, J., & Wu, C. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817484
“(99) Conv3d Min Softmax: the known variables are ’A’ with the dtype torch.float16 and shape [41, 5, 5, 3, 19], ’G’ with the dtype torch.float16 and shape [36, 5, 3, 3, 3], and ’F’ with the dtype torch.float16 and shape [41, 3, 1, 17].”
Brain metaphors generated by humans and large language models: Evidence from metaphor elicitation
Urbonaitė, J., & Shcherbino, Y. (2026) · Studies About Languages · doi:10.5755/j01.sal.48.1.43076
Inference Under Opposition: Counterfactual Evaluation for Answer Stability in Large Language Models
Akhand, M. N. T., Akbar, K. A., & Thuraisingham, B. M. (2026) · IEEE International Conference on Information Reuse and Integration (IRI) · doi:10.1109/iri69576.2026.00059
“Behavioral aspects of LLMs have been widely studied [1].”
Structural and Linguistic Markers for Distinguishing Human Manuscripts from AI-Generated Summaries
Pichappan, P. (2026) · Journal of Digital Information Management · doi:10.6025/jdim/2026/24/3/127-148
Fodor, J., Murawski, C., & Suzuki, S. (2026) · eLife · doi:10.7554/elife.108442.3 · Cited by 1
Fodor, J., Murawski, C., & Suzuki, S. (2026) · eLife · doi:10.7554/elife.108442.3.sa0
Knauff, M., Butz, M. V., Kaup, B., Kunde, W., & Scharlau, I. (2026) · Psychologische Rundschau · doi:10.1026/0033-3042/a000781
He, Z., Wang, Y., & Shi, H. (2026) · Discover Computing · doi:10.1007/s10791-026-10687-9
Cyberlogic-enabled Thinking Space: a closed-loop human-cyber cognitive architecture in CPST
Chen, R., Zhou, L., Ding, J., & Ning, H. (2026) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2026.1899109
Chen, W., Peng, W., Cai, W., Li, M., & Cao, X. (2026) · Applied Geography · doi:10.1016/j.apgeog.2026.104248
Why AI cannot ‘think’ like humans do
Gonzalez, D., & de Chantal, P. (2026) · The Conversation · doi:10.64628/aam.anm3nfrn3
Metadata
- Authors
- Chang, T. A., & Bergen, B. K.
- Affiliations
- University of California San Diego (US)
- ORCID
- Bergen, Benjamin K.
- Venue
- Computational Linguistics
- Publisher
- MIT Press
- Published
- 2024
- Volume
- 50
- Issue
- 1
- Pages
- 293-350
- ISSN
- 0891-2017 (ISSN-L)
- 1530-9312
- OpenAlex
- W4388691863
- Cited by
- 105
- References
- 379
- Citations by year
- 2026: 33 · 2025: 37 · 2024: 30
- FWCI
- 9.89
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Computational Linguistics (publishedVersion, cc-by-nc-nd)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- Medical Library Association
- University of California, San Diego
- Harvard Data Science Initiative, Harvard University
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- large language models, transformer models, syntax, semantics, pragmatics, world knowledge, reasoning, factual errors, memorization, social bias, overgeneralization, undergeneralization, model scaling
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.