A Survey of Large Language Models
Open accessPublished version of a preprint
The DOI matches its Crossref record; 439 of the 583 references it cites were found in a registry or an index, and 128 are not checked yet; 236 of the 360 sources its claims cite were checked for support.
Abstract
The rapid evolution of large language models (LLMs) has driven a transformative shift in artificial intelligence (AI), reshaping both research paradigms and practical applications. Distinguished from their predecessors by unprecedented scale and advanced capabilities, LLMs necessitate new frameworks for understanding their development, behavior, and societal impact. This survey systematically reviews recent advancements in LLM techniques across four key dimensions: (1) pre-training methodologies, which establish core model capabilities through large-scale self-supervised training, architectural innovations, and data curation strategies; (2) post-training techniques, including supervised fine-tuning and reinforcement learning, which adapt foundational models to downstream tasks and enhance their alignment and safety; (3) utilization strategies, such as in-context learning, prompt engineering, and agentic reasoning, that optimize real-world deployment and enable effective interaction with external environments; and (4) evaluation methods, encompassing benchmarks for key ability dimensions such as core language capabilities, reasoning, and safety, which support comprehensive and reliable assessment of model performance. Additionally, we identify critical research issues, including those concerning theoretical foundations, efficient scaling, alignment, and agentic capability, and highlight the open challenges they present. By synthesizing state-of-the-art insights and emerging trends, this survey aims to provide a systematic and comprehensive framework for understanding the trajectory, current limitations, and future directions of LLM progress.
Verified 2026-11-19 · 128 references not checked yet
Overview
CoChat Summary
TL;DR
This survey systematically reviews recent LLM advances across pre-training, post-training, utilization, and evaluation, and identifies open challenges in theory, scaling, alignment, and agentic capability. It synthesizes techniques and representative model/training details rather than presenting new experiments.
Key findings
- The survey organizes LLM development into four interconnected dimensions: pre-training (data, architecture, training, mid-training), post-training (SFT and RL), utilization (prompting, inference-time techniques, agents, efficiency), and evaluation (benchmarks and approaches).
- Pre-training examples include GPT-3 (175B parameters), PaLM (540B), LLaMA-3.1 (405B, 15T tokens), DeepSeek-V3 (671B MoE, 14.8T tokens), Qwen3 (235B MoE, 36T tokens), and Kimi K2 (1T MoE, 15.5T tokens); LLaMA-3.1 allocates 25% of tokens to mathematical/reasoning tasks and 17% to code.
- Post-training instruction mixes typically cover mathematical reasoning (20%–30%), coding (20%–30%), agentic tasks (10%–20%), general tasks (20%–40%), and a minor proportion for AI identity/safety; open-source LLMs often use about one million instruction instances (e.g., DeepSeek-V3 1.5M, Qwen2.5 1M, Tulu-v3 939K).
- Evaluation and capability trends include DeepSeek-R1 achieving over 90% accuracy on MMLU, while the paper notes benchmark saturation and open issues in fairness, practical utility, and superintelligence assessment.
Methods
- Design: a systematic literature survey, described as an updated and refined version of an earlier arXiv article and expanded into a book, covering four critical stages of the LLM lifecycle.
- Data/sample: published research literature, technical reports, benchmarks, and model documentation; no primary data collection or new experiments are reported.
- Main analysis: qualitative synthesis with comparative tables, including detailed formulations for network configurations, training details of representative LLMs, and a comparison of post-training methods; the authors also maintain a curated GitHub repository of LLM resources.
Limitations
- The paper does not state limitations of the survey itself; it identifies open research challenges including theoretical foundations, architecture innovations, scaling enhancement, data barrier, training efficacy, alignment and safety, capacity assessment, agentic AI, and sustainability/responsibility.
- Evident from the text: as a survey, it reports no new experiments or empirical validation and relies on cited literature and technical reports.
- Evident from the text: no systematic search or inclusion protocol is described, and coverage is selective and likely to date quickly in a fast-moving field.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures



























Verification
Claims the paper makes
178 Valid · 15 Not valid · 50 Not checked · 70 Own results and interpretations
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“As a result, most models, such as MiniMax-M1 [94], Qwen3-Next [95], and Kimi Linear [92], mix linear attention with full attention at a fixed ratio, typically 3:1.”
What the sources say
- Disputedcites [94] Li A, Gong B, Yang B, Shan B, Liu C, et al. Minimax-01: scaling foundation models with lightning attention. 2025, arXiv preprint arXiv: 2501.08313 sourceExperimental — accuracy not yet measured
The Model Architecture section describes MiniMax-Text-01 as mixing linear and softmax attention at a fixed ratio, but specifies seven linear-attention blocks for every one softmax-attention block—not 3:1. The paper does not establish what ratio most models use. (LLM, conf=0.90)
Checked by a reasoning model — may be wrong.
- Not matchedcites [95] Team Q. Qwen3-Next: towards ultimate training & inference efficiency. See qwen.ai/blog?id=4074cca80393150c248e50 8aa62983f9 cb7d27cd website, 2025
We searched by its title and found no matching paper.
- Partly supportedcites [92] Team K, Zhang Y, Lin Z, Yao X, Hu J, et al. Kimi linear: an expressive, efficient attention architecture. 2025, arXiv preprint arXiv: 2510.26692 sourceExperimental — accuracy not yet measured
Section 7.2, “Hybrid Model,” says Kimi Linear interleaves linear and full-attention layers at a fixed 3:1 ratio and describes predefined-ratio inter-layer hybrids as widely adopted. The paper does not establish that 3:1 is typical of most models or that the other named models use that ratio. No page number is available in our copy. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Large language models (LLMs) [13] achieve substantially enhanced capabilities by scaling up model size, training data, and computational resources in accordance with established scaling laws [14].”
What the sources say
- Not matchedcites [13] Brown T B, Mann B, Ryder N, Subbiah M, Kaplan J, et al. Language models are few-shot learners. In: Proceedings of the 34th International Conference on Neural Information Processing Systems. 2020, 159
We searched by its title and found no matching paper.
- Partly supportedcites [14] Kaplan J, McCandlish S, Henighan T, Brown T B, Chess B, Child R, Gray S, Radford A, Wu J, Amodei D. Scaling laws for neural language models. 2020, arXiv preprint arXiv: 2001.08361Experimental — accuracy not yet measured
The Summary (p. 2) reports that language-model performance improves as model size, dataset size, and training compute increase, following empirical power laws. But the paper measures performance primarily by cross-entropy loss; the Discussion (p. 19) says it remains important to test whether lower loss translates into improvement on language tasks. Thus the broader claim of substantially enhanced capabilities goes beyond what the paper establishes. (LLM, conf=0.85)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Empirical evidence demonstrates that this annealing approach can substantially improve model evaluation performance [39,41,52].”
What the sources say
- Supportedcites [39] Grattafiori A, Dubey A, Jauhri A, Pandey A, Kadian A, et al. The llama 3 herd of models. 2024, arXiv preprint arXiv: 2407.21783 source
Section 3.1.3 reports that annealing with high-quality code and mathematical data improved the Llama 3 8B model’s performance on two validation benchmarks. Section 3.4.3 places the adjusted data mix in the final stage of pre-training. The reported gains were negligible for the 405B model, so the evidence supports that this approach can improve performance, not that it does so for every model. (LLM, conf=0.93)
- Not matchedcites [41] Hu Y, Song H, Chen J, Deng J, Wang J, Zhou K, Zhu Y, Jiang J, Dong Z, Lu Y, Miao X, Zhao W X, Wen J R. YuLan-Mini: Pushing the limits of open data-efficient language model. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025, 5374-5400
We searched by its title and found no matching paper.
- Supportedcites [52] Hu S, Tu Y, Han X, He C, Cui G, et al. MiniCPM: Unveiling the potential of small language models with scalable training strategies. 2024, arXiv preprint arXiv: 2404.06395 source
Section 5 (pp. 9–10) describes adding high-quality data during the annealing or decay stage and reports that this improves results compared with adding it only during supervised fine-tuning. Table 1 (p. 10) shows substantial gains on several evaluation benchmarks for the 2.4B model when that data is included during decay. (LLM, conf=0.93)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Initial scaling laws [14] favored allocating more resources toward increasing model size over data size.”
What the sources say
- Supportedcites [14] Kaplan J, McCandlish S, Henighan T, Brown T B, Chess B, Child R, Gray S, Radford A, Wu J, Amodei D. Scaling laws for neural language models. 2020, arXiv preprint arXiv: 2001.08361
The Summary (p. 4, Figure 3) says most additional compute in optimally compute-efficient training should go toward increased model size, with a relatively small increase in data. The Summary of Scaling Laws (p. 5) likewise says an increased compute budget should be spent primarily on larger models, without dramatic increases in dataset size. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The combination of pre-training (optionally enhanced by mid-training) and post-training has established the prevailing methodology for developing capable and aligned LLMs [26,27].”
What the sources say
- Partly supportedcites [26] Yang A, Yang B, Zhang B, Hui B, Zheng B, et al. Qwen2.5 technical report. 2024, arXiv preprint arXiv: 2412.15115 sourceExperimental — accuracy not yet measured
The Abstract and sections 3–4 describe Qwen2.5’s pre-training and post-training stages and link them to improved capabilities and human preference. The report does not establish that this combination is the prevailing methodology across LLM development, and the passages found do not describe optional mid-training. (LLM, conf=0.84)
- Partly supportedcites [27] Lacroix T, Rozière B, Goyal N, Hambro E, Azhar F, Rodriguez A, Joulin A, Grave E, Lample G. LLaMA: Open and efficient foundation language models. 2023, arXiv preprint arXiv: 2302.13971 sourceExperimental — accuracy not yet measured
The Approach section (p. 2) describes training on large amounts of text, and Instruction Finetuning (p. 7) reports that subsequent finetuning improved LLaMA’s MMLU performance and instruction-following. These passages support a pre-training–then-finetuning sequence, but do not establish it as the prevailing methodology or address optional mid-training or alignment. (LLM, conf=0.85)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
117 more claims · 77 with their cited sources checked
Create a free account to see all 313 claims and to verify the 19 references still open.
- It may state two separate claims as one. 96
- Probably not a substantial claim. 132
- The quote only partly states this claim. 1
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 6×
“Although scaling is mainly conducted in model size (with similar architectures and pre-training tasks), these large-sized PLMs display different behaviors from smaller PLMs (e.g., 330M-parameter BERT and 1.5Bparameter GPT-2) and show surprising abilities (called emergent abilities [31]) in solving a series of complex…”
InfluentialPyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473 · cited 9×
“• DeepSpeed [64] is a PyTorch-based deep learning optimization library developed by Microsoft, which has been used to train a number of LLMs, such as GPT-Neo [115] and BLOOM [66].”
InfluentialEfficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 1×
“Further, word2vec [19, 20] was proposed to build a simplified shal-”
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1×
“Further, word2vec [19, 20] was proposed to build a simplified shal-”
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., et al. (2025) · arXiv · doi:10.48550/arxiv.2501.12948 · Cited by 18,918
Self-citationMastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 2×
“As an early attempt, ELMo [21] was proposed to capture context-aware word representations by first pre-training a bidirectional LSTM (biLSTM) network (instead of learning fixed word representations) and then fine-tuning the biLSTM network according to specific downstream tasks.”
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625 · cited 5×
“icy Optimization (PPO) [216]) to adapt LLMs to human feedback by learning a reward model.”
InfluentialI.—COMPUTING MACHINERY AND INTELLIGENCE
TURING, A. M. (1950) · Mind · doi:10.1093/mind/lix.236.433 · Cited by 10,290 · cited 1×
“It has been a longstanding research challenge to achieve this goal, to enable machines to read, write, and communicate like humans [3].”
Research works printed but not linked: 3
- Touvron H, Lavril T, Izacard G, Martinet X, Lachaux M A, Front. Comput. Sci., 2026, 20(12): 2012627 Frontiers of Computer Science | Issue 12 | Volume 20 | December 2026 | 2012627-27
- Guo D, Yang D, Zhang H, Song J, Wang P, et al. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 2025, 645(8081): 633-638 · doi:10.1038/s41586-025-09422-z · Paper page
- Sutton R S, McAllester D, Singh S, Mansour Y. Policy gradient methods for reinforcement learning with function approximation. In: Front. Comput. Sci., 2026, 20(12): 2012627 Frontiers of Computer Science | Issue 12 | Volume 20 | December 2026 | 2012627-33
Cited by
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 4,688
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254
Self-citationA survey on large language model based autonomous agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40231-1 · Cited by 1,941
“Ac-tually, simulating the human negative aspects can be even more important, since an important goal of simulation is to discover and solve problems, and without negative aspects means no problem to be solved.”
Self-citationEvaluating Object Hallucination in Large Vision-Language Models
Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, X., & Wen, J.-R. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.20 · Cited by 559
Self-citation
Metadata
- Authors
- Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., Wang, X., Du, Y., Yang, C., Chen, Y., Chen, Z., Jiang, J., Ren, R., Li, Y., Tang, X., Liu, Z., Hu, Y., Nie, J.-Y., & Wen, J.-R.
- Affiliations
- Renmin University of China (CN)
- Université de Montréal (CA)
- Corresponding author
- Wen, Ji-Rong
- ORCID
- Venue
- Frontiers of Computer Science
- Published
- 2026-05-09
- Volume
- 20
- Issue
- 12
- ISSN
- 2095-2228 (ISSN-L)
- 2095-2236
- OpenAlex
- W4362515116
- Semantic Scholar
- 36709000a9272e941338244ef80ff5ab1dd2bba1
- Cited by
- 2,093
- References
- 355
- Citations by year
- 2026: 386
- FWCI
- 1299.20
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Frontiers of Computer Science (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, public-domain)
- Deep Blue (University of Michigan) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2121789)
- National Natural Science Foundation of China (92470205)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- large language models, pre-training, self-supervised learning, model architecture, data curation, fine-tuning, reinforcement learning, alignment, safety, in-context learning, prompt engineering, agentic reasoning, LLM evaluation, benchmark, reasoning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A Survey of Large Language Models. https://app.cochat.ai/papers/10.1007/s11704-026-60308-3
- The paper
- Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., Wang, X., Du, Y., Yang, C., Chen, Y., Chen, Z., Jiang, J., Ren, R., Li, Y., Tang, X., . . . Wen, J.-R. (2026). A Survey of Large Language Models. Frontiers of Computer Science. https://doi.org/10.1007/s11704-026-60308-3
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.