CoChat

A Survey of Large Language Models

Open access
Zhao, Wayne Xin, Zhou, Kun, Li, Junyi, Tang, Tianyi, Dong, Zican, Hou, Yupeng, Zhang, Beichen, Min, Yingqian, Zhang, Junjie, Liu, Peiyu, Wang, Xiaolei, Du, Yifan, Yang, Chen, Chen, Yushuo, Chen, Zhipeng, Jiang, Jinhao, Ren, Ruiyang, Li, Yifan, Tang, Xinyu, Liu, Zikang, Hu, Yiwen, Nie, Jian-Yun, Wen, Ji-Rong · 2026 · Frontiers of Computer Science

Published version of a preprint

The DOI matches its Crossref record; 439 of the 583 references it cites were found in a registry or an index, and 128 are not checked yet; 236 of the 360 sources its claims cite were checked for support.

Abstract

The rapid evolution of large language models (LLMs) has driven a transformative shift in artificial intelligence (AI), reshaping both research paradigms and practical applications. Distinguished from their predecessors by unprecedented scale and advanced capabilities, LLMs necessitate new frameworks for understanding their development, behavior, and societal impact. This survey systematically reviews recent advancements in LLM techniques across four key dimensions: (1) pre-training methodologies, which establish core model capabilities through large-scale self-supervised training, architectural innovations, and data curation strategies; (2) post-training techniques, including supervised fine-tuning and reinforcement learning, which adapt foundational models to downstream tasks and enhance their alignment and safety; (3) utilization strategies, such as in-context learning, prompt engineering, and agentic reasoning, that optimize real-world deployment and enable effective interaction with external environments; and (4) evaluation methods, encompassing benchmarks for key ability dimensions such as core language capabilities, reasoning, and safety, which support comprehensive and reliable assessment of model performance. Additionally, we identify critical research issues, including those concerning theoretical foundations, efficient scaling, alignment, and agentic capability, and highlight the open challenges they present. By synthesizing state-of-the-art insights and emerging trends, this survey aims to provide a systematic and comprehensive framework for understanding the trajectory, current limitations, and future directions of LLM progress.

Verified 2026-11-19 · 128 references not checked yet

Overview

CoChat Summary

TL;DR

This survey systematically reviews recent LLM advances across pre-training, post-training, utilization, and evaluation, and identifies open challenges in theory, scaling, alignment, and agentic capability. It synthesizes techniques and representative model/training details rather than presenting new experiments.

Key findings

  • The survey organizes LLM development into four interconnected dimensions: pre-training (data, architecture, training, mid-training), post-training (SFT and RL), utilization (prompting, inference-time techniques, agents, efficiency), and evaluation (benchmarks and approaches).
  • Pre-training examples include GPT-3 (175B parameters), PaLM (540B), LLaMA-3.1 (405B, 15T tokens), DeepSeek-V3 (671B MoE, 14.8T tokens), Qwen3 (235B MoE, 36T tokens), and Kimi K2 (1T MoE, 15.5T tokens); LLaMA-3.1 allocates 25% of tokens to mathematical/reasoning tasks and 17% to code.
  • Post-training instruction mixes typically cover mathematical reasoning (20%–30%), coding (20%–30%), agentic tasks (10%–20%), general tasks (20%–40%), and a minor proportion for AI identity/safety; open-source LLMs often use about one million instruction instances (e.g., DeepSeek-V3 1.5M, Qwen2.5 1M, Tulu-v3 939K).
  • Evaluation and capability trends include DeepSeek-R1 achieving over 90% accuracy on MMLU, while the paper notes benchmark saturation and open issues in fairness, practical utility, and superintelligence assessment.

Methods

  • Design: a systematic literature survey, described as an updated and refined version of an earlier arXiv article and expanded into a book, covering four critical stages of the LLM lifecycle.
  • Data/sample: published research literature, technical reports, benchmarks, and model documentation; no primary data collection or new experiments are reported.
  • Main analysis: qualitative synthesis with comparative tables, including detailed formulations for network configurations, training details of representative LLMs, and a comparison of post-training methods; the authors also maintain a curated GitHub repository of LLM resources.

Limitations

  • The paper does not state limitations of the survey itself; it identifies open research challenges including theoretical foundations, architecture innovations, scaling enhancement, data barrier, training efficacy, alignment and safety, capacity assessment, agentic AI, and sustainability/responsibility.
  • Evident from the text: as a survey, it reports no new experiments or empirical validation and relies on cited literature and technical reports.
  • Evident from the text: no systematic search or inclusion protocol is described, and coverage is selective and likely to date quickly in a fast-moving field.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s11704-026-60308-3

Cites

439 of 583 references verified0 citation errors · 16 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 A timeline of representative LLMs released in recent years. Models with publicly available checkpoints are highlighted in yellow
Fig. 1 A timeline of representative LLMs released in recent years. Models with publicly available checkpoints are highlighted in yellowFrom the PDF
Fig. 2 An overview of pre-training, post-training, utilization and evaluation for LLMs
Fig. 2 An overview of pre-training, post-training, utilization and evaluation for LLMsFrom the PDF
Fig. 3 An illustration of a typical data preparation pipeline for pre-training LLMs
Fig. 3 An illustration of a typical data preparation pipeline for pre-training LLMsFrom the PDF
Fig. 4 The overall pipeline of LLM post-training
Fig. 4 The overall pipeline of LLM post-trainingFrom the PDF
Fig. 5 An overview of LLM utilization and deployment approaches
Fig. 5 An overview of LLM utilization and deployment approachesFrom the PDF
Fig. 6 Performance trend of LLMs on mainstream benchmarks in recent years
Fig. 6 Performance trend of LLMs on mainstream benchmarks in recent yearsFrom the PDF
Image 7
From the PDF
Image 8
From the PDF
Image 9
From the PDF
Image 10
From the PDF
Image 11
From the PDF
Image 12
From the PDF
Image 13
From the PDF
Image 14
From the PDF
Image 15
From the PDF
Image 16
From the PDF
Image 17
From the PDF
Image 18
From the PDF
Image 19
From the PDF
Image 20
From the PDF
Image 21
From the PDF
Image 22
From the PDF
reasoning.
reasoning.From the PDF
models.
models.From the PDF
Image 25
From the PDF
Image 26
From the PDF
Image 27
From the PDF

Verification

Claims the paper makes

178 Valid · 15 Not valid · 50 Not checked · 70 Own results and interpretations

117 more claims · 77 with their cited sources checked

Create a free account to see all 313 claims and to verify the 19 references still open.

See all claims — free account
  • It may state two separate claims as one. 96
  • Probably not a substantial claim. 132
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Citations

Cites

Show all 583 references

Research works printed but not linked: 3

  1. Touvron H, Lavril T, Izacard G, Martinet X, Lachaux M A, Front. Comput. Sci., 2026, 20(12): 2012627 Frontiers of Computer Science | Issue 12 | Volume 20 | December 2026 | 2012627-27
  2. Guo D, Yang D, Zhang H, Song J, Wang P, et al. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 2025, 645(8081): 633-638 · doi:10.1038/s41586-025-09422-z · Paper page
  3. Sutton R S, McAllester D, Singh S, Mansour Y. Policy gradient methods for reinforcement learning with function approximation. In: Front. Comput. Sci., 2026, 20(12): 2012627 Frontiers of Computer Science | Issue 12 | Volume 20 | December 2026 | 2012627-33

Cited by

See all 8,643

Metadata

Authors
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., Wang, X., Du, Y., Yang, C., Chen, Y., Chen, Z., Jiang, J., Ren, R., Li, Y., Tang, X., Liu, Z., Hu, Y., Nie, J.-Y., & Wen, J.-R.
Affiliations
  • Renmin University of China (CN)
  • Université de Montréal (CA)
Corresponding author
Wen, Ji-Rong
Venue
Frontiers of Computer Science
Published
2026-05-09
Volume
20
Issue
12
ISSN
  • 2095-2228 (ISSN-L)
  • 2095-2236
OpenAlex
W4362515116
Cited by
2,093
References
355
Citations by year
2026: 386
FWCI
1299.20
Type
journal article
Language
en
Open access
Licence
Locations
  • Frontiers of Computer Science (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, public-domain)
  • Deep Blue (University of Michigan) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (2121789)
  • National Natural Science Foundation of China (92470205)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
large language models, pre-training, self-supervised learning, model architecture, data curation, fine-tuning, reinforcement learning, alignment, safety, in-context learning, prompt engineering, agentic reasoning, LLM evaluation, benchmark, reasoning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A Survey of Large Language Models. https://app.cochat.ai/papers/10.1007/s11704-026-60308-3
The paper
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., Wang, X., Du, Y., Yang, C., Chen, Y., Chen, Z., Jiang, J., Ren, R., Li, Y., Tang, X., . . . Wen, J.-R. (2026). A Survey of Large Language Models. Frontiers of Computer Science. https://doi.org/10.1007/s11704-026-60308-3

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.