The DOI matches its Crossref record; 309 of the 309 references it cites were found in a registry or an index; 17 of the 163 sources its claims cite were checked for support.
Pre-trained models for natural language processing: A survey
Open accessAbstract
Recently, the emergence of pre-trained models (PTMs) has brought natural language processing (NLP) to a new era. In this survey, we provide a comprehensive review of PTMs for NLP. We first briefly introduce language representation learning and its research progress. Then we systematically categorize existing PTMs based on a taxonomy from four different perspectives. Next, we describe how to adapt the knowledge of PTMs to downstream tasks. Finally, we outline some potential directions of PTMs for future research. This survey is purposed to be a hands-on guide for understanding, using, and developing PTMs for various NLP tasks.
Overview
CoChat Summary
TL;DR
This survey reviews pre-trained models (PTMs) for natural language processing, covering background, architectures, pre-training tasks, extensions, adaptation methods, resources, and applications. It proposes a four-perspective taxonomy of PTMs and outlines future directions including efficiency, compression, knowledge transfer, and interpretability.
Key findings
- Proposes a taxonomy of PTMs from four perspectives: representation type, model architecture, pre-training task type, and extensions for specific scenarios (knowledge-enriched, multilingual/language-specific, multi-modal, domain-specific, and compressed PTMs).
- Categorizes pre-training tasks into supervised, unsupervised, and self-supervised learning, and details LM, MLM, PLM, DAE, CTL, NSP/SOP, and RTD; notes that the necessity of NSP has been questioned and that ALBERT replaces it with SOP.
- Reports representative PTM GLUE scores in Table 2, including BERT 81.9, RoBERTa 88.5, XLNet 90.5, ELECTRA 88.6, and T5 89.7; compression examples include BERT-of-Theseus at 1.94× faster while retaining more than 98% performance, and TinyBERT at 9.4× speedup with 14.5M parameters and GLUE 76.5.
- Describes adaptation approaches including feature extraction versus fine-tuning, layer selection, and prompt-based tuning; fine-tuning has become the mainstream adaptation method since ULMFiT and BERT.
Methods
- Survey/review design: comprehensive review of PTMs for NLP, synthesizing background knowledge, model architecture, pre-training tasks, extensions, adaptation approaches, resources, and applications.
- Data or sample: published literature and open-source resources; no new experiments or datasets are collected.
- Main analysis: qualitative synthesis and a proposed taxonomy from four perspectives, with tables comparing representative PTMs and compressed PTMs.
Limitations
- Paper-stated: the Transformer’s computation complexity is quadratic in input length, and most current PTMs cannot process sequences longer than 512 tokens due to GPU memory limits.
- Paper-stated: fine-tuning is parameter-inefficient because every downstream task has its own fine-tuned parameters; interpretability and reliability remain limited, and PTMs are vulnerable to adversarial attacks.
- Evident from the text: as a survey, it does not report a systematic search protocol or new empirical evaluation; coverage is organized around representative models and resources rather than an exhaustive meta-analysis.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
3 Valid · 19 Not valid · 116 Not checked · 75 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“They initialized LSTMs with a language model (LM) or a sequence autoencoder, and found the pre-training can improve the training and generalization of LSTMs in many text classification tasks.”
What the sources say
- Misattributedcites [35] Ramachandran P, Liu P J, Le Q. Unsupervised pretraining for sequence to sequence learning. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. Copenhagen, 2017. 383–391 sourceExperimental — accuracy not yet measured
Our copy of Ramachandran et al. (2017) contains no text-classification experiments — its results are WMT English→German machine translation (§3.1) and CNN/Daily Mail abstractive summarization. The only passage touching the claim is the related-work paragraph (section 'Pretrain'), which mentions the point solely to credit it to another work: 'Dai and Le (2015); Radford et al. (2017) are amongst the rare studies which showed the benefits of pretraining in a semi-supervised learning setting. Their methods are similar to ours except that they did not have a decoder network.' The reference list (entry 9) names that work as Dai and Le, 'Semi-supervised sequence learning', NIPS 2015. The specifics the sentence attributes — initializing LSTMs with an LM or a sequence autoencoder, gains on many text classification tasks, improved training and generalization — are not reported or relied on here, so the finding belongs to Dai and Le (2015), not to this paper. (LLM, conf=0.70) It matches “Semi-supervised sequence learning” (2015) instead.
Checked by a reasoning model — may be wrong.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Misattributed
Every source the paper cites for this claim was checked and supports it.
In the paper
“Peters et al. [14] pre-trained 2-layer LSTM encoder with a bidirectional language model (BiLM), consisting of a forward LM and a backward LM.”
What the sources say
- Supportedcites [14] Peters M E, Neumann M, Iyyer M, et al. Deep contextualized word representations. In: Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. New Orleans, 2018. 2227–2237 source
Section 3.4 states that the final pre-trained model uses two biLSTM layers. Section 3.1 defines its bidirectional language model as combining a forward LM and a backward LM, and the Introduction says the bidirectional LSTM is trained with a language-model objective. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Tenney et al. [120], Liu et al. [121] found that BERT performs well on many syntactic tasks such as part-of-speech tagging and constituent labeling.”
What the sources say
- Could not verifycites [120] Xie R, Liu Z, Jia J, et al. Representation learning of knowledge graphs with entity descriptions. In: Proceedings of the AAAI Conference on Artificial Intelligence. Phoenix, 2016. 2659–2665 sourceExperimental — accuracy not yet measured
Our copy of Xie et al. (2016) is entirely about the DKRL model for knowledge-graph representation learning from entity descriptions (FB15K/FB20K, knowledge graph completion, entity type classification, zero-shot scenario); no passage mentions BERT, part-of-speech tagging, or constituent labeling. Searches on 'BERT pre-trained language model syntactic tasks part-of-speech tagging constituent labeling' and on 'Liu et al. BERT representations linguistic capabilities' returned only the Introduction (p. 1), Datasets and Experiment Settings (pp. 4-5), Knowledge Graph Completion / Entity Classification (pp. 5-6) and the References (p. 7). The reference list (p. 7) contains no Tenney et al. and no 2019 BERT-analysis paper by Liu et al., so our copy neither supports nor contradicts the claim nor credits it to another work. (LLM, conf=0.80)
- Could not verifycites [121] Xu J, Qiu X, Chen K, et al. Knowledge graph representation with jointly structural and textual encoding. In: Proceedings of the International Joint Conference on Artificial Intelligence. Melbourne, 2017. 1318–1324 sourceExperimental — accuracy not yet measured
Our copy of the IJCAI-17 knowledge graph embedding paper addresses only jointly structural and textual encoding for triplet classification and link prediction on WN18/FB15K (§5 Experiment, §5.2 Link Prediction, §7 Conclusion tables). No passage mentions BERT, pre-trained contextual language-model evaluation, part-of-speech tagging, or constituent labeling; searches for 'BERT pre-trained language model syntactic tasks' and 'part-of-speech tagging constituent labeling' returned only semantically adjacent KG results, and the reference list (§References) contains no Tenney et al. or comparable BERT-probing work. Since our copy simply does not speak to the claim — and names no alternative source for it — the attribution can be neither supported nor contradicted here. (LLM, conf=0.80)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“The distributed word vectors, along with a simple supervised model, can learn to predict numeric and binary attributes of entities with a reasonable degree of accuracy.”
What the sources say
- Could not verifycites [119] Zhong H, Zhang J, Wang Z, et al. Aligning knowledge and text embeddings by entity descriptions. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. Lisbon, 2015. 267–272 sourceExperimental — accuracy not yet measured
The Experiments section reports link prediction and binary classification of whether a relation triplet is a correct fact, not prediction of an entity’s binary attributes. Searches for numeric attributes, regression, and a simple supervised attribute-prediction model found no passage addressing the claimed result. (LLM, conf=0.75)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“However, BERT is not good enough at semantic and fine-grained syntactic tasks, compared with simple syntactic tasks.”
What the sources say
- Could not verifycites [120] Xie R, Liu Z, Jia J, et al. Representation learning of knowledge graphs with entity descriptions. In: Proceedings of the AAAI Conference on Artificial Intelligence. Phoenix, 2016. 2659–2665 sourceExperimental — accuracy not yet measured
The Introduction (p. 1) compares CBOW and convolutional encoders for entity descriptions, while Knowledge Graph Completion (p. 5) evaluates DKRL on entity and relation prediction. Searches for BERT and for semantic versus fine-grained syntactic tasks found no passage assessing that comparison. (LLM, conf=0.78)
- Could not verifycites [121] Xu J, Qiu X, Chen K, et al. Knowledge graph representation with jointly structural and textual encoding. In: Proceedings of the International Joint Conference on Artificial Intelligence. Melbourne, 2017. 1318–1324 sourceExperimental — accuracy not yet measured
The Experiment section evaluates knowledge-graph link prediction and triplet classification, not BERT’s performance on semantic or syntactic tasks. Searches for BERT and the claimed task comparison found no passage addressing it in our copy. (LLM, conf=0.78)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
80 more claims · 3 with their cited sources checked
Create a free account to see all 213 claims and to verify the 146 references still open.
- Probably not a substantial claim. 100
- It may state two separate claims as one. 50
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Although there are comprehensive analyses (Min et al. 2021; Qiu et al. 2020) of Pretrained Foundation Models (PFMs) and their application to various NLP tasks, these analyses may not be directly applicable to LLMs due to inherent differences.”
From Large language models (LLMs): survey, technical frameworks, and future challenges (2024) · cites this paper as Qiu et al. 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.5 Model Compression · p. 13
“There are five ways to compress PTMs [166]: (1) model pruning , which removes less important parameters, (2) weight quantization [167], which uses fewer bits to represent the parameters, (3) parameter sharing across similar model units, (4) knowledge distillation [168], which trains a smaller student model that learns from intermediate outputs from the original model and (5) module replacing”
§ 4.5.4 Knowledge Distillation · p. 13
“Di erent to model compression, distillation techniques learn a small student model from a fixed teacher model through some optimization objectives, while compression techniques aiming at searching a sparser architecture.”
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 3×
“With the development of deep learning, various neural networks have been widely used to solve Natural Language Processing (NLP) tasks, such as convolutional neural networks (CNNs) [80, 86, 49], recurrent neural networks (RNNs) [173, 109], graph-based neural networks (GNNs) [159, 174, 124] and attention mechanisms [7,…”
BackgroundMethodHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 2×
“(2) Sequential models Sequential models usually capture the contextual representations of words in sequential order, such as LSTMs [63] and GRUs [23].”
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 10×
“, OpenAI GPT (Generative Pre-training) [142] and BERT (Bidirectional Encoder Representation from Transformer) [36].”
BackgroundMethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 10×
“, OpenAI GPT (Generative Pre-training) [142] and BERT (Bidirectional Encoder Representation from Transformer) [36].”
BackgroundMethodInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Semi-supervised classification with graph convolutional networks
Kipf, T. N., & Welling, M. (2016) · arXiv · doi:10.48550/arxiv.1609.02907 · Cited by 24,853 · cited 1×
“Some popular non-sequence models include Recursive NN [159], TreeLSTM [174, 222], and GCN [88].”
MethodDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Pan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · cited 1×
“With the development of deep learning, various neural networks have been widely used to solve Natural Language Processing (NLP) tasks, such as convolutional neural networks (CNNs) [80, 86, 49], recurrent neural networks (RNNs) [173, 109], graph-based neural networks (GNNs) [159, 174, 124] and attention mechanisms [7,…”
Method
Research works printed but not linked: 4
- Bahdanau D, Cho K, Bengio Y. Neural machine translation by jointly learning to align and translate. In: Proceedings of the International Conference on Learning Representations. San Diego, 2015
- Hewitt J, Manning C D. A structural probe for finding syntax in word representations. In: Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Minneapolis, 2019. 4129–4138
- Reif E, Yuan A, Wattenberg M, et al. Visualizing and measuring the geometry of BERT. In: Proceedings of the Advances in Neural Information Processing Systems. Vancouver, 2019. 8592–8600
- Michel P, Levy O, Neubig G. Are sixteen heads really better than one? In: Proceedings of the Advances in Neural Information Processing Systems. Vancouver, 2019. 14014–14024
Cited by
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
“For a more in-depth survey of pretrained LMs, see [105].”
BackgroundHan (2023) · doi:10.1109/tpami.2022.3152247 · Cited by 4,682
Large language models in medicine
Thirunavukarasu, A. J., Ting, D. S. J., Elangovan, K., Gutierrez, L., Tan, T. F., & Ting, D. S. W. (2023) · Nature Medicine · doi:10.1038/s41591-023-02448-8 · Cited by 3,944
“The pretraining task underlying published GPT models is termed language modeling: predicting the next and/or previous ‘token’ (usually analogous to ‘word’) in a sequence or sentence 11,21 .”
MethodDeep Learning--based Text Classification: A Comprehensive Review
Minaee, S., Kalchbrenner, N., Cambria, E., Nikzad, N., Chenaghlu, M., & Gao, J. (2022) · ACM Computing Surveys · doi:10.1145/3439726 · Cited by 1,538
“[145] categories popular PLMs by their representation types, model architectures, pre-training tasks, and downstream tasks.”
BackgroundSelf-supervised Learning: Generative or Contrastive
Liu, X., Zhang, F., Hou, Z., Mian, L., Wang, Z., Zhang, J., & Tang, J. (2021) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3090866 · Cited by 1,087
Metadata
- Authors
- Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X.
- Affiliations
- Fudan University (CN)
- Corresponding author
- Qiu, XiPeng
- Venue
- Science China Technological Sciences
- Published
- 2020-09-15
- Volume
- 63
- Issue
- 10
- Pages
- 1872-1897
- ISSN
- 1869-1900 (ISSN-L)
- 1674-7321
- OpenAlex
- W3011574394
- Cited by
- 1,591
- References
- 276
- Citations by year
- 2026: 123 · 2025: 238 · 2024: 286 · 2023: 383 · 2022: 317 · 2021: 200 · 2020: 42
- FWCI
- 91.08
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- www.springer.com/tdm (tdm)
- www.springer.com/tdm (vor)
- Locations
- Science China Technological Sciences (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (61751201, 61672162)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- pre-trained models, language representation learning, transfer learning, fine-tuning, downstream tasks, natural language processing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Pre-trained models for natural language processing: A survey. https://app.cochat.ai/papers/10.1007/s11431-020-1647-3
- The paper
- Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020). Pre-trained models for natural language processing: A survey. Science China Technological Sciences. https://doi.org/10.1007/s11431-020-1647-3
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.