Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 309 of the 309 references it cites were found in a registry or an index; 17 of the 163 sources its claims cite were checked for support.

Pre-trained models for natural language processing: A survey

Open access
Qiu, XiPeng, Sun, TianXiang, Xu, YiGe, Shao, YunFan, Dai, Ning, Huang, XuanJing · 2020 · Science China Technological Sciences

Abstract

Recently, the emergence of pre-trained models (PTMs) has brought natural language processing (NLP) to a new era. In this survey, we provide a comprehensive review of PTMs for NLP. We first briefly introduce language representation learning and its research progress. Then we systematically categorize existing PTMs based on a taxonomy from four different perspectives. Next, we describe how to adapt the knowledge of PTMs to downstream tasks. Finally, we outline some potential directions of PTMs for future research. This survey is purposed to be a hands-on guide for understanding, using, and developing PTMs for various NLP tasks.

Overview

CoChat Summary

TL;DR

This survey reviews pre-trained models (PTMs) for natural language processing, covering background, architectures, pre-training tasks, extensions, adaptation methods, resources, and applications. It proposes a four-perspective taxonomy of PTMs and outlines future directions including efficiency, compression, knowledge transfer, and interpretability.

Key findings

  • Proposes a taxonomy of PTMs from four perspectives: representation type, model architecture, pre-training task type, and extensions for specific scenarios (knowledge-enriched, multilingual/language-specific, multi-modal, domain-specific, and compressed PTMs).
  • Categorizes pre-training tasks into supervised, unsupervised, and self-supervised learning, and details LM, MLM, PLM, DAE, CTL, NSP/SOP, and RTD; notes that the necessity of NSP has been questioned and that ALBERT replaces it with SOP.
  • Reports representative PTM GLUE scores in Table 2, including BERT 81.9, RoBERTa 88.5, XLNet 90.5, ELECTRA 88.6, and T5 89.7; compression examples include BERT-of-Theseus at 1.94× faster while retaining more than 98% performance, and TinyBERT at 9.4× speedup with 14.5M parameters and GLUE 76.5.
  • Describes adaptation approaches including feature extraction versus fine-tuning, layer selection, and prompt-based tuning; fine-tuning has become the mainstream adaptation method since ULMFiT and BERT.

Methods

  • Survey/review design: comprehensive review of PTMs for NLP, synthesizing background knowledge, model architecture, pre-training tasks, extensions, adaptation approaches, resources, and applications.
  • Data or sample: published literature and open-source resources; no new experiments or datasets are collected.
  • Main analysis: qualitative synthesis and a proposed taxonomy from four perspectives, with tables comparing representative PTMs and compressed PTMs.

Limitations

  • Paper-stated: the Transformer’s computation complexity is quadratic in input length, and most current PTMs cannot process sequences longer than 512 tokens due to GPU memory limits.
  • Paper-stated: fine-tuning is parameter-inefficient because every downstream task has its own fine-tuned parameters; interpretability and reliability remain limited, and PTMs are vulnerable to adversarial attacks.
  • Evident from the text: as a survey, it does not report a systematic search protocol or new empirical evaluation; coverage is organized around representative models and resources rather than an exhaustive meta-analysis.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s11431-020-1647-3

Cites

309 of 309 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

3 Valid · 19 Not valid · 116 Not checked · 75 Own results and interpretations

80 more claims · 3 with their cited sources checked

Create a free account to see all 213 claims and to verify the 146 references still open.

See all claims — free account
  • Probably not a substantial claim. 100
  • It may state two separate claims as one. 50
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 276 references

Research works printed but not linked: 4

  1. Bahdanau D, Cho K, Bengio Y. Neural machine translation by jointly learning to align and translate. In: Proceedings of the International Conference on Learning Representations. San Diego, 2015
  2. Hewitt J, Manning C D. A structural probe for finding syntax in word representations. In: Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Minneapolis, 2019. 4129–4138
  3. Reif E, Yuan A, Wattenberg M, et al. Visualizing and measuring the geometry of BERT. In: Proceedings of the Advances in Neural Information Processing Systems. Vancouver, 2019. 8592–8600
  4. Michel P, Levy O, Neubig G. Are sixteen heads really better than one? In: Proceedings of the Advances in Neural Information Processing Systems. Vancouver, 2019. 14014–14024

Cited by

See all 1,591

Metadata

Authors
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X.
Affiliations
Fudan University (CN)
Corresponding author
Qiu, XiPeng
Venue
Science China Technological Sciences
Published
2020-09-15
Volume
63
Issue
10
Pages
1872-1897
ISSN
  • 1869-1900 (ISSN-L)
  • 1674-7321
OpenAlex
W3011574394
Cited by
1,591
References
276
Citations by year
2026: 123 · 2025: 238 · 2024: 286 · 2023: 383 · 2022: 317 · 2021: 200 · 2020: 42
FWCI
91.08
Type
journal article
Language
en
Open access
Licence
Locations
  • Science China Technological Sciences (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Natural Science Foundation of China (61751201, 61672162)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
pre-trained models, language representation learning, transfer learning, fine-tuning, downstream tasks, natural language processing

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Pre-trained models for natural language processing: A survey. https://app.cochat.ai/papers/10.1007/s11431-020-1647-3
The paper
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020). Pre-trained models for natural language processing: A survey. Science China Technological Sciences. https://doi.org/10.1007/s11431-020-1647-3

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.