CoChat
Sign Up
PreprintCited by 4,692Topic ModelingAdvanced Neural Network ApplicationsDomain Adaptation and Few-Shot Learning

DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

Open access
Sanh, Victor, Debut, Lysandre, Chaumond, Julien, Wolf, Thomas·2019·arXiv
DOI 10.48550/arxiv.1910.01108arXiv 1910.01108
Open this paper in CoChat — free accountOpen PDF

Abstract

As Transfer Learning from large-scale pre-trained models becomes more prevalent in Natural Language Processing (NLP), operating these large models in on-the-edge and/or under constrained computational training or inference budgets remains challenging. In this work, we propose a method to pre-train a smaller general-purpose language representation model, called DistilBERT, which can then be fine-tuned with good performances on a wide range of tasks like its larger counterparts. While most prior work investigated the use of distillation for building task-specific models, we leverage knowledge distillation during the pre-training phase and show that it is possible to reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster. To leverage the inductive biases learned by larger models during pre-training, we introduce a triple loss combining language modeling, distillation and cosine-distance losses. Our smaller, faster and lighter model is cheaper to pre-train and we demonstrate its capabilities for on-device computations in a proof-of-concept experiment and a comparative on-device study.

AI snapshot

Written from the paper’s full text.

TL;DR

DistilBERT is a distilled version of BERT pre-trained with knowledge distillation, reducing parameters by 40% while retaining 97% of BERT’s GLUE performance and running 60% faster at inference. It is presented as a general-purpose, lighter model suitable for constrained and on-device NLP applications.

Key findings

  • DistilBERT has 66M parameters vs. BERT-base’s 110M and takes 410s vs. 668s for a CPU inference pass on STS-B, i.e. 40% smaller and 60% faster.
  • On GLUE dev sets, DistilBERT scores 77.0 macro-score vs. BERT-base’s 79.5, retaining 97% of performance; it is always on par or improving over the ELMo baseline.
  • On IMDb test accuracy, DistilBERT reaches 92.82 vs. BERT-base’s 93.46; on SQuAD v1.1, DistilBERT reaches 77.7/85.8 EM/F1 vs. BERT-base’s 81.2/88.5.
  • Ablations show removing the cosine loss costs -2.96 GLUE macro-score, removing distillation loss costs -1.46, removing MLM loss costs -0.31, and random initialization costs -3.69.

Methods

  • Knowledge distillation during pre-training: a student with BERT’s architecture but half the layers, no token-type embeddings and no pooler, initialized from every other teacher layer.
  • Training objective is a triple loss combining distillation loss \(L_{ce}\), masked language modeling loss \(L_{mlm}\), and cosine embedding loss \(L_{cos}\), using softmax-temperature.
  • Trained on English Wikipedia + Toronto Book Corpus with large batches up to 4K examples, dynamic masking, no next-sentence prediction, on 8 16GB V100 GPUs for approximately 90 hours; evaluated on GLUE, IMDb, SQuAD v1.1, CPU inference, and an iPhone 7 Plus on-device proof-of-concept.

Limitations

  • The paper does not state an explicit limitations section.
  • Evident from the text: evaluation is limited to English benchmarks and a small set of downstream tasks, and the on-device study is described as a proof-of-concept; the model could be further reduced with quantization.
  • Evident from the text: the SQuAD second-distillation result is reported inconsistently—the prose says 79.8 F1 and 70.4 EM, while Table 2 lists DistilBERT (D) as 79.1/86.9.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.1910.01108

Cites

23 of 23 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

3 of 4 cited sources checked · 2 supported · 0 flagged · accuracy not yet measured

  • Findingown data
    § Abstract · p. 1

    DistilBERT has 40% fewer parameters than BERT.

    “we leverage knowledge distillation during the pre-training phase and show that it is possible to reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster.”
    • Also stated in: § Size and inference speed · p. 4 · plain
      “DistilBERT has 40% fewer parameters than BERT and is 60% faster than BERT.”
    • Also stated in: § 6 Conclusion and future work · p. 5 · plain
      “We introduced DistilBERT, a general-purpose pre-trained version of BERT, 40% smaller, 60% faster, that retains 97% of the language understanding capabilities.”
    Quote in the paperSubstantial claim
  • Findingown data
    § Abstract · p. 1

    DistilBERT can be fine-tuned with good performances on a wide range of tasks like its larger counterparts.

    “we propose a method to pre-train a smaller generalpurpose language representation model, called DistilBERT, which can then be finetuned with good performances on a wide range of tasks like its larger counterparts.”
    • Also stated in: § 1 Introduction · p. 2 · plain
      “Our general-purpose pre-trained models can be fine-tuned with good performances on several downstream tasks, keeping the flexibility of larger models.”
    Quote in the paperSubstantial claim
  • Findingown data
    § 1 Introduction · p. 2

    All components of the triple loss are important for best performances.

    “Further ablation studies indicate that all the components of the triple loss are important for best performances.”
    Quote in the paperNot a substantial claim
  • Findingown data
    § Size and inference speed · p. 4

    DistilBERT model weighs 207 MB.

    “Excluding the tokenization step, DistilBERT is 71% faster than BERT, and the whole model weighs 207 MB (which could be further reduced with quantization).”
    Quote in the paperSubstantial claim
  • Findingown data
    § Abstract · p. 1

    DistilBERT is 60% faster than BERT at inference time.

    “we leverage knowledge distillation during the pre-training phase and show that it is possible to reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster.”
    • Also stated in: § Size and inference speed · p. 4 · plain
      “DistilBERT has 40% fewer parameters than BERT and is 60% faster than BERT.”
    • Also stated in: § 6 Conclusion and future work · p. 5 · plain
      “We introduced DistilBERT, a general-purpose pre-trained version of BERT, 40% smaller, 60% faster, that retains 97% of the language understanding capabilities.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Inference speed measured on a single GLUE task (STS-B) on CPU is representative of DistilBERT's speed advantage generally.
  • Classified as probably not a substantial claim. 7
  • May combine two claims. 2

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 1 Introduction · p. 1

    The environmental cost of exponentially scaling these models' computational requirements is a concern.

    “environmental cost of exponentially scaling these models' computational requirements as mentioned in Schwartz et al. [2019], Strubell et al. [2019].”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites Schwartz et al., 2019 source
    • cites Strubell et al., 2019 source

      The Introduction (page unavailable) says that computationally hungry NLP models require considerable energy and have associated environmental costs. It also says model development can multiply those costs thousands of times, but it does not establish that the models’ computational requirements are scaling exponentially. (LLM, conf=0.84)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation
    § 3 DistilBERT: a distilled version of BERT · p. 3

    DistilBERT was trained following best practices for training BERT proposed in Liu et al. [2019].

    “We applied best practices for training BERT model recently proposed in Liu et al. [2019].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Liu et al., 2019 source

  • May combine two claims.

    Findingown data
    § 4 Experiments · p. 3

    On GLUE, DistilBERT is always on par or improving over the ELMo baseline, up to 19 points of accuracy on STS-B.

    “Among the 9 tasks, DistilBERT is always on par or improving over the ELMo baseline (up to 19 points of accuracy on STS-B).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.1 Downstream task benchmark · p. 3

    With an additional distillation step during fine-tuning on SQuAD, DistilBERT reaches 79.8 F1 and 70.4 EM, within 3 points of the full model.

    “In this case, we were able to reach interesting performances given the size of the model: 79.8 F1 and 70.4 EM, i.e. within 3 points of the full model.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationown data
    § 6 Conclusion and future work · p. 5

    DistilBERT is a compelling option for edge applications.

    “We further demonstrated that DistilBERT is a compelling option for edge applications.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § 3 DistilBERT: a distilled version of BERT · p. 3

    BERT and DistilBERT results are the medians of 5 runs with different seeds.

    “BERT and DistilBERT results are the medians of 5 runs with different seeds.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation
    § 1 Introduction · p. 1

    Training even larger pre-trained models still leads to better performances on downstream tasks.

    “current research 1 on pre-trained models indicates that training even larger models still leads to better performances on downstream tasks.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites MegatronLM

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § 4 Experiments · p. 3

    DistilBERT was fine-tuned without ensembling or multi-tasking.

    “We report scores on the development sets for each task by fine-tuning DistilBERT without the use of ensembling or multi-tasking scheme for fine-tuning (which are mostly orthogonal to the present work).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given
    § Abstract · p. 1

    Most prior work investigated the use of distillation for building task-specific models.

    “While most prior work investigated the use of distillation for building task-specific models, we leverage knowledge distillation during the pre-training phase”
    Quote in the paperNot a substantial claim
9 of 9

  • The paper relies on this without support.

    Operating large pre-trained models on-device in real-time is hampered by their growing computational and memory requirements.

    “the growing computational and memory requirements of these models may hamper wide adoption.”

  • The paper relies on this without support.

    Near-zero probabilities in a trained model's output distribution reflect its generalization capabilities.

    “some of these”

  • The paper relies on this without support.

    Inference speed measured on a single GLUE task (STS-B) on CPU is representative of DistilBERT's speed advantage generally.

    “Inference time of a full pass of GLUE task STS-B (sentiment analysis) on CPU with a batch size of 1.”
    “DistilBERT has 40% fewer parameters than BERT and is 60% faster than BERT.”
3 of 3

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Distillation has been successfully applied to LLMs, where distilled students approximate the generation patterns of larger teachers while retaining much of their performance.

    “This idea has been successfully applied to LLMs, where distilled students approximate the generation patterns of larger teachers while retaining much of their performance (Sanh et al., 2019).”

    From Vision-Guided Iterative Refinement for Frontend Code Generation (2026) · cites this paper as Sanh et al., 2019

    Section 2 (p. 2) describes a student trained to reproduce a larger teacher’s behavior using its output probabilities, and the Abstract (p. 1) reports that DistilBERT retains 97% of BERT’s language-understanding performance. The paper studies masked-language-model distillation and understanding tasks; it does not establish that students approximate larger teachers’ text-generation patterns. (LLM, conf=0.88)

    Experimental — accuracy not yet measured

  • Benchmarking is not a reliable estimator of parameter count.

    “even benchmarking is not a reliable estimator of parameter count [12], [15], [28], [29]”

    From Hollow-LLM Attack: Computationally Trivial Weights in Zero-Knowledge Verification of LLM Inference (2026) · cites this paper as [29]

    The Experiments section (p. 3) reports that DistilBERT retains 97% of BERT’s benchmark performance with 40% fewer parameters. The model comparisons in Tables 1 and 3 (p. 3) also show that benchmark scores do not track parameter counts consistently. These examples support caution about estimating parameter count from benchmark results, but the paper does not assess or establish the broader claim that benchmarking is an unreliable estimator. (LLM, conf=0.81)

    Experimental — accuracy not yet measured

  • DistilBERT achieves similar performance to the original BERT model while using fewer resources, reducing the size of general BERT by 40% and retaining 97% language understanding capability.

    “It achieves similar performance to the original BERT model while using fewer resources. The approach leveraged knowledge distillation during the pretraining phase and reduced the size of general BERT by 40%, yet it retained a language understanding capability of 97%.”

    From A Review on Deep-Learning-Based Cyberbullying Detection (2023) · cites this paper as [53]

  • Knowledge distillation does not provide any quality improvement for their model sizes.

    “Unlike in many similar circumstances, we do not employ knowledge distillation [56], because we discovered that for our sizes of the models, this technique does not provide any quality improvement.”

    From Full Line Code Completion: Bringing AI to Desktop (2024) · cites this paper as [56]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • SQuAD distillation results conflict with Table 2

    Major

    The prose reports 79.8 F1 and 70.4 EM for the additional-distillation setting, while Table 2 reports 86.9 F1 and 79.1 EM for DistilBERT (D).

  • STS-B improvement over ELMo is not 19 points

    Minor

    Table 1 gives DistilBERT 86.9 and ELMo 70.4 on STS-B, an improvement of 16.5 points rather than 19.

    Reported 19 · recomputed 16.5

2 of 2

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In KDD , 2006.

Cited by

Sorted by citations

Related

Authors
Sanh, V., Debut, L., Chaumond, J., & Wolf, T.
Venue
arXiv
Publisher
arXiv
Published
2019
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1910.01108
arXiv
1910.01108
OpenAlex
W2978017171
Cited by
4,653
References
18
Citations by year
2026: 67 · 2025: 443 · 2024: 801 · 2023: 1136 · 2022: 736 · 2021: 968 · 2020: 486 · 2019: 15 · 2018: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
  • Transactions on Graph Data and Knowledge (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
DistilBERT, knowledge distillation, BERT, pre-training, model compression, on-device computation, language modeling, transfer learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-23.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-23.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review