The DOI matches its DataCite record; 23 of the 23 references it cites were found in a registry or an index; 3 of the 4 sources its claims cite were checked for support.
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Open accessAbstract
As Transfer Learning from large-scale pre-trained models becomes more prevalent in Natural Language Processing (NLP), operating these large models in on-the-edge and/or under constrained computational training or inference budgets remains challenging. In this work, we propose a method to pre-train a smaller general-purpose language representation model, called DistilBERT, which can then be fine-tuned with good performances on a wide range of tasks like its larger counterparts. While most prior work investigated the use of distillation for building task-specific models, we leverage knowledge distillation during the pre-training phase and show that it is possible to reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster. To leverage the inductive biases learned by larger models during pre-training, we introduce a triple loss combining language modeling, distillation and cosine-distance losses. Our smaller, faster and lighter model is cheaper to pre-train and we demonstrate its capabilities for on-device computations in a proof-of-concept experiment and a comparative on-device study.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
DistilBERT is a distilled version of BERT pre-trained with knowledge distillation, reducing parameters by 40% while retaining 97% of BERT’s GLUE performance and running 60% faster at inference. It is presented as a general-purpose, lighter model suitable for constrained and on-device NLP applications.
Key findings
- DistilBERT has 66M parameters vs. BERT-base’s 110M and takes 410s vs. 668s for a CPU inference pass on STS-B, i.e. 40% smaller and 60% faster.
- On GLUE dev sets, DistilBERT scores 77.0 macro-score vs. BERT-base’s 79.5, retaining 97% of performance; it is always on par or improving over the ELMo baseline.
- On IMDb test accuracy, DistilBERT reaches 92.82 vs. BERT-base’s 93.46; on SQuAD v1.1, DistilBERT reaches 77.7/85.8 EM/F1 vs. BERT-base’s 81.2/88.5.
- Ablations show removing the cosine loss costs -2.96 GLUE macro-score, removing distillation loss costs -1.46, removing MLM loss costs -0.31, and random initialization costs -3.69.
Methods
- Knowledge distillation during pre-training: a student with BERT’s architecture but half the layers, no token-type embeddings and no pooler, initialized from every other teacher layer.
- Training objective is a triple loss combining distillation loss \(L_{ce}\), masked language modeling loss \(L_{mlm}\), and cosine embedding loss \(L_{cos}\), using softmax-temperature.
- Trained on English Wikipedia + Toronto Book Corpus with large batches up to 4K examples, dynamic masking, no next-sentence prediction, on 8 16GB V100 GPUs for approximately 90 hours; evaluated on GLUE, IMDb, SQuAD v1.1, CPU inference, and an iPhone 7 Plus on-device proof-of-concept.
Limitations
- The paper does not state an explicit limitations section.
- Evident from the text: evaluation is limited to English benchmarks and a small set of downstream tasks, and the on-device study is described as a proof-of-concept; the model could be further reduced with quantization.
- Evident from the text: the SQuAD second-distillation result is reported inconsistently—the prose says 79.8 F1 and 70.4 EM, while Table 2 lists DistilBERT (D) as 79.1/86.9.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
15 Own result · 6 Own interpretation · 2 Own method · 2 No basis given · 1 Supported · 1 Partly supported · 1 Sources not found · accuracy not yet measured
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we leverage knowledge distillation during the pre-training phase and show that it is possible to reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster.”
- Also stated in: § Size and inference speed · p. 4 · plain
“DistilBERT has 40% fewer parameters than BERT and is 60% faster than BERT.”
- Also stated in: § 6 Conclusion and future work · p. 5 · plain
“We introduced DistilBERT, a general-purpose pre-trained version of BERT, 40% smaller, 60% faster, that retains 97% of the language understanding capabilities.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Size and inference speed · p. 4 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we propose a method to pre-train a smaller generalpurpose language representation model, called DistilBERT, which can then be finetuned with good performances on a wide range of tasks like its larger counterparts.”
- Also stated in: § 1 Introduction · p. 2 · plain
“Our general-purpose pre-trained models can be fine-tuned with good performances on several downstream tasks, keeping the flexibility of larger models.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 Introduction · p. 2 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Further ablation studies indicate that all the components of the triple loss are important for best performances.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Excluding the tokenization step, DistilBERT is 71% faster than BERT, and the whole model weighs 207 MB (which could be further reduced with quantization).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we leverage knowledge distillation during the pre-training phase and show that it is possible to reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster.”
- Also stated in: § Size and inference speed · p. 4 · plain
“DistilBERT has 40% fewer parameters than BERT and is 60% faster than BERT.”
- Also stated in: § 6 Conclusion and future work · p. 5 · plain
“We introduced DistilBERT, a general-purpose pre-trained version of BERT, 40% smaller, 60% faster, that retains 97% of the language understanding capabilities.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Inference speed measured on a single GLUE task (STS-B) on CPU is representative of DistilBERT's speed advantage generally.
- Also stated in: § Size and inference speed · p. 4 · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“we propose a method to pre-train a smaller generalpurpose language representation model, called DistilBERT”
- Also stated in: § 6 Conclusion and future work · p. 5 · plain
“We introduced DistilBERT, a general-purpose pre-trained version of BERT”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 6 Conclusion and future work · p. 5 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found it beneficial to add a cosine embedding loss ( L cos ) which will tend to align the directions of the student and teacher hidden states vectors.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On SQuAD, DistilBERT is within 3.9 points of the full BERT.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we leverage knowledge distillation during the pre-training phase and show that it is possible to reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster.”
- Also stated in: § 4 Experiments · p. 3 · plain
“DistilBERT also compares surprisingly well to BERT, retaining 97% of the performance with 40% fewer parameters.”
- Also stated in: § 6 Conclusion and future work · p. 5 · plain
“We introduced DistilBERT, a general-purpose pre-trained version of BERT, 40% smaller, 60% faster, that retains 97% of the language understanding capabilities.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 4 Experiments · p. 3 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“removing the Masked Language Modeling loss has little impact while the two distillation losses account for a large portion of the performance.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 7
- It may state two separate claims as one. 2
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“environmental cost of exponentially scaling these models' computational requirements as mentioned in Schwartz et al. [2019], Strubell et al. [2019].”
What the sources say
- Supportedcites Schwartz et al., 2019 source
The Abstract (p. 1) says the computations required for deep learning research have been doubling every few months and have a large carbon footprint. Section 2, “Red AI” (p. 4), likewise warns that the environmental cost of increasingly large models will not decrease soon. (LLM, conf=0.93)
- Partly supportedcites Strubell et al., 2019 sourceExperimental — accuracy not yet measured
The Introduction (page unavailable) says that computationally hungry NLP models require considerable energy and have associated environmental costs. It also says model development can multiply those costs thousands of times, but it does not establish that the models’ computational requirements are scaling exponentially. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“We applied best practices for training BERT model recently proposed in Liu et al. [2019].”
What the sources say
- Supportedcites Liu et al., 2019 source
The Introduction (p. 1) says Liu et al. propose an improved recipe for training BERT, and Section 4 (p. 4) evaluates which choices matter for BERT pretraining. The citing authors’ statement that they applied those practices to DistilBERT describes their own work, not a finding attributed to Liu et al. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Among the 9 tasks, DistilBERT is always on par or improving over the ELMo baseline (up to 19 points of accuracy on STS-B).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this case, we were able to reach interesting performances given the size of the model: 79.8 F1 and 70.4 EM, i.e. within 3 points of the full model.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We further demonstrated that DistilBERT is a compelling option for edge applications.”
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“BERT and DistilBERT results are the medians of 5 runs with different seeds.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“current research 1 on pre-trained models indicates that training even larger models still leads to better performances on downstream tasks.”
What the sources say
- Not matchedcites MegatronLM
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We report scores on the development sets for each task by fine-tuning DistilBERT without the use of ensembling or multi-tasking scheme for fine-tuning (which are mostly orthogonal to the present work).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“While most prior work investigated the use of distillation for building task-specific models, we leverage knowledge distillation during the pre-training phase”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The paper relies on this without support.
Operating large pre-trained models on-device in real-time is hampered by their growing computational and memory requirements.
“the growing computational and memory requirements of these models may hamper wide adoption.”
The paper relies on this without support.
Near-zero probabilities in a trained model's output distribution reflect its generalization capabilities.
“some of these”
The paper relies on this without support.
Inference speed measured on a single GLUE task (STS-B) on CPU is representative of DistilBERT's speed advantage generally.
“Inference time of a full pass of GLUE task STS-B (sentiment analysis) on CPU with a batch size of 1.”
“DistilBERT has 40% fewer parameters than BERT and is 60% faster than BERT.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This idea has been successfully applied to LLMs, where distilled students approximate the generation patterns of larger teachers while retaining much of their performance (Sanh et al., 2019).”
From Vision-Guided Iterative Refinement for Frontend Code Generation (2026) · cites this paper as Sanh et al., 2019
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“even benchmarking is not a reliable estimator of parameter count [12], [15], [28], [29]”
From Hollow-LLM Attack: Computationally Trivial Weights in Zero-Knowledge Verification of LLM Inference (2026) · cites this paper as [29]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“It achieves similar performance to the original BERT model while using fewer resources. The approach leveraged knowledge distillation during the pretraining phase and reduced the size of general BERT by 40%, yet it retained a language understanding capability of 97%.”
From A Review on Deep-Learning-Based Cyberbullying Detection (2023) · cites this paper as [53]
The cited source says what the claim attributes to it.
In the citing paper
“Unlike in many similar circumstances, we do not employ knowledge distillation [56], because we discovered that for our sizes of the models, this technique does not provide any quality improvement.”
From Full Line Code Completion: Bringing AI to Desktop (2024) · cites this paper as [56]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 4.1 Downstream task benchmark · p. 3
“In this case, we were able to reach interesting performances given the size of the model: 79.8 F1 and 70.4 EM, i.e. within 3 points of the full model.”
§ 3 DistilBERT: a distilled version of BERT · p. 3
“| DistilBERT (D) | - | 79.1/86.9 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“DistilBERT is always on par or improving over the ELMo baseline (up to 19 points of accuracy on STS-B).”
Reported 19 · recomputed 16.5
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
Self-citationSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222
Learning Word Vectors for Sentiment Analysis
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., & Potts, C. (2011) · Cited by 3,710
Energy and Policy Considerations for Deep Learning in NLP
Strubell, E., Ganesh, A., & McCallum, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1355 · Cited by 3,684
Research works printed but not linked: 1
- Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In KDD , 2006.
Cited by
Sorted by citations
Items 1,301–1,400 of 4,692
DoT: An efficient Double Transformer for NLP tasks with tables
Krichene, S., Müller, T., & Eisenschlos, J. M. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.289 · Cited by 17
ROSITA: Refined BERT cOmpreSsion with InTegrAted techniques
Liu, Y., Lin, Z., & Yuan, F. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i10.17056 · Cited by 14
SMURF: SeMantic and linguistic UndeRstanding Fusion for Caption Evaluation via Typicality Analysis
Feinglass, J., & Yang, Y. (2021) · doi:10.18653/v1/2021.acl-long.175 · Cited by 17
On the use of summarization and transformer architectures for profiling résumés
Bondielli, A., & Marcelloni, F. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.115521 · Cited by 18
Improving question answering performance using knowledge distillation and active learning
Boreshban, Y., Mirbostani, S. M., Ghassem-Sani, G., Mirroshandel, S. A., & Amiriparian, S. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106137 · Cited by 22
Character-level HyperNetworks for Hate Speech Detection
Wullach, T., Adler, A., & Minkov, E. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117571 · Cited by 18
Transliteration based Generative Pre-trained Transformer 2 Model for Tamil Text Summarization
Dhivyaa, C. R., Nithya, K., Janani, T., et al. (2022) · International Conference on Computer Communication and Informatics (ICCCI) · doi:10.1109/iccci54379.2022.9740991 · Cited by 15
IsoScore: Measuring the Uniformity of Embedding Space Utilization
Rudman, W., Gillman, N., Rayne, T., & Eickhoff, C. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.262 · Cited by 16
Extracting Decision Model and Notation models from text using deep learning techniques
Goossens, A., Smedt, J. D., & Vanthienen, J. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118667 · Cited by 22
Akca, O., Bayrak, G., Issifu, A. M., & Ganiz, M. C. (2022) · International Conference on Innovations in Intelligent Systems and Applications (INISTA) · doi:10.1109/inista55318.2022.9894051 · Cited by 20
Textual Pre-Trained Models for Gender Identification Across Community Question-Answering Members
Schwarzenberg, P., & Figueroa, A. (2023) · IEEE Access · doi:10.1109/access.2023.3235735 · Cited by 17
Evaluating Embedding APIs for Information Retrieval
Kamalloo, E., Zhang, X., Ogundepo, O., Thakur, N., Alfonso-Hermelo, D., Rezagholizadeh, M., & Lin, J. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-industry.50 · Cited by 24
FORK: A Bite-Sized Test Set for Probing Culinary Cultural Biases in Commonsense Reasoning Models
Palta, S., & Rudinger, R. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.631 · Cited by 16
Remy, F., Demuynck, K., & Demeester, T. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.104 · Cited by 23
Zhang, S., Zheng, X., Li, G., Yang, C., Li, Y., Wang, Y., Chao, F., Wang, M., Li, S., & Ji, R. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128140 · Cited by 23
The Hitchhiker’s Guide to Computational Linguistics in Suicide Prevention
Ophir, Y., Tikochinski, R., Klomek, A. B., & Reichart, R. (2021) · Clinical Psychological Science · doi:10.1177/21677026211022013 · Cited by 16
sigmoidF1: A Smooth F1 Score Surrogate Loss for Multilabel ClassificationPreprint
Bénédict, G., Koops, H. V., Odijk, D., & de Rijke, M. (2021) · UvA-DARE (University of Amsterdam) · doi:10.48550/arxiv.2108.10566 · Cited by 32
A comparative study on the impact of model compression techniques on fairness in language models
Ramesh, K., Chavan, A., Pandit, S., & Sitaram, S. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.878 · Cited by 15
Automatic Mixed-Precision Quantization Search of BERT
Zhao, C., Hua, T., Shen, Y., Lou, Q., & Jin, H. (2021) · Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/472 · Cited by 16
Improving Healthcare Outcomes with Learning Models for Machine Commonsense Reasoning Systems
Saeedi, S., Fong, A. C. M., Gupta, A. K., & Carr, S. (2022) · IEEE International Conference on Recent Advances in Systems Science and Engineering (RASSE) · doi:10.1109/rasse54974.2022.10019733 · Cited by 14
Active in-context learning for cross-domain entity resolution
Zhang, Z., Zeng, W., Tang, J., Huang, H., & Zhao, X. H. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102816 · Cited by 16
Mishra, C., Sarma, H., & Saravanan, M. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.104045 · Cited by 17
Structural failure risk assessment of shield tunnel using large language model
Njock, P. G. A., Yin, Z., Xu, H.-R., & Zhang, N. (2025) · Tunnelling and Underground Space Technology · doi:10.1016/j.tust.2025.106882 · Cited by 15
FLSys: Toward an Open Ecosystem for Federated Learning Mobile Apps
Jiang, X., Hu, H., On, T., Lai, P., Mayyuri, V. D., Chen, A., Shila, D. M., Larmuseau, A., Jin, R., Borcea, C., et al. (2022) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2022.3223578 · Cited by 15
GAML-BERT: Improving BERT Early Exiting by Gradient Aligned Mutual Learning
Zhu, W., Wang, X., Ni, Y., & Xie, G. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.242 · Cited by 15
BLINK with Elasticsearch for Efficient Entity Linking in Business Conversations
Laskar, M. T. R., Chen, C., Martsinovich, A., Johnston, J., Fu, X.-Y., Shashi Bhushan, T. N., & Corston-Oliver, S. H. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-industry.38 · Cited by 15
Automated Detection of Typed Links in Issue Trackers
Lüders, C. M., Pietz, T., & Maalej, W. (2022) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re54965.2022.00010 · Cited by 15
Narvala, H., McDonald, G., & Ounis, I. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103274 · Cited by 19
Where did you tweet from? Inferring the origin locations of tweets based on contextual information
Lamsal, R., Harwood, A., & Rodriguez, M. A. (2022) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata55660.2022.10020460 · Cited by 17
SKDBERT: Compressing BERT via Stochastic Knowledge Distillation
Ding, Z., Guoqing, J., Zhang, S., Guo, L., & Lin, W. T. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i6.25902 · Cited by 14
H-AES: Towards Automated Essay Scoring for Hindi
Singh, S., Pupneja, A., Mital, S., Shah, C., Bawkar, M., Gupta, L. P., Kumar, A., Kumar, Y., Gupta, R., & Shah, R. R. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i13.26894 · Cited by 16
Text-based predictions of COVID-19 diagnosis from self-reported chemosensory descriptions
Li, H., Gerkin, R. C., Bakke, A. J., Norel, R., Cecchi, G., Laudamiel, C., Niv, M. Y., Ohla, K., Hayes, J. E., Parma, V., et al. (2023) · Communications Medicine · doi:10.1038/s43856-023-00334-5 · Cited by 15
On the Difference of BERT-style and CLIP-style Text Encoders
Chen, Z., Chen, G., Diao, S., Wan, X., & Wang, B. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.866 · Cited by 18
A Dataset for Hyper-Relational Extraction and a Cube-Filling Approach
Chia, Y. K., Bing, L., Aljunied, S. M., Si, L., & Poria, S. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.688 · Cited by 17
Mosca, E., Abdalla, M. H. I., Basso, P., Musumeci, M., & Groh, G. (2023) · Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023) · doi:10.18653/v1/2023.trustnlp-1.17 · Cited by 14
FedBFPT: An Efficient Federated Learning Framework for Bert Further Pre-training
Wang, X., Li, H., Chen, K., & Shou, L. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/483 · Cited by 14
Towards Fast Adaptation of Pretrained Contrastive Models for Multi-channel Video-Language Retrieval
Lin, X., Tiwari, S., Huang, S., Li, M., Shou, M. Z., Ji, H., & Chang, S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01426 · Cited by 15
Learning experience assessment through players chat content in multiplayer online games
Rezapour, M. M., Fatemi, A., & Nematbakhsh, M. A. (2023) · Computers in Human Behavior · doi:10.1016/j.chb.2023.108003 · Cited by 16
A Comparative Study of Transformer-Based Neural Text Representation Techniques on Bug Triaging
Dipongkor, A. K., & Moran, K. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00217 · Cited by 19
Burger, C., Chen, L., & Le, T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.792 · Cited by 15
MCC-KD: Multi-CoT Consistent Knowledge Distillation
Chen, H., Wu, S., Quan, X., Wang, R., Yan, M., & Zhang, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.454 · Cited by 36
Chakraborty, M., Tonmoy, S. M. T. I., Zaman, S. M. M., Gautam, S., Kumar, T., Sharma, K., Barman, N. R., Gupta, C., Jain, V., Chadha, A. R., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.136 · Cited by 15
Vonitsanos, G., Kanavos, A., & Phivos Mylonas, B. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386655 · Cited by 16
Graph Receptive Transformer Encoder for Text Classification
Aras, A. C., Alikaşifoğlu, T., & Koç, A. (2024) · IEEE Transactions on Signal and Information Processing over Networks · doi:10.1109/tsipn.2024.3380362 · Cited by 14
Unsupervised Layer-Wise Score Aggregation for Textual OOD Detection
Darrin, M., Staerman, G., Gomes, E. D. C., Cheung, J. C. K., Piantanida, P., & Colombo, P. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29742 · Cited by 17
Li, Y. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109369 · Cited by 15
Natural Language Processing and Deep Learning for Bankruptcy Prediction: An End-to-End Architecture
Lombardo, G., Bertogalli, A., Consoli, S., & Recupero, D. R. (2024) · IEEE Access · doi:10.1109/access.2024.3476621 · Cited by 16
Real-Time Email Phishing Detection Using a Custom DistilBERT Model
Damatie, E. M., Eleyan, A., & Bejaoui, T. (2024) · International Symposium on Networks, Computers and Communications (ISNCC) · doi:10.1109/isncc62547.2024.10759011 · Cited by 14
Leveraging sentiment analysis via text mining to improve customer satisfaction in UK banks
Ghadiridehkordi, A., Shao, J., Boojihawon, D. K., Wang, Q., & Li, H. (2024) · International Journal of Bank Marketing · doi:10.1108/ijbm-05-2024-0323 · Cited by 17
Distilled BERT Model in Natural Language Processing
Vakili, Y. Z., Fallah, A., & Sajedi, H. (2024) · International Conference on Computer and Knowledge Engineering · doi:10.1109/iccke65377.2024.10874673 · Cited by 16
BERT-Based Sentiment Analysis Using Distillation
Lehečka, J., Švec, J., Ircing, P., & Šmídl, L. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-59430-5_5 · Cited by 14
Answering Binary Causal Questions: A Transfer Learning Based Approach
Kayesh, H., Islam, M. S., Wang, J., Anirban, S., Kayes, A. S. M., & Watters, P. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9207662 · Cited by 14
Neural Mask Generator: Learning to Generate Adaptive Word Maskings for Language Model Adaptation
Kang, M., Han, M., & Hwang, S. J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.493 · Cited by 15
Ask2Transformers: Zero-Shot Domain labelling with Pretrained Language Models
Sainz, O., & Rigau, G. (2021) · Global WordNet Conference (GWC) · doi:10.18653/v1/2021.gwc-1.6 · Cited by 17
CHA: A Caching Framework for Home-based Voice Assistant Systems
Xu, L., Iyengar, A., & Shi, W. (2020) · IEEE/ACM Symposium on Edge Computing (SEC) · doi:10.1109/sec50012.2020.00041 · Cited by 14
Linguistic Dependencies and Statistical Dependence
Hoover, J. L., Du, W., Sordoni, A., & O’Donnell, T. J. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.234 · Cited by 15
DOSA: Dravidian Code-Mixed Offensive Span Identification Dataset
Ravikiran, M., & Annamalai, S. (2021) · Cited by 14
Graph-based Multilingual Product Retrieval in E-Commerce Search
Lu, H., Hu, Y., Zhao, T., Wu, T., Song, Y., & Yin, B. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-industry.19 · Cited by 14
Making Punctuation Restoration Robust and Fast with Multi-Task Learning and Knowledge Distillation
Hentschel, M., Tsunoo, E., & Okuda, T. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414518 · Cited by 15
Distribution Matching for Rationalization
Huang, Y., Chen, Y., Du, Y., & Yang, Z. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17547 · Cited by 14
Attention Temperature Matters in Abstractive Summarization Distillation
Zhang, S., Zhang, X., Bao, H., & Wei, F. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.11 · Cited by 15
Exploring the Representation of Word Meanings in Context: A Case Study on Homonymy and Synonymy
García, M. (2021) · doi:10.18653/v1/2021.acl-long.281 · Cited by 18
Improving task-agnostic BERT distillation with layer mapping search
Jiao, X., Chang, H., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F., & Liu, Q. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.07.050 · Cited by 14
Leveraging Pre-Trained Language Model for Speech Sentiment Analysis
Shon, S., Brusco, P., Pan, J. S., Han, K. J., & Watanabe, S. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1723 · Cited by 19
Does Commonsense help in detecting Sarcasm?
Chowdhury, S. B. R., & Chaturvedi, S. (2021) · doi:10.18653/v1/2021.insights-1.2 · Cited by 14
LMdiff: A Visual Diff Tool to Compare Language Models
Strobelt, H., Hoover, B., Satyanaryan, A., & Gehrmann, S. (2021) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2021.emnlp-demo.12 · Cited by 23
Identifying and Categorizing Malicious Content on Paste Sites: A Neural Topic Modeling Approach
Vahedi, T., Ampel, B., Samtani, S., & Chen, H. (2021) · IEEE International Conference on Intelligence and Security Informatics (ISI) · doi:10.1109/isi53945.2021.9624765 · Cited by 15
Myers-Briggs personality classification from social media text using pre-trained language models
Santos, V. D., & Paraboni, I. (2022) · JUCS - Journal of Universal Computer Science · doi:10.3897/jucs.70941 · Cited by 20
An Automatic and Efficient BERT Pruning for Edge AI Systems
Huang, S., Liu, N., Liang, Y., Peng, H., Li, H., Xu, D., Xie, M., & Ding, C. (2022) · International Symposium on Quality Electronic Design (ISQED) · doi:10.1109/isqed54688.2022.9806197 · Cited by 14
Siamese BERT-Based Model for Web Search Relevance Ranking Evaluated on a New Czech Dataset
Kocián, M., Náplava, J., Stancl, D., & Kadlec, V. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21502 · Cited by 7
Framework for Topic Modeling using BERT, LDA and K-Means
Sethia, K., Saxena, M., Goyal, M., & Yadav, R. K. (2022) · International Conference on Advance Computing and Innovative Technologies in Engineering (ICACITE) · doi:10.1109/icacite53722.2022.9823442 · Cited by 16
Debiasing pretrained text encoders by paying attention to paying attention
Gaci, Y., Benatallah, B., Casati, F., & Benabdeslem, K. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.651 · Cited by 14
Resident Operative Autonomy and Attending Verbal Feedback Differ by Resident and Attending Gender
Filiberto, A. C., Abbott, K. L., Shickel, B., George, B. C., Cochran, A. L., Sarosi, G. A., Upchurch, G. R., & Loftus, T. J. (2023) · Annals of Surgery Open · doi:10.1097/as9.0000000000000256 · Cited by 16
CASSED: Context-based Approach for Structured Sensitive Data Detection
Kužina, V., Petric, A.-M., Barišić, M., & Jović, A. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.119924 · Cited by 16
A Federated Approach for Hate Speech Detection
Gala, J. P., Gandhi, D., Mehta, J., & Talat, Z. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.237 · Cited by 17
Fatemi, B., Rabbi, F., & Opdahl, A. L. (2023) · IEEE Access · doi:10.1109/access.2023.3345414 · Cited by 18
Generating Fact Checking Explanations
Atanasova, P. (2020) · Springer eBooks · doi:10.1007/978-3-031-51518-7_4 · Cited by 14
Enhancing the Identification of Cyberbullying through Participant Roles
Rathnayake, G., Atapattu, T., Herath, M., Zhang, G., & Falkner, K. E. K. (2020) · Workshop on Online Abuse and Harms (WOAH) · doi:10.18653/v1/2020.alw-1.11 · Cited by 18
HULK: An Energy Efficiency Benchmark Platform for Responsible Natural Language ProcessingPreprint
Zhou, X., Chen, Z., Jin, X., & Wang, W. Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.05829 · Cited by 20
Lu, J., & Namee, B. M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.13138 · Cited by 21
Better Distractions: Transformer-based Distractor Generation and Multiple Choice Question FilteringPreprint
Offerijns, J., Verberne, S., & Verhoef, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.09598 · Cited by 32
KDLSQ-BERT: A Quantized Bert Combining Knowledge Distillation with Learned Step Size QuantizationPreprint
Jin, J., Liang, C., Wu, T., Zou, L., & Gan, Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.05938 · Cited by 30
Rissanen Data Analysis: Examining Dataset Characteristics via Description LengthPreprint
Perez, E., Kiela, D., & Cho, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.03872 · Cited by 13
AraCOVID19-MFH: Arabic COVID-19 Multi-label Fake News and Hate Speech Detection DatasetPreprint
Ameur, M. S. H., & Aliane, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.03143 · Cited by 14
A Systematic Review and Implementation Guidelines of Multimodal Foundation Models in Medical ImagingPreprint
Huang, S.-C., Jensen, M., Yeung, S. Y., Lungren, M. P., Poon, H., & Chaudhari, A. S. (2025) · Research Square · doi:10.21203/rs.3.rs-5537908/v1 · Cited by 17
Modeling Legal Reasoning: LM Annotation at the Edge of Human Agreement
Thalken, R., Stiglitz, E., Mimno, D., & Wilkens, M. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.575 · Cited by 13
Investigating Educational and Noneducational Answer Selection for Educational Question Generation
Steuer, T., Filighera, A., & Tregel, T. (2022) · IEEE Access · doi:10.1109/access.2022.3180838 · Cited by 13
“Additionally, we investigate DistilBERT [16], which has a magnitude fewer parameters than other BERT architectures but has nevertheless achieved similar results on a variety of tasks [16].”
MethodResultA novel framework for aspect based sentiment analysis using a hybrid BERT (HybBERT) model
Goud, A., & Garg, B. R. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-17647-1 · Cited by 17
SemEval-2021 Task 4: Reading Comprehension of Abstract Meaning
Zheng, B., Yang, X., Ruan, Y.-P., Ling, Z.-H., Liu, Q., Wei, S., & Zhu, X. (2021) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2021.semeval-1.4 · Cited by 15
Ön Eğitimli Dil Modelleri Kullanarak Türkçe Tweetlerden Cinsiyet Tespiti
İlhami, S. E. L., & Hanbay, D. (2021) · Fırat Üniversitesi Mühendislik Bilimleri Dergisi · doi:10.35234/fumbd.929133 · Cited by 13
Tiered Reasoning for Intuitive Physics: Toward Verifiable Commonsense Language Understanding
Storks, S., Gao, Q., Zhang, Y., & Chai, J. Y. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.422 · Cited by 20
Sentiment Analysis using Machine Learning and Deep Learning Models on Movies Reviews
Eid, Y., & Asal, W. M. (2021) · Novel Intelligent and Leading Emerging Sciences Conference (NILES) · doi:10.1109/niles53778.2021.9600548 · Cited by 14
Efficient Unsupervised Sentence Compression by Fine-tuning Transformers with Reinforcement Learning
Ghalandari, D. G., Hokamp, C., & Ifrim, G. (2022) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.18653/v1/2022.acl-long.90 · Cited by 13
Modarressi, A., Fayyaz, M., Yaghoobzadeh, Y., & Pilehvar, M. T. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.19 · Cited by 20
Effective End-to-End Vision Language Pretraining With Semantic Visual Loss
Yang, X., Liu, F., & Lin, G. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3237166 · Cited by 15
Target Sound Extraction with Variable Cross-Modality Clues
Li, C., Qian, Y., Chen, Z., et al. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095266 · Cited by 13
NCL: Textual Backdoor Defense Using Noise-Augmented Contrastive Learning
Zhai, S., Shen, Q., Chen, X., Wang, W., Li, C., Fang, Y., & Wu, Z. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095007 · Cited by 15
HyperMixer: An MLP-based Low Cost Alternative to Transformers
Mai, F., Pannatier, A., Fehr, F., Chen, H., Marelli, F., Fleuret, F., & Henderson, J. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.871 · Cited by 17
Beermann, J., Benfer, R., Both, M., Müller, J. M., & Diedrich, C. (2023) · IEEE International Conference on Industrial Informatics (INDIN) · doi:10.1109/indin51400.2023.10218154 · Cited by 14
Automating Intended Target Identification for Paraphasias in Discourse Using a Large Language Model
Salem, A. C., Gale, R. C., Fleegle, M., Fergadiotis, G., & Bedrick, S. D. (2023) · Journal of Speech Language and Hearing Research · doi:10.1044/2023_jslhr-23-00121 · Cited by 13
Metadata
- Authors
- Sanh, V., Debut, L., Chaumond, J., & Wolf, T.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1910.01108
- OpenAlex
- W2978017171
- Cited by
- 4,653
- References
- 18
- Citations by year
- 2026: 67 · 2025: 443 · 2024: 801 · 2023: 1136 · 2022: 736 · 2021: 968 · 2020: 486 · 2019: 15
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Transactions on Graph Data and Knowledge (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- DistilBERT, knowledge distillation, BERT, pre-training, model compression, on-device computation, language modeling, transfer learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-23, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-23 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
