The DOI matches its DataCite record; 23 of the 23 references it cites were found in a registry or an index; 3 of the 4 sources its claims cite were checked for support.
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Open accessAbstract
As Transfer Learning from large-scale pre-trained models becomes more prevalent in Natural Language Processing (NLP), operating these large models in on-the-edge and/or under constrained computational training or inference budgets remains challenging. In this work, we propose a method to pre-train a smaller general-purpose language representation model, called DistilBERT, which can then be fine-tuned with good performances on a wide range of tasks like its larger counterparts. While most prior work investigated the use of distillation for building task-specific models, we leverage knowledge distillation during the pre-training phase and show that it is possible to reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster. To leverage the inductive biases learned by larger models during pre-training, we introduce a triple loss combining language modeling, distillation and cosine-distance losses. Our smaller, faster and lighter model is cheaper to pre-train and we demonstrate its capabilities for on-device computations in a proof-of-concept experiment and a comparative on-device study.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
DistilBERT is a distilled version of BERT pre-trained with knowledge distillation, reducing parameters by 40% while retaining 97% of BERT’s GLUE performance and running 60% faster at inference. It is presented as a general-purpose, lighter model suitable for constrained and on-device NLP applications.
Key findings
- DistilBERT has 66M parameters vs. BERT-base’s 110M and takes 410s vs. 668s for a CPU inference pass on STS-B, i.e. 40% smaller and 60% faster.
- On GLUE dev sets, DistilBERT scores 77.0 macro-score vs. BERT-base’s 79.5, retaining 97% of performance; it is always on par or improving over the ELMo baseline.
- On IMDb test accuracy, DistilBERT reaches 92.82 vs. BERT-base’s 93.46; on SQuAD v1.1, DistilBERT reaches 77.7/85.8 EM/F1 vs. BERT-base’s 81.2/88.5.
- Ablations show removing the cosine loss costs -2.96 GLUE macro-score, removing distillation loss costs -1.46, removing MLM loss costs -0.31, and random initialization costs -3.69.
Methods
- Knowledge distillation during pre-training: a student with BERT’s architecture but half the layers, no token-type embeddings and no pooler, initialized from every other teacher layer.
- Training objective is a triple loss combining distillation loss \(L_{ce}\), masked language modeling loss \(L_{mlm}\), and cosine embedding loss \(L_{cos}\), using softmax-temperature.
- Trained on English Wikipedia + Toronto Book Corpus with large batches up to 4K examples, dynamic masking, no next-sentence prediction, on 8 16GB V100 GPUs for approximately 90 hours; evaluated on GLUE, IMDb, SQuAD v1.1, CPU inference, and an iPhone 7 Plus on-device proof-of-concept.
Limitations
- The paper does not state an explicit limitations section.
- Evident from the text: evaluation is limited to English benchmarks and a small set of downstream tasks, and the on-device study is described as a proof-of-concept; the model could be further reduced with quantization.
- Evident from the text: the SQuAD second-distillation result is reported inconsistently—the prose says 79.8 F1 and 70.4 EM, while Table 2 lists DistilBERT (D) as 79.1/86.9.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
15 Own result · 6 Own interpretation · 2 Own method · 2 No basis given · 1 Supported · 1 Partly supported · 1 Sources not found · accuracy not yet measured
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we leverage knowledge distillation during the pre-training phase and show that it is possible to reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster.”
- Also stated in: § Size and inference speed · p. 4 · plain
“DistilBERT has 40% fewer parameters than BERT and is 60% faster than BERT.”
- Also stated in: § 6 Conclusion and future work · p. 5 · plain
“We introduced DistilBERT, a general-purpose pre-trained version of BERT, 40% smaller, 60% faster, that retains 97% of the language understanding capabilities.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Size and inference speed · p. 4 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we propose a method to pre-train a smaller generalpurpose language representation model, called DistilBERT, which can then be finetuned with good performances on a wide range of tasks like its larger counterparts.”
- Also stated in: § 1 Introduction · p. 2 · plain
“Our general-purpose pre-trained models can be fine-tuned with good performances on several downstream tasks, keeping the flexibility of larger models.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 Introduction · p. 2 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Further ablation studies indicate that all the components of the triple loss are important for best performances.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Excluding the tokenization step, DistilBERT is 71% faster than BERT, and the whole model weighs 207 MB (which could be further reduced with quantization).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we leverage knowledge distillation during the pre-training phase and show that it is possible to reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster.”
- Also stated in: § Size and inference speed · p. 4 · plain
“DistilBERT has 40% fewer parameters than BERT and is 60% faster than BERT.”
- Also stated in: § 6 Conclusion and future work · p. 5 · plain
“We introduced DistilBERT, a general-purpose pre-trained version of BERT, 40% smaller, 60% faster, that retains 97% of the language understanding capabilities.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Inference speed measured on a single GLUE task (STS-B) on CPU is representative of DistilBERT's speed advantage generally.
- Also stated in: § Size and inference speed · p. 4 · plain
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“we propose a method to pre-train a smaller generalpurpose language representation model, called DistilBERT”
- Also stated in: § 6 Conclusion and future work · p. 5 · plain
“We introduced DistilBERT, a general-purpose pre-trained version of BERT”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 6 Conclusion and future work · p. 5 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found it beneficial to add a cosine embedding loss ( L cos ) which will tend to align the directions of the student and teacher hidden states vectors.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On SQuAD, DistilBERT is within 3.9 points of the full BERT.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we leverage knowledge distillation during the pre-training phase and show that it is possible to reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster.”
- Also stated in: § 4 Experiments · p. 3 · plain
“DistilBERT also compares surprisingly well to BERT, retaining 97% of the performance with 40% fewer parameters.”
- Also stated in: § 6 Conclusion and future work · p. 5 · plain
“We introduced DistilBERT, a general-purpose pre-trained version of BERT, 40% smaller, 60% faster, that retains 97% of the language understanding capabilities.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 4 Experiments · p. 3 · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“removing the Masked Language Modeling loss has little impact while the two distillation losses account for a large portion of the performance.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 7
- It may state two separate claims as one. 2
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“environmental cost of exponentially scaling these models' computational requirements as mentioned in Schwartz et al. [2019], Strubell et al. [2019].”
What the sources say
- Supportedcites Schwartz et al., 2019 source
The Abstract (p. 1) says the computations required for deep learning research have been doubling every few months and have a large carbon footprint. Section 2, “Red AI” (p. 4), likewise warns that the environmental cost of increasingly large models will not decrease soon. (LLM, conf=0.93)
- Partly supportedcites Strubell et al., 2019 sourceExperimental — accuracy not yet measured
The Introduction (page unavailable) says that computationally hungry NLP models require considerable energy and have associated environmental costs. It also says model development can multiply those costs thousands of times, but it does not establish that the models’ computational requirements are scaling exponentially. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“We applied best practices for training BERT model recently proposed in Liu et al. [2019].”
What the sources say
- Supportedcites Liu et al., 2019 source
The Introduction (p. 1) says Liu et al. propose an improved recipe for training BERT, and Section 4 (p. 4) evaluates which choices matter for BERT pretraining. The citing authors’ statement that they applied those practices to DistilBERT describes their own work, not a finding attributed to Liu et al. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Among the 9 tasks, DistilBERT is always on par or improving over the ELMo baseline (up to 19 points of accuracy on STS-B).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In this case, we were able to reach interesting performances given the size of the model: 79.8 F1 and 70.4 EM, i.e. within 3 points of the full model.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We further demonstrated that DistilBERT is a compelling option for edge applications.”
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“BERT and DistilBERT results are the medians of 5 runs with different seeds.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“current research 1 on pre-trained models indicates that training even larger models still leads to better performances on downstream tasks.”
What the sources say
- Not matchedcites MegatronLM
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We report scores on the development sets for each task by fine-tuning DistilBERT without the use of ensembling or multi-tasking scheme for fine-tuning (which are mostly orthogonal to the present work).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“While most prior work investigated the use of distillation for building task-specific models, we leverage knowledge distillation during the pre-training phase”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
The paper relies on this without support.
Operating large pre-trained models on-device in real-time is hampered by their growing computational and memory requirements.
“the growing computational and memory requirements of these models may hamper wide adoption.”
The paper relies on this without support.
Near-zero probabilities in a trained model's output distribution reflect its generalization capabilities.
“some of these”
The paper relies on this without support.
Inference speed measured on a single GLUE task (STS-B) on CPU is representative of DistilBERT's speed advantage generally.
“Inference time of a full pass of GLUE task STS-B (sentiment analysis) on CPU with a batch size of 1.”
“DistilBERT has 40% fewer parameters than BERT and is 60% faster than BERT.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This idea has been successfully applied to LLMs, where distilled students approximate the generation patterns of larger teachers while retaining much of their performance (Sanh et al., 2019).”
From Vision-Guided Iterative Refinement for Frontend Code Generation (2026) · cites this paper as Sanh et al., 2019
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“even benchmarking is not a reliable estimator of parameter count [12], [15], [28], [29]”
From Hollow-LLM Attack: Computationally Trivial Weights in Zero-Knowledge Verification of LLM Inference (2026) · cites this paper as [29]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“It achieves similar performance to the original BERT model while using fewer resources. The approach leveraged knowledge distillation during the pretraining phase and reduced the size of general BERT by 40%, yet it retained a language understanding capability of 97%.”
From A Review on Deep-Learning-Based Cyberbullying Detection (2023) · cites this paper as [53]
The cited source says what the claim attributes to it.
In the citing paper
“Unlike in many similar circumstances, we do not employ knowledge distillation [56], because we discovered that for our sizes of the models, this technique does not provide any quality improvement.”
From Full Line Code Completion: Bringing AI to Desktop (2024) · cites this paper as [56]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 4.1 Downstream task benchmark · p. 3
“In this case, we were able to reach interesting performances given the size of the model: 79.8 F1 and 70.4 EM, i.e. within 3 points of the full model.”
§ 3 DistilBERT: a distilled version of BERT · p. 3
“| DistilBERT (D) | - | 79.1/86.9 |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“DistilBERT is always on par or improving over the ELMo baseline (up to 19 points of accuracy on STS-B).”
Reported 19 · recomputed 16.5
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
Self-citationSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222
Learning Word Vectors for Sentiment Analysis
Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., & Potts, C. (2011) · Cited by 3,710
Energy and Policy Considerations for Deep Learning in NLP
Strubell, E., Ganesh, A., & McCallum, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1355 · Cited by 3,684
Research works printed but not linked: 1
- Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In KDD , 2006.
Cited by
Sorted by citations
Items 4,401–4,500 of 4,692
Wu, Y., Ye, L., Li, B., Zhang, H., & Cao, J. (2023) · IEEE International Conference on Cloud Networking · doi:10.1109/cloudnet59005.2023.10490080
Towards resource-aware dialogue systems and sentiment analysis
Pandelea, V. (2024) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/173846
Sentiment Analysis of User Reactions to Meta's Threads Launch and Twitter's X Renaming
Sharma, A., Bali, R., Kumar, P., Nandana, G. M., & Mala, S. (2024) · Advances in business information systems and analytics book series · doi:10.4018/979-8-3693-1210-0.ch015 · Cited by 1
Wang, Z., Duan, J., Yuan, C., Chen, Q., Chen, T., Yao, H., Zhang, Y., Wang, R., Xu, K., & Shi, X. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4791792
Missing Evidence: Tracking Academic Data Use around the WorldPreprint
Stacy, B. W., Kitzmüller, L., Wang, X., et al. (2024) · Research Square · doi:10.21203/rs.3.rs-4219045/v1
Unmasking Disinformation: Detection of Fake News Online Using Learning Techniques
Shaikh, N. A., Petkar, P. B., & Sonawane, S. S. (2024) · International Conference on Emerging Smart Computing and Informatics (ESCI) · doi:10.1109/esci59607.2024.10497384
Performance Analysis of Suggestion Mining for Mobile AppsPreprint
Mali, M. (2024) · Research Square · doi:10.21203/rs.3.rs-4268476/v1
SOCIAL MEDIA ANALYSIS THROUGH SENTIMENT AND INFLUENTIAL NODE ANALYSIS
(2024) · International Research Journal of Modernization in Engineering Technology and Science · doi:10.56726/irjmets52544
Winatmoko, Y. A., & Septiandri, A. A. (2023) · Workshop on Financial Technology and Natural Language Processing (FinNLP) · doi:10.18653/v1/2023.finnlp-2.10
Zhou, B., & Huang, Y. (2024) · IEEE Advanced Information Technology, Electronic and Automation Control Conference (IAEAC) · doi:10.1109/iaeac59436.2024.10503735
Leveraging DistilBERT and BERT for the Detection of Online Sexism: A Comparative Analysis
Mitra, S., Kanungoe, P., & Chowdhury, A. N. (2023) · International Conference on Ubiquitous Computing and Intelligent Information Systems (ICUIS) · doi:10.1109/icuis60567.2023.00066
ANALYSIS OF MODERN MODELS OF CONTEXT-AWARE QUESTION-ANSWERING SYSTEMS
Vyshniak, M. Y., & Pyrozhenko, M. Y. (2024) · Collection of Scientific Publications NUS · doi:10.15589/znp2024.1(494).16
Factorized and progressive knowledge distillation for CTC-based ASR models
Tian, S., Li, Z., Lyv, Z., et al. (2024) · Speech Communication · doi:10.1016/j.specom.2024.103071
Towards human-level artificial intelligence agents
Leung, J. C. (2024) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/174532
Baghdasaryan, V. H. (2024) · International Journal Of Scientific Advances · doi:10.51542/ijscia.v5i2.11
Masked Language Modeling for Predicting Missing Words in Damaged Ancient Greek Texts
Riccardi, K., & Bandara, D. (2024) · doi:10.18260/1-2--45774
Jeyakodi, G., Balakrishnan, S., & Bala, P. S. (2024) · Salud Ciencia y Tecnología - Serie de Conferencias · doi:10.56294/sctconf2024758
ChatParse: A New Application for Persian Bourse Chatbot
Shahedi, A., & Seyedin, S. (2023) · Sharq. · doi:10.61186/itrc.15.2.49 · Cited by 1
Towards a Pre-Trained Question-Answering Language Model for Kinyarwanda
Tuyizere, D., Ihirwe, F., Ihabwikuzo, R., Blessed, G., & Luhanga, E. T. (2023) · IEEE International Conference on Signal, Control and Communication (SCC) · doi:10.1109/scc59637.2023.10527503
Cabada, R. Z., Estrada, M. L. B., Graff, M., Beltrán, V. M. B., & Sapien, R. A. C. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4836778
Iktidar, A., & Murad, H. (2024) · International Conference on Electrical Engineering and Information Communication Technology · doi:10.1109/iceeict62016.2024.10534556
Lama, S. K., Zhang, H., Monaghan, C. K., et al. (2024) · Nephrology Dialysis Transplantation · doi:10.1093/ndt/gfae069.792
Rumors detection on X using pre-trained language models
Meghatria, R., Belhenniche, H., & Ferrah, S. (2024) · International Conference on Image and Signal Processing and their Applications (ISPA) · doi:10.1109/ispa59904.2024.10536825
Empci: Empathetic Response Generation with Common Sense and IntentsPreprint
Wang, X., Liu, Z., Liu, T., et al. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4852335
Low-Complexity Vision Transformer with Adaptive Channel Partitioning Method
Lee, S., Oh, S.-S., Han, S.-H., Lim, S., Lee, J., & Sim, D. (2024) · Journal of Broadcast Engineering · doi:10.5909/jbe.2024.29.3.291
Enhancing Neural Network Efficiency with Streamlined Pruned Linear Adapters
Mao, C., Zhu, Z., Guo, Y., & Leng, T. (2024) · International Conference on Advanced Algorithms and Control Engineering (ICAACE) · doi:10.1109/icaace61206.2024.10549729
Different Approaches for Reading Comprehension of Abstract Meaning
Mihai, C. M., Mocan, C. M., Nandra, C. I., & Chifu, E. Ş. (2024) · International Convention on Information and Communication Technology, Electronics and Microelectronics · doi:10.1109/mipro60963.2024.10569822
Alphenaar, G., & Rafiq, R. I. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-63616-5_15
Martín-Fernández, I., Esteban-Romero, S., Gil-Martín, M., et al. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4888854
A cross-domain access control mechanism based on model migration and semantic reasoning
Tan, M., Liu, A., Wang, X., Shang, S., Wang, N., & Du, X. (2024) · KSII Transactions on Internet and Information Systems · doi:10.3837/tiis.2024.06.010
Pretraining an Encoder: The BERT Language Model
Nugues, P. (2024) · Cognitive technologies · doi:10.1007/978-3-031-57549-5_16
DisContNet: Contradiction Detection in Texts using Transformers
Kutty, R. J., Roshni, P. N., & Adiga, S. S. (2023) · International Conference on Smart Technologies in Computing, Electrical and Electronics (ICSTCEE) · doi:10.1109/icstcee60504.2023.10585189
A survey on short text similarity measurement methods
Zadeh, A. R., & Amirkhani, H. (2023) · Archives de l'Institut d'Hessarek. · doi:10.61186/jsdp.20.3.103
Alpdemir, Y., & Alpdemir, M. N. (2024) · IEEE International Conference on Advanced Systems and Emergent Technologies (IC_ASET) · doi:10.1109/ic_aset61847.2024.10596255
Atmani, H., & Moutacalli, M. T. (2024) · IEEE International Conference on Advanced Systems and Emergent Technologies (IC_ASET) · doi:10.1109/ic_aset61847.2024.10596207
THE BAT: Thoughts Hierarchical Enhancement Beyond Arbitrary Text Style Transfer
Zeng, B., Liang, J., Hua, Y., Li, R., Deng, H., Peng, Y., & Wang, R. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5672-8_32
Empirical approach to understanding natural language models
Bansal, B. (2024) · Journal of Student Research · doi:10.47611/jsrhs.v13i1.6103
Pu, X., Xu, X., & Yu, Y. (2024) · Sensors and Materials · doi:10.18494/sam5062
Not What I was Trained for – Out-of-Distribution-Tests for Interactive AIs
Severin, B., Werger, O., & Hesenius, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-59235-5_12
Hybrid Models for Recognizing Indonesian Textual Entailment
Ahmadian, H., Abidin, T. F., Riza, H., et al. (2024) · International Conference on Informatics and Computational Sciences (ICICoS) · doi:10.1109/icicos62600.2024.10636863
Extended ProMap datasets for product mapping
Macková, K., & Pilát, M. (2024) · Electronic Commerce Research · doi:10.1007/s10660-024-09892-9
Multi-Modal Learning with Joint Image-Text Embeddings and Decoder Networks
Chemmanam, A. J., Jose, B. A., & Moopan, A. (2024) · IEEE International Conference on Industrial Cyber-Physical Systems (ICPS) · doi:10.1109/icps59941.2024.10639946
Balouchzahi, F., Butt, S., Sarker, A., et al. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4930517
Incident Duration Prediction Fusing Chinese Text: An End-to-End Deep Learning FrameworkPreprint
Gao, P., Tan, E., Li, L., et al. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4948388
Enhancing Data Augmentation with Knowledge-enriched Data Generation via Dynamic Prompt-tuning Method
Qi, Q., Bao, Q., Peng, A. Y., Liu, J., & Witbrock, M. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10651072
From Static to Dynamic: A Deeper, Faster, and Adaptive Language Modeling Approach
Li, J., Li, Q., & Wang, P. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650050
LRER: A Low-Resource Entity Resolution Framework with Hybrid Information
Lian, Z., Tie-zheng, N., Shen, D., & Kou, Y. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10651166
Information Extraction to Identify Novel Technologies and Trends in Renewable Energy
Maclean, C., & Cavallucci, D. (2024) · IFIP advances in information and communication technology · doi:10.1007/978-3-031-75923-9_22
Dataset Expansion with Pseudo-Labeling: Case Study for Optimizing Chatbot Intent Recognition
Kuligowska, K., & Kowalczuk, B. (2024) · Humanities & Social Sciences Reviews · doi:10.18510/hssr.2024.12214
Cong, W., Liang, H., Fan, Z., Wang, P., Jiang, Y., Xu, D., Öztireli, C., & Wang, Z. (2024) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw63382.2024.00293
LinguaShrink:Reducing Token Overhead with Psycholinguistics
Ma, F., Liang, X., & Tao, M. (2024) · IEEE Access · doi:10.1109/access.2024.3470512
Gradient-guided Simulated Annealing for Insensitivity-based Text Attack
Li, C. C., Yang, X., Liu, B., & Liu, W. (2024) · doi:10.1109/icnlp60986.2024.10692983
Toward Improving Robustness of Coreference Resolution for Thai Language
Suwannapichat, P., Tarnpradab, S., & Prom–on, S. (2024) · doi:10.1109/icnlp60986.2024.10692844
High-Quality Keyword Refinement Techniques Considering the Domain Attributes of Keywords
Kim, M., & Jung, Y. (2024) · The Journal of Korean Institute of Information Technology · doi:10.14801/jkiit.2024.22.9.23
Investigação sobre a Leitura de Documentos de Modelos de Perguntas e Respostas no Domínio Esportivo
Camargos, L. F., Moraes, L. M. P., & Ciferri, C. D. D. A. (2024) · Simpósio Brasileiro de Bancos de Dados (SBBD) · doi:10.5753/sbbd_estendido.2024.243679
The Universal Module for Integration of an Intelligent Assistant into iOS Applications
Levchuk, V. Y. (2024) · Control Systems and Computers · doi:10.15407/csc.2024.03.053
Climate Change Skeptics and the Power of Negativity
Liu, Q., Kim, Y., & Hemsley, J. (2024) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.1166
Unveiling personality traits through Bangla speech using Morlet wavelet transformation and BiG
Sk., M. S. I., & Alam, M. G. R. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2024.100113 · Cited by 1
Deep Word Embedding and Language Model based Sentimental Analysis of COVID-19 Tweets
Qaiser, S., & Akram, M. U. (2023) · International Bhurban Conference on Applied Sciences and Technology (IBCAST) · doi:10.1109/ibcast59916.2023.10712912
SACMR: Sentiment Analysis in Chinese Language using Modified RoBERTa
Ma, Y., Zhan, B., Yu, J., & Yan, S. (2024) · International Conference on Computational Intelligence and Applications (ICCIA) · doi:10.1109/iccia62557.2024.10719112
Combining computational linguistics with sentence embedding to create a zero-shot NLIDB
Perezhohin, Y., Peres, F., & Castelli, M. (2024) · Array · doi:10.1016/j.array.2024.100368
Navigating the Roadblocks: Lessons Learned from a Gardening Society Conversational RAG Bot
Tamanna, Rana, S., & Choudhary, P. (2024) · IEEE World Conference on Applied Intelligence and Computing (AIC) · doi:10.1109/aic61668.2024.10731053 · Cited by 1
TAFE-AI: A Trust-Augmented Fog and Edge AI Enabled Network Integrated with Decentralized Identity
Kaleel, S. B., & Abhari, A. (2024) · Annual Modeling and Simulation Conference (ANNSIM) · doi:10.23919/annsim61499.2024.10732127
An Accurate Classification and Recommendation Method of Competitive Math Problems
Shao, Y. (2024) · IEEE International Conference on Big Data Computing Service and Applications (BigDataService) · doi:10.1109/bigdataservice62917.2024.00021 · Cited by 1
From Conversations to Knowledge: Enriching Movie Datasets with Knowledge Graph
Parmar, D., Nagrani, T., Babre, R., & Mazumdar, P. (2024) · International Conference on Computing, Communication and Networking Technologies (ICCCNT) · doi:10.1109/icccnt61001.2024.10723870
What Matters While Rating a Hotel by a User? An Indian Perspective
Dasgupta, S., Roy, S., & Sen, J. (2024) · International Conference on Computing, Communication and Networking Technologies (ICCCNT) · doi:10.1109/icccnt61001.2024.10724522
A short-dive into transformer based language models: A literature review
Matisons, R., Pirta-Dreimane, R., & Grabis, J. (2024) · International Scientific Conference on Information Technology and Management Science of Riga Technical University (ITMS) · doi:10.1109/itms64072.2024.10741950
Deep Learning based Poet Attribution model for Punjabi Poetry
Tariq, F., Gopchandani, R., Nizamani, R. H., Samad, A., & Anwar, M. M. (2024) · International Conference on Emerging Trends in Smart Technologies (ICETST) · doi:10.1109/icetst62952.2024.10737982
Senior Solutions Architect, D. S. D. B., & Akheel, S. A. (2024) · INTERANTIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT · doi:10.55041/ijsrem15118
Privacy-Decision Assisting Techniques
Kqiku, L. (2024) · eDiss (Georg-August-Universität Göttingen) · doi:10.53846/goediss-10852
Exploring Transformer Models and Domain Adaptation for Detecting Opinion Spam in Reviews
Harris, C. G. (2024) · Conference of Open Innovations Association (FRUCT) · doi:10.23919/fruct64283.2024.10749897
Tan, Z., Chen, Q., Zhu, W., Huang, Y., & Liang, C. (2024) · Neural computing & applications (Print) · doi:10.1007/s00521-024-10669-2
Position Papers of the 19th Conference on Computer Science and Intelligence Systems
(2024) · Annals of Computer Science and Information Systems · doi:10.15439/978-83-969601-9-1
Evaluating Language Model Vulnerability to Poisoning Attacks in Low-Resource Settings
Plant, R., Giuffrida, M. V., Pitropakis, N., & Gkatzia, D. (2024) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3507565 · Cited by 1
Second language learning of degree expressions: A computational approach
Cong, Y. (2024) · Natural language processing. · doi:10.1017/nlp.2024.56
Cross-loop contrast of heterogeneous graphs for interdisciplinary journal recommendation
Li, Y. C., Liang, Y., Li, Q., Sun, H., Du, W., Jiang, Y., & Ma, Q. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125949
ODSBAHIA-PTBR: A Natural Language Processing Model to Support Sustainable Development Goals
Santos, E. S., & Moraes, L. E. (2024) · Revista de Gestão Social e Ambiental · doi:10.24857/rgsa.v18n12-039 · Cited by 1
Bidirectional Collaborative Data Management
Ishihara, Y., Kato, H., Onizuka, M., Takeichi, M., & Xiao, C. (2024) · Springer eBooks · doi:10.1007/978-981-97-6429-7_4
Optimizing Sentiment Inference with Multi-Expert Models via Real-Time GPU Resource Monitoring
Huang, J., Meng, C., Wang, S., et al. (2024) · IEEE International Conference on Smart Internet of Things (SmartIoT) · doi:10.1109/smartiot62235.2024.00029
Language Conditioned Robotics: Current Trends and Future Directions – a Survey PaperPreprint
Ninov, N. N., & Asif, W. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.5073634
Customer Query Classification Based on DistilBERT and TextCNN
Satidkarn, T., & Imsombut, A. (2024) · doi:10.1109/incit63192.2024.10810613
Cao, Q., Dao-Hoang, P., Nguyen, D.-T., et al. (2024) · International Conference on Control, Automation and Information Sciences (ICCAIS) · doi:10.1109/iccais63750.2024.10814364
An Accurate Classification Method of Competitive Math Problems
Shao, Y. (2024) · Lecture notes on data engineering and communications technologies · doi:10.1007/978-981-97-9255-9_6
Koukal, F., Dařena, F., Ježdík, R., & Přichystal, J. (2024) · European Journal of Business Science and Technology · doi:10.11118/ejobsat.2024.010 · Cited by 3
Unraveling Customer Sentiment in Indian Fashion E-Commerce: A BERT-Based Approach
Taneja, O., Alpana, K. R. S., Rathod, D. C., R, A. F. M., & Vishalakshi Prabhu, H. (2024) · International Conference on Computation System and Information Technology for Sustainable Solutions (CSITSS) · doi:10.1109/csitss64042.2024.10816956 · Cited by 3
Fine-tuning Specialized NER Model for Symptom Extraction from Slovenian Medical Texts
Sallauka, R., Ariöz, U., & Mlakar, I. (2024) · International Conference on Mathematics and Computers in Sciences and Industry (MCSI) · doi:10.1109/mcsi63438.2024.00032
Measuring Personal Values Of African Reddit Users Using a Lexicon-Based Approach
Oulahbib, I., Benhaddi, M., & Hadaj, S. E. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-74491-4_24
-, D. B., -, K. S., -, A. B., & -, S. M. A. H. (2024) · International Journal For Multidisciplinary Research · doi:10.36948/ijfmr.2024.v06i06.32271
Comparison of Ensemble Methods for Sentiment Analysis in Serbian Language
Čikić, Đ., Đorđević, N., & Jankovic, D. S. (2024) · Telecommunications Forum (TELFOR) · doi:10.1109/telfor63250.2024.10819129
Depression Classification using Bert Embedding Model on Social Media Posts
Nareshkumar, R., & Nimala, K. (2025) · Journal of Machine and Computing · doi:10.53759/7669/jmc202505037
GanTextKnockoff: stealing text sentiment analysis model functionality using synthetic data
Pham, C., Hoang, T.-N., Tran, C.-T., & Do, V.-B. (2024) · Journal of Military Science and Technology · doi:10.54939/1859-1043.j.mst.csce8.2024.76-86
A Deep Learning Ensemble Model Based on ECG for the Diagnosis of Variant Angina
Ryu, J., Park, Y., & Yang, S. (2024) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm62325.2024.10821835
Advancing Trans-Domain Classification With Knowledge Distillation: Bridging LIDAR and Image Data
Ortíz, J. E., & Creixell, W. (2025) · IEEE Access · doi:10.1109/access.2025.3530445
TRUExT: Trustworthiness Regressor Unified Explainable Tool
Donner, C., Danala, G., Jentner, W., & Ebert, D. S. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825100
ForumPFN: Online Forum Post Fusion Network for Volatility Index Movement Prediction
Ueda, K., Suwa, H., Umehara, E., Ogawa, Y., Yamashita, T., Tsubouchi, K., & Yasumoto, K. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825644
Pedersen, M. G., Fazal, B. K., & Kim, K.-S. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825708
Factify: An Automated Fact-Checker for Web Information
Adesokan, A., & Elbassuoni, S. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825147
Multi-task Learning with Pre-trained Language Models for Mental Illness Screening
Shrestha, A., Tlachac, M., Flores, R., Hickey, K., & Rundensteiner, E. A. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825698
HAL 9000: a Risk Manager for ITSs
Freitas, T., Novo, C., Soares, J., et al. (2024) · IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA) · doi:10.1109/tps-isa62245.2024.00044
Metadata
- Authors
- Sanh, V., Debut, L., Chaumond, J., & Wolf, T.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2019
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1910.01108
- OpenAlex
- W2978017171
- Cited by
- 4,653
- References
- 18
- Citations by year
- 2026: 67 · 2025: 443 · 2024: 801 · 2023: 1136 · 2022: 736 · 2021: 968 · 2020: 486 · 2019: 15
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Transactions on Graph Data and Knowledge (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- DistilBERT, knowledge distillation, BERT, pre-training, model compression, on-device computation, language modeling, transfer learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-23, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-23 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
