The DOI matches its OpenAlex record; 56 of the 56 references it cites were found in a registry or an index.
DeBERTa: Decoding-enhanced BERT with Disentangled Attention
Open accessOverview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 1×
“1 [31], SQuAD v2.”
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342
Cited by
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 13,453
Self-citationEvaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
“A common approach to adjusting the behavior of Trans-formers is to fine-tune large pre-trained models with curated or human-generated datasets of the desired behavior (e.g., Raffel et al. (2020); He et al. (2020)).”
BackgroundLin, T., Wang, Y., Liu, X., & Qiu, X. (2022) · AI Open · doi:10.1016/j.aiopen.2022.10.001 · Cited by 1,422
Self-citationGeometry-enhanced molecular representation learning for property prediction
Fang, X., Liu, L., Lei, J., He, D., Zhang, S., Zhou, J., Wang, F., Wu, H., & Wang, H. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-021-00438-4 · Cited by 597
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352
Conditional Positional Encodings for Vision Transformers
Chu, X., Tian, Z., Zhang, B., Wang, X., & Shen, C. (2021) · arXiv · doi:10.48550/arxiv.2102.10882 · Cited by 489
“The relative positional encoding is further improved in XLNet (Yang et al., 2019b) and DeBERTa (He et al., 2020), showing better performance.”
BackgroundMemory in the Age of AI Agents
Hu, Y., Liu, S., Yue, Y., Zhang, G., Liu, B., Zhu, F., Lin, J., Guo, H., Dou, S., Xi, Z., et al. (2025) · arXiv · doi:10.48550/arxiv.2512.13564 · Cited by 389
Large language models (LLMs): survey, technical frameworks, and future challenges
Kumar, P. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10888-y · Cited by 395
ChatGPT in healthcare: A taxonomy and systematic review
Li, J., Dada, A., Puladi, B., Kleesiek, J., & Egger, J. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108013 · Cited by 452
A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335
Revisiting Stereo Depth Estimation From a Sequence-to-Sequence Perspective with Transformers
Li, Z., Liu, X., Drenkow, N., Ding, A., Creighton, F. X., Taylor, R. H., & Unberath, M. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00614 · Cited by 335
Self-citationMeasuring Mathematical Problem Solving With the MATH Dataset
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., & Steinhardt, J. (2021) · arXiv · doi:10.48550/arxiv.2103.03874 · Cited by 306
polyBERT: a chemical language model to enable fully machine-driven ultrafast polymer informatics
Kuenneth, C., & Ramprasad, R. (2023) · Nature Communications · doi:10.1038/s41467-023-39868-6 · Cited by 275
P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks
Liu, X., Ji, K., Fu, Y., Tam, W. L., Du, Z., Yang, Z., & Tang, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.07602 · Cited by 254
Self-citationA Survey on Text Classification Algorithms: From Text to Predictions
Gasparetto, A., Marcuzzo, M., Zangari, A., & Albarelli, A. (2022) · Information · doi:10.3390/info13020083 · Cited by 253
A Survey of Large Language Models in Medicine: Progress, Application, and Challenge
Zhou, H., Liu, F., Gu, B., Zou, X., Huang, J., Wu, J., Li, Y., Chen, S. S., Zhou, P., Liu, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05112 · Cited by 240
“…laws” [37, 38] predict how much improvement can be expected in a model’s performance as BERT[34] 110M/340M 3.3B tokens ERNIE [39] 110M 173M sentences ALBERT[40] 12M/18M/60M/235M 16GB ELECTRA[41] 14M/110M/335M 33B tokens RoBERTa[42, 43] 123M/355M 161GB DeBERTa[44] 1.5B 160GB Decoder-only(Sec.”
BackgroundMethodWARP: Word-level Adversarial ReProgramming
Hambardzumyan, K., Khachatrian, H., & May, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.381 · Cited by 243
Exploring the Potential of Large Language Models (LLMs)in Learning on Graphs
Chen, Z., Mao, H., Li, H., Jin, W., Wen, H., Wei, X., Wang, S., Yin, D., Fan, W., Liu, H., et al. (2024) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3655103.3655110 · Cited by 209
“The embedding generated by LLMs will be adopted as the initial features of GNNs.”
BackgroundMethodScaling Self-Supervised Learning for Histopathology with Masked Image Modeling
Filiot, A., Ghermi, R., Olivier, A., Jacob, P., Fidon, L., Camara, A., Mac Kain, A., Saillard, C., & Schiratti, J.-B. (2023) · doi:10.1101/2023.07.21.23292757 · Cited by 148
Eight Things to Know about Large Language Models
Bowman, S. R. (2023) · arXiv · doi:10.48550/arxiv.2304.00612 · Cited by 137
Fang, X., Wang, F., Liu, L., He, J., Lin, D., Xiang, Y., Zhu, K., Zhang, X., Wu, H., Li, H., et al. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00721-6 · Cited by 132
What Artificial Neural Networks Can Tell Us about Human Language Acquisition
Warstadt, A., & Bowman, S. R. (2022) · Algebraic Structures in Natural Language · doi:10.1201/9781003205388-2 · Cited by 111
Wang, F., Zhang, Z., Zhang, X., Wu, Z., Mo, T., Lu, Q., Wang, W., Li, R., Xu, J., Tang, X., et al. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3768165 · Cited by 106
Pellert, M., Lechner, C. M., Wagner, C., Rammstedt, B., & Strohmaier, M. (2024) · Perspectives on Psychological Science · doi:10.1177/17456916231214460 · Cited by 97
Application of large language models in medicine
Liu, F., Zhou, H., Gu, B., Zou, X., Huang, J., Wu, J., Li, Y., Chen, S. S., Hua, Y., Zhou, P., et al. (2025) · Nature Reviews Bioengineering · doi:10.1038/s44222-025-00279-5 · Cited by 94
From Word Embeddings to Pre-Trained Language Models: A State-of-the-Art Walkthrough
Mars, M. (2022) · Applied Sciences · doi:10.3390/app12178805 · Cited by 91
A review of LLMs and their applications in the architecture, engineering and construction industry
Kampelopoulos, D., Tsanousa, A., Vrochidis, S., & Kompatsiaris, I. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11241-7 · Cited by 69
Kim, J., Suh, S., Chilton, L. B., & Xia, H. (2023) · Proceedings of the 2023 ACM Designing Interactive Systems Conference · doi:10.1145/3563657.3595996 · Cited by 61
“To determine which model would be appropriate for generating extended metaphors, we compared four representative transformers for text generation [41, 47, 71], i.e., RoBERTa [48], XLNet [80], GPT-2 [55], and GPT-3 (text-davinci-002) [18].”
MethodLength-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
Kim, G., & Cho, K. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.508 · Cited by 59
“Pre-trained language models (Peters et al., 2018; Devlin et al., 2018; Radford et al., 2019; Yang et al., 2019; He et al., 2020) have achieved notable improvements in various natural language processing (NLP) tasks.”
BackgroundKuhn, L., Gal, Y., & Farquhar, S. (2023) · arXiv · doi:10.48550/arxiv.2302.09664 · Cited by 700
“In our case, we use a Deberta-large model (He et al., 2020a) that is fine-tuned on the NLI data set MNLI (Williams et al., 2017).”
MethodA Length-Extrapolatable Transformer
Sun, Y., Dong, L., Patra, B., Ma, S., Huang, S., Benhaim, A., Chaudhary, V., Song, X., & Wei, F. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.816 · Cited by 59
Yu, Z., Peng, C., Yang, X., Dang, C., Adekkanattu, P., Gopal Patra, B., Peng, Y., Pathak, J., Wilson, D. L., Chang, C.-Y., et al. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104642 · Cited by 37
Hanley, H. W. A., & Durumeric, Z. (2024) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v18i1.31333 · Cited by 36
Hanley, H. W. A., & Durumeric, Z. (2023) · arXiv · doi:10.48550/arxiv.2305.09820 · Cited by 36
“To do this, we utilize data from 19 models, as well as adversarial data from article perturbation/re-writes and paraphrases, to train a DeBERTa-based model (He et al. 2021) to detect English-language synthetic news articles.”
MethodGubelmann, R. (2024) · Philosophy & Technology · doi:10.1007/s13347-024-00696-1 · Cited by 36
Multimodal Emotion Recognition and Sentiment Analysis via Attention Enhanced Recurrent Model
Sun, L., Xu, M., Lian, Z., Liu, B., Tao, J., Wang, M., & Cheng, Y. (2021) · Proceedings of the 2nd on Multimodal Sentiment Analysis Challenge · doi:10.1145/3475957.3484456 · Cited by 36
Exploring Platform Migration Patterns between Twitter and Mastodon: A User Behavior Study
Jeong, U., Sheth, P., Tahir, A., Alatawi, F., Bernard, H. R., & Liu, H. (2024) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v18i1.31348 · Cited by 34
Investigating the Limitations of Transformers with Simple Arithmetic Tasks
Nogueira, R., Jiang, Z., & Lin, J. (2021) · arXiv · doi:10.48550/arxiv.2102.13019 · Cited by 29
Socratic Questioning of Novice Debuggers: A Benchmark Dataset and Preliminary Evaluations
Al-Hossami, E., Bunescu, R., Teehan, R., Powell, L., Mahajan, K., & Dorodchi, M. (2023) · Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023) · doi:10.18653/v1/2023.bea-1.57 · Cited by 23
Comparing Test Sets with Item Response Theory
Vania, C., Htut, P. M., Huang, W., Mungra, D., Pang, R. Y., Phang, J., Liu, H., Cho, K., & Bowman, S. R. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.92 · Cited by 18
Measuring and Mitigating Gender Bias in Legal Contextualized Language Models
Bozdag, M., Sevim, N., & Koç, A. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3628602 · Cited by 15
Adaptation with Self-Evaluation to Improve Selective Prediction in LLMs
Chen, J., Yoon, J., Ebrahimi, S., Arik, S., Pfister, T., & Jha, S. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.345 · Cited by 13
Large Language Models Can Self-Correct with Key Condition Verification
Wu, Z., Zeng, Q., Zhang, Z., Tan, Z., Shen, C., & Jiang, M. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.714 · Cited by 36
“Li et al. (2023) fine-tuned DeBERTa-v3-large (He et al., 2021) to predict the probability that the generated reasoning path leads to a correct answer.”
BackgroundRed Teaming with Mind Reading: White-Box Adversarial Policies Against RL Agents
Casper, S., Killian, T., Kreiman, G., & Hadfield-Menell, D. (2022) · arXiv · doi:10.48550/arxiv.2209.02167 · Cited by 10
Bigger is not always better: The importance of human-scale language modeling for psycholinguistics
Wilcox, E. G., Hu, M. Y., Mueller, A., Warstadt, A., Choshen, L., Zhuang, C., Williams, A., Cotterell, R., & Linzen, T. (2025) · Journal of Memory and Language · doi:10.1016/j.jml.2025.104650 · Cited by 9
Enhancing the Robustness of AI-Generated Text Detectors: A Survey
Liu, X., Li, Y., & Li, K. (2025) · Mathematics · doi:10.3390/math13132145 · Cited by 10
Self-citationPre-training with a rational approach for antibody sequence representation
Gao, X., Cao, C., He, C., & Lai, L. (2024) · Frontiers in Immunology · doi:10.3389/fimmu.2024.1468599 · Cited by 9
Zeng, Z., Liu, S., Sha, L., Li, Z., Yang, K., Liu, S., Gasevic, D., & Chen, G. (2024) · Proceedings of the Thirty-ThirdInternational Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2024/835 · Cited by 8
Lameiro, F., Dunagan, L., Card, D., Gilbert, E., & Haimson, O. (2025) · Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3715275.3732095 · Cited by 6
“We tested two models: a Logistic Regression model and a fine-tuned DeBERTa model [37].”
Anchor Points: Benchmarking Models with Much Fewer Examples
Vivek, R., Ethayarajh, K., Yang, D., & Kiela, D. (2023) · arXiv · doi:10.48550/arxiv.2309.08638 · Cited by 4
Cheng, Z., Zhou, L., Jiang, F., Wang, B., & Li, H. (2025) · Proceedings of the ACM on Web Conference 2025 · doi:10.1145/3696410.3714770 · Cited by 3
MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Models
Pandit, S., Xu, J., Hong, J., Wang, Z., Chen, T., Xu, K., & Ding, Y. (2025) · arXiv · doi:10.48550/arxiv.2502.14302 · Cited by 31
Peerify: Benchmarking Peer-Review Claim Verification
Daghighfarsoodeh, A., Ebrahimi, S., Ghorbanpour, A., Sadeghian, S., Cheraghi, R., Arabzadeh, N., & Bagheri, E. (2026) · arXiv · doi:10.48550/arxiv.2609.25046
“We tested this directly with three zero-shot entailment models trained on MultiNLI (Williams et al., 2018): RoBERTa-large-MNLI (Liu et al., 2019), BART-large-MNLI (Lewis et al., 2020a), and DeBERTa-large-MNLI (He et al., 2021).”
Re2: A Consistency-ensured Dataset for Full-stage Peer Review and Multi-turn Rebuttal Discussions
Zhang, D., Bao, Z., Du, S., Zhao, Z., Zhang, K., Bao, D., & Yang, Y. (2025) · arXiv · doi:10.48550/arxiv.2505.07920 · Cited by 26
VLSP 2025 Shared Task: Vietnamese Temporal Question Answering (TemporalQA)
Duc, P. T., Linh, H. M., Toan, L. N., & Huyen, N. T. M. (2026) · VNU Journal of Science: Computer Science and Communication Engineering · doi:10.25073/2588-1086/vnucsce.6505
SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoning
Miao, N., Teh, Y. W., & Rainforth, T. (2023) · arXiv · doi:10.48550/arxiv.2308.00436 · Cited by 18
Metadata
- Authors
- He, P., Liu, X., Gao, J., & Chen, W.
- Venue
- arXiv (Cornell University)
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2006.03654
- OpenAlex
- W3033187248
- Cited by
- 420
- References
- 56
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.