The DOI matches its DataCite record; 48 of the 48 references it cites were found in a registry or an index.
ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
Open accessAbstract
Masked language modeling (MLM) pre-training methods such as BERT corrupt the input by replacing some tokens with [MASK] and then train a model to reconstruct the original tokens. While they produce good results when transferred to downstream NLP tasks, they generally require large amounts of compute to be effective. As an alternative, we propose a more sample-efficient pre-training task called replaced token detection. Instead of masking the input, our approach corrupts it by replacing some tokens with plausible alternatives sampled from a small generator network. Then, instead of training a model that predicts the original identities of the corrupted tokens, we train a discriminative model that predicts whether each token in the corrupted input was replaced by a generator sample or not. Thorough experiments demonstrate this new pre-training task is more efficient than MLM because the task is defined over all input tokens rather than just the small subset that was masked out. As a result, the contextual representations learned by our approach substantially outperform the ones learned by BERT given the same model size, data, and compute. The gains are particularly strong for small models; for example, we train a model on one GPU for 4 days that outperforms GPT (trained using 30x more compute) on the GLUE natural language understanding benchmark. Our approach also works well at scale, where it performs comparably to RoBERTa and XLNet while using less than 1/4 of their compute and outperforms them when using the same amount of compute.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We will use FLOPs, as they allow us to estimate the needed energy relating hardware FLOPS with required FLOPs for a model [Hollemans, 2018, Clark et al., 2020].”
From Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning (2023) · cites this paper as Clark et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In NLP, they count FLOPs as single mathematical operations and not as a single hardware instructions (like in CV).”
From Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning (2023) · cites this paper as Clark et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The authors corrupted the tokens with plausible alternatives sampled from a small generator network that improves the model performance significantly.”
From A Review on Deep-Learning-Based Cyberbullying Detection (2023) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“but we ultimately found out this information explained in [Clark et al., 2020].”
From Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning (2023) · cites this paper as Clark et al., 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Self-citationEfficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
GAN（Generative Adversarial Nets）
淳司, 柴. (2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 20,507
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Unsupervised representation learning with deep convolutional generative adversarial networks
Radford, A., Metz, L., & Chintala, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06434 · Cited by 12,235
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 8,608
Flagged first, then by citations
Research works printed but not linked: 4
- Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS , 2014.
- Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942 , 2019. · arXiv
- Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. English gigaword, fifth edition. Technical report, Linguistic Data Consortium, Philadelphia, 2011.
- Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. MobileBERT: Task-agnostic compression of bert for resource limited devices, 2019b. URL https: //openreview.net/forum?id=SJxjVaNKwB .
Cited by
Sorted by citations
Items 101–200 of 1,010
CharBERT: Character-aware Pre-trained Language Model
Ma, W., Cui, Y., Si, C., Liu, T., Wang, S., & Hu, G. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.4 · Cited by 127
Widmann, T., & Wich, M. (2022) · Political Analysis · doi:10.1017/pan.2022.15 · Cited by 106
Hallucination Detection: Robustly Discerning Reliable Answers in Large Language Models
Chen, Y., Fu, Q., Yuan, Y., Wen, Z., Fan, G., Liu, D., Zhang, D., Li, Z., & Xiao, Y. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3614905 · Cited by 128
Language Models are Few-shot Multilingual Learners
Winata, G. I., Madotto, A., Lin, Z., Liu, R., Yosinski, J., & Fung, P. (2021) · Workshop on Multilingual Representation Learning (MRL) · doi:10.18653/v1/2021.mrl-1.1 · Cited by 150
A Review on Deep-Learning-Based Cyberbullying Detection
Hasan, M. T., Hossain, M. A. E., Mukta, M. S. H., Akter, A., Ahmed, M., & Islam, S. (2023) · Future Internet · doi:10.3390/fi15050179 · Cited by 103
Improved Text Classification via Contrastive Adversarial Training
Pan, L., Hang, C.-W., Sil, A., & Potdar, S. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21362 · Cited by 105
VIOLET : End-to-End Video-Language Transformers with Masked Visual-token ModelingPreprint
Fu, T.-J., Li, L., Gan, Z., Lin, K., Wang, W. Y., Wang, L., & Liu, Z. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.12681 · Cited by 212
Assessing Generalizability of CodeBERT
Zhou, X., Han, D., & Lo, D. (2021) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme52107.2021.00044 · Cited by 97
Debiasing Pre-trained Contextualised Embeddings
Kaneko, M., & Bollegala, D. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.107 · Cited by 134
When brain-inspired AI meets AGI
Zhao, L., Zhang, L., Wu, Z., Chen, Y., Dai, H., Yu, X., Liu, Z., Zhang, T., Hu, X., Jiang, X., et al. (2023) · Meta-Radiology · doi:10.1016/j.metrad.2023.100005 · Cited by 116
Machine/Deep Learning for Software Engineering: A Systematic Literature Review
Wang, S., Huang, L., Gao, A., Ge, J., Zhang, T., Feng, H., Satyarth, I., Li, M., Zhang, H., & Ng, V. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3173346 · Cited by 128
From Word Embeddings to Pre-Trained Language Models: A State-of-the-Art Walkthrough
Mars, M. (2022) · Applied Sciences · doi:10.3390/app12178805 · Cited by 91
Distinguishing Reality from AI: Approaches for Detecting Synthetic Content
Ghiurău, D., & Popescu, D. E. (2024) · Computers · doi:10.3390/computers14010001 · Cited by 122
Shen, D., Zheng, M., Shen, Y., Qu, Y., & Chen, W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.13818 · Cited by 114
Multimodal Emotion Recognition with High-Level Speech and Text Features
Makiuchi, M. R., Uto, K., & Shinoda, K. (2021) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru51503.2021.9688036 · Cited by 92
Overview and Discussion of the Competition on Legal Information Extraction/Entailment (COLIEE) 2021
Rabelo, J. C. B., Goebel, R. G., Kim, M., Kano, Y., Yoshioka, M., & Satoh, K. (2022) · The Review of Socionetwork Strategies · doi:10.1007/s12626-022-00105-z · Cited by 104
Kotei, E., & Thirunavukarasu, R. (2023) · Information · doi:10.3390/info14030187 · Cited by 89
LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval
Sun, S., Chen, Y.-C., Li, L., Wang, S., Fang, Y., & Liu, J. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.77 · Cited by 94
ELECTRIcity: An Efficient Transformer for Non-Intrusive Load Monitoring
Sykiotis, S., Kaselimi, M., Doulamis, A. D., & Doulamis, N. D. (2022) · Sensors · doi:10.3390/s22082926 · Cited by 93
MixMAE: Mixed and Masked Autoencoder for Efficient Pretraining of Hierarchical Vision Transformers
Liu, J., Huang, X., Zheng, J., Liu, Y., & Li, H. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00605 · Cited by 114
Understanding and Overcoming the Challenges of Efficient Transformer Quantization
Bondarenko, Y., Nagel, M., & Blankevoort, T. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.627 · Cited by 172
With Little Power Comes Great Responsibility
Card, D., Henderson, P., Khandelwal, U., Jia, R., Mahowald, K., & Jurafsky, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.745 · Cited by 84
Advancing microbial production through artificial intelligence-aided biology
Gong, X., Zhang, J., Gan, Q., Teng, Y., Hou, J., Lyu, Y., Liu, Z., Wu, Z., Dai, R., Zou, Y., et al. (2024) · Biotechnology Advances · doi:10.1016/j.biotechadv.2024.108399 · Cited by 85
Fake news detection: Taxonomy and comparative study
Farhangian, F., Cruz, R. M. O., & Cavalcanti, G. D. C. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.102140 · Cited by 91
Haynes, K., Shirazi, H. M., & Ray, I. (2021) · Procedia Computer Science · doi:10.1016/j.procs.2021.07.040 · Cited by 90
Quantum Natural Language Processing: Challenges and Opportunities
Guarasci, R., Pietro, G. D., & Esposito, M. (2022) · Applied Sciences · doi:10.3390/app12115651 · Cited by 85
An Empirical Comparison of Pre-Trained Models of Source Code
Niu, C., Li, C., Ng, V., Chen, D., Ge, J., & Luo, B. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00180 · Cited by 115
PARADE: Passage Representation Aggregation forDocument Reranking
Li, C., Yates, A., MacAvaney, S., He, B., & Sun, Y. (2023) · ACM Transactions on Information Systems · doi:10.1145/3600088 · Cited by 184
POINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training
Zhang, Y., Wang, G., Li, C., Gan, Z., Brockett, C., & Dolan, B. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.698 · Cited by 110
Not All Negatives are Equal: Label-Aware Contrastive Loss for Fine-grained Text Classification
Suresh, V., & Ong, D. C. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.359 · Cited by 82
Zhang, Z., Yu, W., Yu, M., Guo, Z., & Jiang, M. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.66 · Cited by 108
(2024) · ACM eBooks · doi:10.1145/3674127 · Cited by 86
Teng, T. H., & Varathan, K. D. (2023) · IEEE Access · doi:10.1109/access.2023.3275130 · Cited by 77
OmniMAE: Single Model Masked Pretraining on Images and Videos
Girdhar, R., El-Nouby, A., Singh, M., Alwala, K. V., Joulin, A., & Misra, I. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01003 · Cited by 101
SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code RepresentationPreprint
Wang, X. E., Wang, Y., Mi, F., Zhou, P., Wan, Y., Xiao, L., Li, L., Wu, H., Liu, J., & Jiang, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.04556 · Cited by 170
Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision
Tan, H. X., & Bansal, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.162 · Cited by 103
Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary
Deutsch, D., Bedrax-Weiss, T., & Roth, D. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00397 · Cited by 75
When FLUE Meets FLANG: Benchmarks and Large Pretrained Language Model for Financial Domain
Shah, R., Chawla, K., Eidnani, D., Shah, A., Du, W., Chava, S., Raman, N., Smiley, C., Chen, J., & Yang, D. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.148 · Cited by 74
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
Chi, Z., Huang, S., Dong, L., Ma, S., Zheng, B., Singhal, S., Bajaj, P., Song, X., Mao, X.-L., Huang, H., et al. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.427 · Cited by 85
Devika, R., Subramaniyaswamy, V., Mahenthar, C. S. J., Varadarajan, V., & Kotecha, K. V. (2021) · IEEE Access · doi:10.1109/access.2021.3133651 · Cited by 85
BioM-Transformers: Building Large Biomedical Language Models with BERT, ALBERT and ELECTRA
Alrowili, S., & Shanker, V. K. (2021) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2021.bionlp-1.24 · Cited by 80
Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora
Warstadt, A., Mueller, A., Choshen, L., Wilcox, E., Zhuang, C., Ciro, J., Mosquera, R., Paranjabe, B., Williams, A., Linzen, T., et al. (2023) · Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-babylm.1 · Cited by 74
Topic-Aware Multi-turn Dialogue Modeling
Xu, Y., Zhao, H., & Zhang, Z. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17668 · Cited by 73
Consistency Analysis of ChatGPT
Jang, M., & Lukasiewicz, T. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.991 · Cited by 117
MisRoBÆRTa: Transformers versus Misinformation
Truică, C., & Apostol, E. (2022) · Mathematics · doi:10.3390/math10040569 · Cited by 78
Mahmud, T., Ptaszynski, M. E., & Masui, F. (2023) · Applied Sciences · doi:10.3390/app132111875 · Cited by 73
DialogBERT: Discourse-Aware Response Generation via Learning to Recover and Rank Utterances
Gu, X., Yoo, K. M., & Ha, J.-W. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17527 · Cited by 74
Compressing Visual-linguistic Model via Knowledge Distillation
Fang, Z., Wang, J., Hu, X., Wang, L., Yang, Y., & Liu, Z. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00146 · Cited by 81
Code Summarization with Structure-induced Transformer
Wu, H., Zhao, H., & Zhang, M. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.93 · Cited by 83
Neural Media Bias Detection Using Distant Supervision With BABE - Bias Annotations By Experts
Spinde, T., Plank, M., Krieger, J.-D., Ruas, T., Gipp, B., & Aizawa, A. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.101 · Cited by 98
Transformer-Based Named Entity Recognition in Construction Supply Chain Risk Management in Australia
Shishehgarkhaneh, M. B., Moehler, R., Fang, Y., Hijazi, A. A., & Aboutorab, H. (2024) · IEEE Access · doi:10.1109/access.2024.3377232 · Cited by 79
Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questions
Aliannejadi, M., Kiseleva, J., Chuklin, A., Dalton, J., & Burtsev, M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.367 · Cited by 93
Lai, F.-L., & Gao, F. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad070 · Cited by 73
Neural natural language processing for long texts: A survey on classification and summarization
Tsirmpas, D., Gkionis, I., Papadopoulos, G. T., & Mademlis, I. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108231 · Cited by 77
ChatGPT: A Study on its Utility for Ubiquitous Software Engineering Tasks
Sridhara, G., G., R. H., & Mazumdar, S. (2023) · arXiv · doi:10.48550/arxiv.2305.16837 · Cited by 68
BiasFinder: Metamorphic Test Generation to Uncover Bias for Sentiment Analysis Systems
Asyrofi, M. H., Yang, Z., Yusuf, I. N. B., Kang, H. J., Thung, F., & Lo, D. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3136169 · Cited by 81
Pre-Training Transformers as Energy-Based Cloze Models
Clark, K., Luong, M.-T., Le, Q. V., & Manning, C. D. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.20 · Cited by 70
Self-citationOn cross-lingual retrieval with multilingual text encoders
Litschko, R., Vulić, I., Ponzetto, S. P., & Glavaš, G. (2022) · Information Retrieval · doi:10.1007/s10791-022-09406-x · Cited by 80
Ansari, M. Y., Yaqoob, M., Ishaq, M., Flushing, E. F., Mangalote, I. A. C., Dakua, S. P., Aboumarzouk, O. M., Righetti, R., & Qaraqe, M. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11259-x · Cited by 76
CIL: Contrastive Instance Learning Framework for Distantly Supervised Relation Extraction
Chen, T., Shi, H., Tang, S., Chen, Z., Wu, F. Q., & Zhuang, Y. (2021) · doi:10.18653/v1/2021.acl-long.483 · Cited by 66
Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color
Abdou, M., Kulmizev, A., Hershcovich, D., Frank, S., Pavlick, E., & Søgaard, A. (2021) · Proceedings of the 25th Conference on Computational Natural Language Learning · doi:10.18653/v1/2021.conll-1.9 · Cited by 64
Aburass, S., Dorgham, O. M., & Shaqsi, J. A. (2024) · Systems and Soft Computing · doi:10.1016/j.sasc.2024.200110 · Cited by 71
Findings of the Association for Computational Linguistics: EMNLP 2020
Dumitrescu, S., Avram, A.-M., & Pyysalo, S. (2022) · UTUPub (University of Turku) · Cited by 71
(2024) · doi:10.1109/sp54263.2024.00182 · Cited by 62
BARTpho: Pre-trained Sequence-to-Sequence Models for Vietnamese
Tran, N. L., Le, D., & Nguyen, D. Q. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10177 · Cited by 80
SEMI-FND: Stacked ensemble based multimodal inferencing framework for faster fake news detection
Singh, P., Srivastava, R., Rana, K. P. S., & Kumar, V. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.119302 · Cited by 65
Toward Label-Efficient Emotion and Sentiment Analysis
Zhao, S., Hong, X., Yang, J., Zhao, Y., & Ding, G. (2023) · Proceedings of the IEEE · doi:10.1109/jproc.2023.3309299 · Cited by 62
Efficient Passage Retrieval with Hashing for Open-domain Question Answering
Yamada, I., Asai, A., & Hajishirzi, H. (2021) · doi:10.18653/v1/2021.acl-short.123 · Cited by 88
MAGMA: An Optimization Framework for Mapping Multiple DNNs on Multiple Accelerator Cores
Kao, S.-C., & Krishna, T. (2022) · 2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA) · doi:10.1109/hpca53966.2022.00065 · Cited by 58
Few-shot Learning for Multi-label Intent Detection
Hou, Y., Lai, Y., Wu, Y., Che, W., & Liu, T. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17541 · Cited by 60
Are Pretrained Convolutions Better than Pretrained Transformers?
Tay, Y., Dehghani, M., Gupta, J. P., Aribandi, V., Bahri, D., Qin, Z., & Metzler, D. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.335 · Cited by 99
Learning Rich Representation of Keyphrases from Text
Kulkarni, M., Mahata, D., Arora, R., & Bhowmik, R. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.67 · Cited by 80
VulExplainer: A Transformer-Based Hierarchical Distillation for Explaining Vulnerability Types
Fu, M. C., Nguyen, V., Tantithamthavorn, C. K., Le, T., & Phung, D. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3305244 · Cited by 62
Li, Z., Wallace, E., Shen, S., Lin, K., Keutzer, K., Klein, D., & Gonzalez, J. E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.11794 · Cited by 107
Gaur, M., Gunaratna, K., Srinivasan, V., & Jin, H. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21312 · Cited by 57
Oliaee, A. H., Das, S., Liu, J., & Rahman, M. A. (2023) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100007 · Cited by 65
Transformer Based Language Identification for Malayalam-English Code-Mixed Text
Thara, S., & Poornachandran, P. (2021) · IEEE Access · doi:10.1109/access.2021.3104106 · Cited by 59
Session-based cyberbullying detection in social media: A survey
Yi, P., & Zubiaga, A. (2023) · Online Social Networks and Media · doi:10.1016/j.osnem.2023.100250 · Cited by 55
The effectiveness of MAE pre-pretraining for billion-scale pretraining
Singh, M., Duval, Q., Alwala, K. V., Fan, H., Aggarwal, V., Adcock, A., Joulin, A., Dollár, P., Feichtenhofer, C., Girshick, R., et al. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00505 · Cited by 70
Self-Supervised Knowledge Triplet Learning for Zero-Shot Question Answering
Banerjee, P., & Baral, C. R. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.11 · Cited by 54
Wav2KWS: Transfer Learning From Speech Representations for Keyword Spotting
Seo, D., Oh, H., & Jung, Y. (2021) · IEEE Access · doi:10.1109/access.2021.3078715 · Cited by 65
AMBERT: A Pre-trained Language Model with Multi-Grained Tokenization
Zhang, X., Li, P., & Li, H. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.37 · Cited by 58
CRISPR-DIPOFF: an interpretable deep learning approach for CRISPR Cas-9 off-target prediction
Toufikuzzaman, M., Samee, M. A. H., & Rahman, M. S. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbad530 · Cited by 60
DomBERT: Domain-oriented Language Model for Aspect-based Sentiment Analysis
Xu, H., Liu, B., Shu, L., & Yu, P. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.156 · Cited by 61
Glavaš, G., & Vulić, I. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.270 · Cited by 65
Commonsense Knowledge Reasoning and Generation with Pre-trained Language Models: A Survey
Bhargava, P., & Ng, V. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21496 · Cited by 62
Predicting medical specialty from text based on a domain-specific pre-trained BERT
Kim, Y., Kim, J.-H., Kim, Y., Song, S., & Joo, H. J. (2022) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2022.104956 · Cited by 57
Jin, Z., Lalwani, A., Vaidhya, T., Shen, X., Ding, Y., Lyu, Z., Sachan, M., Mihalcea, R., & Schölkopf, B. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.532 · Cited by 52
Nguyen, X.-B., Bui, D. T., Duong, C. N., Bui, T. D., & Luu, K. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01070 · Cited by 55
Train No Evil: Selective Masking for Task-Guided Pre-Training
Gu, Y., Zhang, Z., Wang, X., Liu, Z., & Sun, M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.566 · Cited by 58
A deep-learning based citation count prediction model with paper metadata semantic features
Ma, A., Liu, Y., Xu, X., & Dong, T. (2021) · Scientometrics · doi:10.1007/s11192-021-04033-7 · Cited by 57
Domain Adaptation with Pre-trained Transformers for Query-Focused Abstractive Text Summarization
Laskar, M. T. R., Hoque, E., & Huang, J. X. (2022) · Computational Linguistics · doi:10.1162/coli_a_00434 · Cited by 58
Cortiz, D. (2022) · International Conference on Control, Robotics and Intelligent System (CCRIS) · doi:10.1145/3562007.3562051 · Cited by 57
Bangla Documents Classification using Transformer Based Deep Learning Models
Rahman, M. M., Pramanik, M. A., Sadik, R., Roy, M., & Chakraborty, P. (2020) · International Conference on Sustainable Technologies for Industry 5.0 (STI) · doi:10.1109/sti50764.2020.9350394 · Cited by 51
An emotional classification method of Chinese short comment text based on ELECTRA
Zhang, S., Yu, H., & Zhu, G. (2021) · Connection Science · doi:10.1080/09540091.2021.1985968 · Cited by 51
An analysis of protein language model embeddings for fold prediction
Villegas-Morcillo, A., Gomez, A. M., & Sanchez, V. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac142 · Cited by 46
Yang, M., Huang, L., Huang, H., Tang, H., Zhang, N., Yang, H., Wu, J., & Mu, F. (2022) · Nucleic Acids Research · doi:10.1093/nar/gkac326 · Cited by 56
Improved Methods to Aid Unsupervised Evidence-Based Fact Checking for Online Heath News
Deka, P., Jurek, A., & Deepak, P. (2022) · Journal of Data Intelligence · doi:10.26421/jdi3.4-5 · Cited by 58
ExploitGen: Template-augmented exploit code generation based on CodeBERT
Yang, G., Zhou, Y., Chen, X., Zhang, X., Han, T., & Chen, T. (2022) · Journal of Systems and Software · doi:10.1016/j.jss.2022.111577 · Cited by 53
Shared-Private Memory Networks For Multimodal Sentiment Analysis
Zhao, X., Chen, Y., Liu, S., & Tang, B. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3222023 · Cited by 50
Metadata
- Authors
- Clark, K., Luong, M.-T., Le, Q. V., & Manning, C. D.
- Affiliations
- Stanford University (US)
- Google (United States) (US)
- ORCID
- Le, Quoc V.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2003.10555
- OpenAlex
- W2996035354
- Cited by
- 1,844
- References
- 48
- Citations by year
- 2026: 13 · 2025: 129 · 2024: 240 · 2023: 572 · 2022: 415 · 2021: 375 · 2020: 100
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Learning Representations
- Funders
- Canadian Institute for Advanced Research
- Programa Operacional Regional do Centro (Portugal 2020)
- Fundação para a Ciência e a Tecnologia (PT2020, -01-0247-, 00326, UID/CEC/00326/2020)
- European Regional Development Fund (PT2020)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- Electra, masked language modeling, text encoder, sample efficiency, context representation, natural language understanding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
