The DOI matches its DataCite record; 48 of the 48 references it cites were found in a registry or an index.
ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
Open accessAbstract
Masked language modeling (MLM) pre-training methods such as BERT corrupt the input by replacing some tokens with [MASK] and then train a model to reconstruct the original tokens. While they produce good results when transferred to downstream NLP tasks, they generally require large amounts of compute to be effective. As an alternative, we propose a more sample-efficient pre-training task called replaced token detection. Instead of masking the input, our approach corrupts it by replacing some tokens with plausible alternatives sampled from a small generator network. Then, instead of training a model that predicts the original identities of the corrupted tokens, we train a discriminative model that predicts whether each token in the corrupted input was replaced by a generator sample or not. Thorough experiments demonstrate this new pre-training task is more efficient than MLM because the task is defined over all input tokens rather than just the small subset that was masked out. As a result, the contextual representations learned by our approach substantially outperform the ones learned by BERT given the same model size, data, and compute. The gains are particularly strong for small models; for example, we train a model on one GPU for 4 days that outperforms GPT (trained using 30x more compute) on the GLUE natural language understanding benchmark. Our approach also works well at scale, where it performs comparably to RoBERTa and XLNet while using less than 1/4 of their compute and outperforms them when using the same amount of compute.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We will use FLOPs, as they allow us to estimate the needed energy relating hardware FLOPS with required FLOPs for a model [Hollemans, 2018, Clark et al., 2020].”
From Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning (2023) · cites this paper as Clark et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In NLP, they count FLOPs as single mathematical operations and not as a single hardware instructions (like in CV).”
From Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning (2023) · cites this paper as Clark et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The authors corrupted the tokens with plausible alternatives sampled from a small generator network that improves the model performance significantly.”
From A Review on Deep-Learning-Based Cyberbullying Detection (2023) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“but we ultimately found out this information explained in [Clark et al., 2020].”
From Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning (2023) · cites this paper as Clark et al., 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Self-citationEfficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
GAN（Generative Adversarial Nets）
淳司, 柴. (2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 20,507
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Unsupervised representation learning with deep convolutional generative adversarial networks
Radford, A., Metz, L., & Chintala, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06434 · Cited by 12,235
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 8,608
Flagged first, then by citations
Research works printed but not linked: 4
- Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS , 2014.
- Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942 , 2019. · arXiv
- Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. English gigaword, fifth edition. Technical report, Linguistic Data Consortium, Philadelphia, 2011.
- Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. MobileBERT: Task-agnostic compression of bert for resource limited devices, 2019b. URL https: //openreview.net/forum?id=SJxjVaNKwB .
Cited by
Sorted by citations
Items 301–400 of 1,010
Unleashing the transformers: NLP models detect AI writing in education
Campino, J. (2024) · Journal of Computers in Education · doi:10.1007/s40692-024-00325-y · Cited by 39
Soudeep, S., Aurthy, M. L. N., Jim, J. R., Mridha, M. F., & Kabir, M. M. (2024) · Sustainable Cities and Society · doi:10.1016/j.scs.2024.105882 · Cited by 32
Evaluating Pretrained Transformer-based Models for COVID-19 Fake News Detection
Hande, A., Puranik, K., Priyadharshini, R., Thavareesan, S., & Chakravarthi, B. R. (2021) · International Conference on Computing Methodologies and Communication (ICCMC) · doi:10.1109/iccmc51019.2021.9418446 · Cited by 32
SlovakBERT: Slovak Masked Language Model
Pikuliak, M., Grivalský, Š., Konôpka, M., Blšták, M., Tamajka, M., Bachratý, V., Šimko, M., Balážik, P., Trnka, M., & Uhlárik, F. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.530 · Cited by 36
Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks
Asai, A., Gardner, M., & Hajishirzi, H. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.162 · Cited by 47
Call for Customized Conversation: Customized Conversation Grounding Persona and Knowledge
Jang, Y., Lim, J., Hur, Y., Oh, D., Son, S., Lee, Y., Shin, D., Kim, S., & Lim, H. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21326 · Cited by 39
Introduction to Neural Transfer Learning With Transformers for Social Science Text Analysis
Wankmüller, S. (2022) · Sociological Methods & Research · doi:10.1177/00491241221134527 · Cited by 45
Clickbait Spoiling via Question Answering and Passage Retrieval
Hagen, M., Fröbe, M., Jurk, A., & Potthast, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.484 · Cited by 35
Onan, A., & Balbal, K. F. (2024) · IEEE Access · doi:10.1109/access.2024.3349971 · Cited by 35
Advancing bioinformatics with language models: components, applications, and perspectives
Liu, J., Yang, M., Yu, Y., Xu, H., Wang, T., Li, K., & Zhou, X. (2026) · Briefings in Bioinformatics · doi:10.1093/bib/bbag367 · Cited by 66
Safety analysis in the era of large language models: A case study of STPA using ChatGPT
Qi, Y., Zhao, X., Khastgir, S., & Huang, X. (2025) · Machine Learning with Applications · doi:10.1016/j.mlwa.2025.100622 · Cited by 49
Liu, S., Kong, W., Liu, Z., Sun, J., Liu, S., & Gašević, D. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127339 · Cited by 32
Pretrained Language Model Embryology: The Birth of ALBERT
Chiang, C.-H., Huang, S.-F., & Lee, H.-Y. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.553 · Cited by 42
Sentiment Analysis Using Language Models: A Study
Kumawat, S., Yadav, I., Pahal, N., & Goel, D. (2021) · International Conference on Cloud Computing, Data Science & Engineering (Confluence) · doi:10.1109/confluence51648.2021.9377043 · Cited by 38
Obtaining Better Static Word Embeddings Using Contextual Embedding Models
Gupta, P., & Jaggi, M. (2021) · doi:10.18653/v1/2021.acl-long.408 · Cited by 36
Dialogue Graph Modeling for Conversational Machine Reading
Ouyang, S., Zhang, Z., & Zhao, H. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.279 · Cited by 33
bert2BERT: Towards Reusable Pretrained Language Models
Chen, C., Yin, Y., Shang, L., Jiang, X., Qin, Y., Wang, F., Wang, Z., Chen, X., Liu, Z., & Liu, Q. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.151 · Cited by 42
Lexical Complexity Prediction: An Overview
North, K., Zampieri, M., & Shardlow, M. (2022) · ACM Computing Surveys · doi:10.1145/3557885 · Cited by 43
Nowakowski, K., Ptaszynski, M. E., Murasaki, K., & Nieuważny, J. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103148 · Cited by 32
AI-based Question Answering Assistance for Analyzing Natural-language Requirements
Ezzini, S., Abualhaija, S., Arora, C., & Sabetzadeh, M. (2023) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse48619.2023.00113 · Cited by 39
Digital forgetting in large language models: a survey of unlearning methods
Blanco-Justicia, A., Jebreel, N., Manzanares-Salor, B., Sánchez, D., Domingo-Ferrer, J., Collell, G., & Eeik Tan, K. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-024-11078-6 · Cited by 31
Disengagement Cause-and-Effect Relationships Extraction Using an NLP Pipeline
Zhang, Y., Yang, X. J., & Zhou, F. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2022.3186248 · Cited by 32
Exploring Neural Models for Query-Focused Summarization
Vig, J., Fabbri, A. R., Kryściński, W., Wu, C.-S., & Liu, W. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.109 · Cited by 45
Leveraging the meta-embedding for text classification in a resource-constrained language
Hossain, M. R., Hoque, M. M., & Siddique, N. H. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106586 · Cited by 31
Improving Sharpness-Aware Minimization with Fisher Mask for Better Generalization on Language Models
Zhong, Q., Ding, L., Shen, L., Mi, P., Liu, J., Du, B., & Tao, D. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.300 · Cited by 47
Mapping medical image-text to a joint space via masked modeling
Chen, Z., Du, Y., Hu, J., Liu, Y., Li, G., Wan, X., & Chang, T. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.103018 · Cited by 31
Group-wise Contrastive Learning for Neural Dialogue Generation
Cai, H., Chen, H., Song, Y., Ding, Z., Bao, Y., Yan, W., & Zhao, X. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.70 · Cited by 35
MASKER: Masked Keyword Regularization for Reliable Text Classification
Moon, S. J., Mo, S., Lee, K., Lee, J., & Shin, J. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17601 · Cited by 33
CoLES: Contrastive Learning for Event Sequences with Self-Supervision
Babaev, D., Ovsov, N., Kireev, I., Ivanova, M., Gusev, G., Nazarov, I., & Tuzhilin, A. (2022) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3514221.3526129 · Cited by 44
HiddenCut: Simple Data Augmentation for Natural Language Understanding with Better Generalizability
Chen, J., Shen, D., Chen, W., & Yang, D. (2021) · doi:10.18653/v1/2021.acl-long.338 · Cited by 35
ConvFiT: Conversational Fine-Tuning of Pretrained Language Models
Vulić, I., Su, P.-H., Coope, S., Gerz, D., Budzianowski, P., Casanueva, I., Mrkšić, N., & Wen, T.-H. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.88 · Cited by 34
A Structure Self-Aware Model for Discourse Parsing on Multi-Party Dialogues
Wang, A., Song, L., Jiang, H., Lai, S., Yao, J., Zhang, M., & Su, J. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/543 · Cited by 29
Multilevel depression status detection based on fine-grained prompt learning
Zhang, J., & Guo, Y. (2024) · Pattern Recognition Letters · doi:10.1016/j.patrec.2024.01.005 · Cited by 34
Learning a deep language model for microbiomes: The power of large scale unlabeled microbiome data
Pope, Q., Varma, R., Tataru, C. A., David, M. M., & Fern, X. Z. (2025) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011353 · Cited by 30
MeDAL: Medical Abbreviation Disambiguation Dataset for Natural Language Understanding Pretraining
Wen, Z., Lù, X. H., & Reddy, S. (2020) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2020.clinicalnlp-1.15 · Cited by 38
Enhanced Speaker-Aware Multi-Party Multi-Turn Dialogue Comprehension
Ma, X., Zhang, Z., & Zhao, H. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3284516 · Cited by 30
GOProFormer: A Multi-Modal Transformer Method for Gene Ontology Protein Function Prediction
Kabir, A., & Shehu, A. (2022) · Biomolecules · doi:10.3390/biom12111709 · Cited by 29
Evolution and emerging trends of named entity recognition: Bibliometric analysis from 2000 to 2023
Yang, J., Zhang, T., Tsai, C.-Y., Lu, Y., & Yao, L. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e30053 · Cited by 33
EntQA: Entity Linking as Question AnsweringPreprint
Zhang, W., Hua, W., & Stratos, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.02369 · Cited by 40
Clinical concept and relation extraction using prompt-based machine reading comprehension
Cheng, P., Yang, X., Yu, Z., Bian, J., Hogan, W. R., & Wu, Y. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad107 · Cited by 37
Exploring transformer models in the sentiment analysis task for the under-resource Bengali language
Hoque, M. N., Salma, U., Uddin, M. J., Ahamad, M. M., & Aktar, S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2024.100091 · Cited by 34
Bao, R., Wang, J., & Zhao, H. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.287 · Cited by 28
Sun, B., Feng, S., Li, Y., Liu, J., & Li, K. (2021) · doi:10.18653/v1/2021.acl-long.437 · Cited by 29
Cross-lingual Text Classification with Heterogeneous Graph Neural Network
Wang, Z., Liu, X., Yang, P., Liu, S., & Wang, Z. (2021) · doi:10.18653/v1/2021.acl-short.78 · Cited by 30
Correlation-Guided Representation for Multi-Label Text Classification
Zhang, Q., Zhang, X., Yan, Z., Liu, R., Cao, Y., & Zhang, M.-L. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/463 · Cited by 31
EVIL: Exploiting Software via Natural Language
Liguori, P., Al-Hossami, E., Orbinato, V., Natella, R., Shaikh, S., Cotroneo, D., & Čukić, B. (2021) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre52982.2021.00042 · Cited by 33
Nguyen, L., Yuan, Z., & Seed, G. (2022) · Languages · doi:10.3390/languages7030220 · Cited by 40
KEBLM: Knowledge-Enhanced Biomedical Language Models
Lai, T., Zhai, C., & Ji, H. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104392 · Cited by 31
XPrompt: Exploring the Extreme of Prompt Tuning
Ma, F., Zhang, C., Ren, L., Wang, J., Wang, Q., Wu, W., Quan, X., & Song, D. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.758 · Cited by 45
Li, W., Hacid, H., Almazrouei, E., & Debbah, M. A. (2023) · AI · doi:10.3390/ai4030039 · Cited by 37
UDAPDR: Unsupervised Domain Adaptation via LLM Prompting and Distillation of Rerankers
Saad-Falcon, J., Khattab, O., Santhanam, K., Florian, R., Franz, M., Roukos, S., Sil, A., Sultan, M. A., & Potts, C. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.693 · Cited by 53
Taneja, K. K., Vashishtha, J., & Ratnoo, S. (2024) · Procedia Computer Science · doi:10.1016/j.procs.2024.04.220 · Cited by 32
Li, Y., & Zhao, H. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.176 · Cited by 28
Multi-Class Grammatical Error Detection for Correction: A Tale of Two Systems
Yuan, Z., Taslimipoor, S., Davis, C. M., & Bryant, C. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.687 · Cited by 31
Learning to Generalize to More: Continuous Semantic Augmentation for Neural Machine Translation
Wei, X., Yu, H., Hu, Y., Weng, R., Luo, W., & Jin, R. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.546 · Cited by 31
CODE-MVP: Learning to Represent Source Code from Multiple Views with Contrastive Pre-Training
Wang, X., Wang, Y., Wan, Y., Wang, J., Zhou, P., Li, L., Wu, H., & Liu, J. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.80 · Cited by 29
Structural Characterization for Dialogue Disentanglement
Ma, X., Zhang, Z., & Zhao, H. Y. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.23 · Cited by 29
Gupta, S., Agarwal, A., Gaur, M., Roy, K., Narayanan, V., Kumaraguru, P., & Sheth, A. (2022) · Proceedings of the Eighth Workshop on Computational Linguistics and Clinical Psychology · doi:10.18653/v1/2022.clpsych-1.12 · Cited by 27
Classifying Drug Ratings Using User Reviews with Transformer-Based Language Models
Shiju, A., & He, Z. (2022) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi54592.2022.00035 · Cited by 29
GS-InGAT: An interaction graph attention network with global semantic for knowledge graph completion
Yin, H., Zhong, J., Wang, C., Li, R., & Li, X. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120380 · Cited by 29
STAR: SQL Guided Pre-Training for Context-dependent Text-to-SQL Parsing
Cai, Z., Li, X., Hui, B., Yang, M., Li, B., Li, B., Cao, Z., Li, W., Huang, F., Si, L., et al. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.89 · Cited by 28
Hua, H., & Yao, C.-J. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1469197 · Cited by 27
HyFusER: Hybrid Multimodal Transformer for Emotion Recognition Using Dual Cross Modal Attention
Yi, M.-H., Kwak, K.-C., & Shin, J. (2025) · Applied Sciences · doi:10.3390/app15031053 · Cited by 30
An entity-graph based reasoning method for fact verification
Chen, C., Cai, F., Hu, X., Zheng, J., Ling, Y., & Chen, H. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2020.102472 · Cited by 28
Learning-Free Unsupervised Extractive Summarization Model
Jang, M., & Kang, P. (2021) · IEEE Access · doi:10.1109/access.2021.3051237 · Cited by 28
Zhang, J., Hashimoto, K., Wan, Y., Liu, Z., Liu, Y., Xiong, C., & Yu, P. (2022) · Workshop on NLP for Conversational AI (NLP4ConvAI) · doi:10.18653/v1/2022.nlp4convai-1.2 · Cited by 34
Structural Pre-training for Dialogue Comprehension
Zhang, Z., & Zhao, H. (2021) · doi:10.18653/v1/2021.acl-long.399 · Cited by 28
Quality Controlled Paraphrase Generation
Bandel, E., Aharonov, R., Shmueli-Scheuer, M., Shnayderman, I., Slonim, N., & Ein‐Dor, L. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.45 · Cited by 27
Transcending Scaling Laws with 0.1% Extra Compute
Tay, Y., Wei, J., Chung, H. W., Tran, V., So, D. R., Shakeri, S., Garcia, X., Zheng, S., Rao, J., Chowdhery, A., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.91 · Cited by 95
Self-citationA Review of Bangla Natural Language Processing Tasks and the Utility of Transformer ModelsPreprint
Alam, F., Hasan, M. A., Alam, T., Khan, A., Tajrin, J., Khan, N., & Chowdhury, S. A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.03844 · Cited by 51
Performance analysis of word embeddings for cyberbullying detection
Pericherla, S., & Ilavarasan, E. (2021) · IOP Conference Series: Materials Science and Engineering · doi:10.1088/1757-899x/1085/1/012008 · Cited by 24
Enhancing Machine-Generated Text Detection: Adversarial Fine-Tuning of Pre-Trained Language Models
Lee, D., & Jang, B. (2024) · IEEE Access · doi:10.1109/access.2024.3396820 · Cited by 27
DiPair: Fast and Accurate Distillation for Trillion-Scale Text Matching and Pair Modeling
Chen, J., Liu, Y., Raman, K., Bendersky, M., Yeh, J.-J., Zhou, Y., Najork, M., Cai, D., & Emadzadeh, E. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.264 · Cited by 29
An Empirical Comparison of BERT, RoBERTa, and Electra for Fact Verification
Naseer, M., Asvial, M., & Sari, R. F. (2021) · International Conference on Artificial Intelligence in Information and Communication (ICAIIC) · doi:10.1109/icaiic51459.2021.9415192 · Cited by 30
Deep Context- and Relation-Aware Learning for Aspect-based Sentiment Analysis
Oh, S., Lee, D., Whang, T., Park, I., Gaeun, S., Kim, E., & Kim, H. (2021) · doi:10.18653/v1/2021.acl-short.63 · Cited by 26
ELECTRA for Neural Coreference Resolution in Italian
Guarasci, R., Minutolo, A., Damiano, E., Pietro, G. D., Fujita, H., & Esposito, M. (2021) · IEEE Access · doi:10.1109/access.2021.3105278 · Cited by 24
Greedy Optimization Method for Extractive Summarization of Scientific Articles
Akhmetov, I., Gelbukh, A. F., & Mussabayev, R. (2021) · IEEE Access · doi:10.1109/access.2021.3136302 · Cited by 27
Exploiting Transformer-Based Multitask Learning for the Detection of Media Bias in News Articles
Spinde, T., Krieger, J.-D., Ruas, T., Mitrović, J., Götz-Hahn, F., Aizawa, A., & Gipp, B. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-96957-8_20 · Cited by 27
Open-domain Dialogue Generation: What We Can Do, Cannot Do, And Should Do Next
Kann, K., Ebrahimi, A., Koh, J. J., Dudy, S., & Roncone, A. (2022) · Workshop on NLP for Conversational AI (NLP4ConvAI) · doi:10.18653/v1/2022.nlp4convai-1.13 · Cited by 28
Nentidis, A., Katsimpras, G., Vandorou, E., Krithara, A., Miranda-Escalada, A., Gascó, L., Krallinger, M., & Παλιούρας, Γ. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-13643-6_22 · Cited by 26
CM-MaskSD: Cross-Modality Masked Self-Distillation for Referring Image Segmentation
Wang, W., He, X., Zhang, Y., Guo, L., Shen, J., Li, J., & Liu, J. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3358085 · Cited by 26
pysentimiento: A Python Toolkit for Opinion Mining and Social NLP tasksPreprint
Perez, J. M., Rajngewerc, M., Giudici, J. C., Furman, D., Luque, F. M., Alemany, L. A., & Martínez, M. V. (2023) · Research Square · doi:10.21203/rs.3.rs-3570648/v1 · Cited by 28
A T5-based interpretable reading comprehension model with more accurate evidence training
Guan, B., Zhu, X., & Yuan, S. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103584 · Cited by 29
Deep Learning Models for Representing Out-of-Vocabulary Words
Lochter, J. V., Silva, R. M., & Almeida, T. A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-61377-8_29 · Cited by 25
HULK: An Energy Efficiency Benchmark Platform for Responsible Natural Language Processing
Zhou, X., Chen, Z., Jin, X., & Wang, W. Y. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/2021.eacl-demos.39 · Cited by 24
AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models
Yin, Y., Chen, C., Shang, L., Jiang, X., Chen, X., & Liu, Q. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.400 · Cited by 24
AutoBERT-Zero: Evolving BERT Backbone from Scratch
Gao, J., Xu, H., Han, S., Ren, X., Yu, P. L. H., Liang, X., Jiang, X., & Li, Z. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21311 · Cited by 32
Suresh, V., & Ong, D. C. (2021) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii52823.2021.9597390 · Cited by 26
Wu, Z., Lu, B.-R., Hajishirzi, H., & Mari Ostendorf, E. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.140 · Cited by 26
THQuAD: Turkish Historic Question Answering Dataset for Reading Comprehension
Soygazi, F., Çiftçi, O., Kok, U., & Cengiz, S. (2021) · International Conference on Computer Science and Engineering (UBMK) · doi:10.1109/ubmk52708.2021.9559013 · Cited by 28
Xu, F., Liu, J., Lin, Q., Pan, Y., & Zhang, L. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3532016 · Cited by 24
Poulsen, M. N., Freda, P. J., Troiani, V., Davoudi, A., & Mowery, D. L. (2022) · Frontiers in Public Health · doi:10.3389/fpubh.2022.850619 · Cited by 24
Cause-and-Effect Analysis of ADAS: A Comparison Study between Literature Review and Complaint Data
Ayoub, J., Wang, Z., Li, M., Guo, H., Sherony, R., Bao, S., & Zhou, F. (2022) · International Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutomotiveUI) · doi:10.1145/3543174.3547117 · Cited by 26
A URL-Based Social Semantic Attacks Detection With Character-Aware Language Model
Almousa, M., & Anwar, M. M. (2023) · IEEE Access · doi:10.1109/access.2023.3241121 · Cited by 30
Catelli, R., Bevilacqua, L., Mariniello, N., di Carlo, V. S., Magaldi, M., Fujita, H., Pietro, G. D., & Esposito, M. (2023) · IEEE Access · doi:10.1109/access.2023.3277490 · Cited by 25
TemplateGEC: Improving Grammatical Error Correction with Detection Template
Li, Y., Liu, X., Wang, S., Gong, P., Wong, D. F., Gao, Y., Huang, H., & Zhang, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.380 · Cited by 24
Sentiment-Aware Word and Sentence Level Pre-training for Sentiment Analysis
Fan, S., Lin, C., Li, H., Lin, Z., Su, J., Zhang, H., Gong, Y., Guo, J., & Duan, N. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.332 · Cited by 25
An Empirical Study on Hyperparameter Optimization for Fine-Tuning Pre-trained Language Models
Liu, X., & Wang, C. (2021) · doi:10.18653/v1/2021.acl-long.178 · Cited by 39
Finding the Law: Enhancing Statutory Article Retrieval via Graph Neural Networks
Louis, A., DIJCK, G. V., & Spanakis, G. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.203 · Cited by 37
The changing landscape of text mining: a review of approaches for ecology and evolution
Farrell, M. J., Guillarme, N. L., Brierley, L., Hunter, S. B., Scheepens, D. R., Willoughby, A. R., Yates, A., & Mideo, N. (2024) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.2024.0423 · Cited by 32
Metadata
- Authors
- Clark, K., Luong, M.-T., Le, Q. V., & Manning, C. D.
- Affiliations
- Stanford University (US)
- Google (United States) (US)
- ORCID
- Le, Quoc V.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2003.10555
- OpenAlex
- W2996035354
- Cited by
- 1,844
- References
- 48
- Citations by year
- 2026: 13 · 2025: 129 · 2024: 240 · 2023: 572 · 2022: 415 · 2021: 375 · 2020: 100
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Learning Representations
- Funders
- Canadian Institute for Advanced Research
- Programa Operacional Regional do Centro (Portugal 2020)
- Fundação para a Ciência e a Tecnologia (PT2020, -01-0247-, 00326, UID/CEC/00326/2020)
- European Regional Development Fund (PT2020)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- Electra, masked language modeling, text encoder, sample efficiency, context representation, natural language understanding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
