The DOI matches its DataCite record; 48 of the 48 references it cites were found in a registry or an index.
ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
Open accessAbstract
Masked language modeling (MLM) pre-training methods such as BERT corrupt the input by replacing some tokens with [MASK] and then train a model to reconstruct the original tokens. While they produce good results when transferred to downstream NLP tasks, they generally require large amounts of compute to be effective. As an alternative, we propose a more sample-efficient pre-training task called replaced token detection. Instead of masking the input, our approach corrupts it by replacing some tokens with plausible alternatives sampled from a small generator network. Then, instead of training a model that predicts the original identities of the corrupted tokens, we train a discriminative model that predicts whether each token in the corrupted input was replaced by a generator sample or not. Thorough experiments demonstrate this new pre-training task is more efficient than MLM because the task is defined over all input tokens rather than just the small subset that was masked out. As a result, the contextual representations learned by our approach substantially outperform the ones learned by BERT given the same model size, data, and compute. The gains are particularly strong for small models; for example, we train a model on one GPU for 4 days that outperforms GPT (trained using 30x more compute) on the GLUE natural language understanding benchmark. Our approach also works well at scale, where it performs comparably to RoBERTa and XLNet while using less than 1/4 of their compute and outperforms them when using the same amount of compute.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We will use FLOPs, as they allow us to estimate the needed energy relating hardware FLOPS with required FLOPs for a model [Hollemans, 2018, Clark et al., 2020].”
From Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning (2023) · cites this paper as Clark et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In NLP, they count FLOPs as single mathematical operations and not as a single hardware instructions (like in CV).”
From Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning (2023) · cites this paper as Clark et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The authors corrupted the tokens with plausible alternatives sampled from a small generator network that improves the model performance significantly.”
From A Review on Deep-Learning-Based Cyberbullying Detection (2023) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“but we ultimately found out this information explained in [Clark et al., 2020].”
From Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning (2023) · cites this paper as Clark et al., 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Self-citationEfficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
GAN（Generative Adversarial Nets）
淳司, 柴. (2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 20,507
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Unsupervised representation learning with deep convolutional generative adversarial networks
Radford, A., Metz, L., & Chintala, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06434 · Cited by 12,235
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 8,608
Research works printed but not linked: 4
- Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS , 2014.
- Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942 , 2019. · arXiv
- Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. English gigaword, fifth edition. Technical report, Linguistic Data Consortium, Philadelphia, 2011.
- Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. MobileBERT: Task-agnostic compression of bert for resource limited devices, 2019b. URL https: //openreview.net/forum?id=SJxjVaNKwB .
Cited by
Sorted by citations
Items 701–800 of 1,010
Training Multilingual Pre-trained Language Model with Byte-level SubwordsPreprint
Wei, V. J., Liu, Q., Guo, Y., & Jiang, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.09469 · Cited by 24
Abstractive text summarization for Hungarian
Yang, Z. G., Agócs, Á., Kusper, G., & Váradi, T. (2021) · Az Eszterházy Károly Tanárképző Főiskola tudományos közleményei. Tanulmányok a matematikai tudományok köréből/Az Eszterházy Károly Főiskola tudományos közleményei. Tanulmányok a matematikai tudományok köréből/Annales mathematicae et informaticae · doi:10.33039/ami.2021.04.002 · Cited by 11
Investigating the Role of Negatives in Contrastive Representation LearningPreprint
Ash, J. T., Goel, S., Krishnamurthy, A., & Misra, D. K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.09943 · Cited by 35
PARADISE: Exploiting Parallel Data for Multilingual Sequence-to-Sequence Pretraining
Reid, M., & Artetxe, M. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.58 · Cited by 16
Idrissi-Yaghir, A., Schäfer, H., Bauer, N., & Friedrich, C. M. (2023) · SN Computer Science · doi:10.1007/s42979-022-01563-6 · Cited by 11
Improving Disfluency Detection with Multi-Scale Self Attention and Contrastive Learning
Wang, P., Duan, C., Chen, M., & He, X. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096298 · Cited by 10
Bengali Fake Review Detection using Semi-supervised Generative Adversarial Networks
Shawon, M. T. R., Shahariar, G. M., Shah, F. M., Alam, M. S., & Mahbub, M. S. (2023) · doi:10.1109/icnlp58431.2023.00011 · Cited by 13
A recollect-tuning method for entity and relation extraction
Yi-zhao, W., Chen, Y., Qin, Y., Tang, R., & Zheng, Q. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.123000 · Cited by 11
Prompt-Based Label-Aware Framework for Few-Shot Multi-Label Text Classification
Thaminkaew, T., Lertvittayakumjorn, P., & Vateekul, P. (2024) · IEEE Access · doi:10.1109/access.2024.3367994 · Cited by 12
AI-accelerated therapeutic antibody development: practical insights
Santuari, L., Salvy, M. B., Xénarios, I., & Arpat, A. B. (2024) · Frontiers in Drug Discovery · doi:10.3389/fddsv.2024.1447867 · Cited by 10
John, A., Isnin, I. F., Madni, S. H. H., & Faheem, M. (2024) · IET Wireless Sensor Systems · doi:10.1049/wss2.12100 · Cited by 15
Predicting the geolocation of tweets using transformer models on customized data
Lutsai, K., & Lampert, C. H. (2024) · Journal of Spatial Information Science · doi:10.5311/josis.2024.29.295 · Cited by 19
URFM: A general Ultrasound Representation Foundation Model for advancing ultrasound image diagnosis
Kang, Q., Lao, Q., Gao, J. Y., Bao, W., He, Z., Du, C., Lu, Q., & Li, K. (2025) · iScience · doi:10.1016/j.isci.2025.112917 · Cited by 18
An Interpretable Deep Learning System for Automatically Scoring Request for Proposals
Maji, S., Appe, A., Bali, R., Chowdhury, A. G., Raghavendra, V. C., & Bhandaru, V. M. (2021) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai52525.2021.00136 · Cited by 11
HIT-SCIR at MRP 2020: Transition-based Parser and Iterative Inference Parser
Dou, L., Feng, Y., Ji, Y., Che, W., & Liu, T. (2020) · doi:10.18653/v1/2020.conll-shared.6 · Cited by 10
paht_nlp @ MEDIQA 2021: Multi-grained Query Focused Multi-Answer Summarization
Zhu, W., He, Y., Chai, L., Fan, Y., Ni, Y., Xie, G., & Wang, X. (2021) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2021.bionlp-1.10 · Cited by 10
Learning to Generate Questions by Learning to Recover Answer-containing Sentences
Back, S., Kedia, A., Chinthakindi, S. C., Lee, H., & Choo, J. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.132 · Cited by 11
Convolutions and Self-Attention: Re-interpreting Relative Positions in Pre-trained Language Models
Chang, T. A., Xu, Y., Xu, W., & Tu, Z. (2021) · doi:10.18653/v1/2021.acl-long.333 · Cited by 12
Pedersen, J. S., Laursen, M. S., Soguero-Ruíz, C., Savarimuthu, T. R., Hansen, R. S., & Vinholt, P. J. (2022) · IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI ...) · doi:10.1109/bhi56158.2022.9926955 · Cited by 10
Al-Hussaini, I., Nakajima An, D., Lee, A. J., Bi, S., & Mitchell, C. S. (2022) · 2022 IEEE International Conference on Big Data (Big Data) · doi:10.1109/bigdata55660.2022.10020807 · Cited by 10
Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning
Patra, B., Singhal, S., Huang, S., Chi, Z., Dong, L., Wei, F., Chaudhary, V., & Song, X. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.856 · Cited by 19
Label-Aware Hyperbolic Embeddings for Fine-grained Emotion Classification
Chen, C. Y., Hung, T. M., Hsu, Y.-L., & Ku, L. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.613 · Cited by 12
EM Pre-training for Multi-party Dialogue Response Generation
Li, Y., & Zhao, H. Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.7 · Cited by 10
Structure-Aware Language Model Pretraining Improves Dense Retrieval on Structured Data
Li, X., Liu, Z., Xiong, C., Shi, Y., Gu, Y., Liu, Z., & Yu, G. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.734 · Cited by 29
SpEL: Structured Prediction for Entity Linking
Shavarani, H. S., & Sarkar, A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.686 · Cited by 13
Oolong: Investigating What Makes Transfer Learning Hard with Controlled Studies
Wu, Z., Tamkin, A., & Papadimitriou, I. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.198 · Cited by 17
Did they answer? Subjective acts and intents in conversational discourse
Ferracane, E., Durrett, G., Li, J. J., & Erk, K. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.129 · Cited by 10
Kamigaito, H., & Hayashi, K. (2021) · doi:10.18653/v1/2021.acl-long.429 · Cited by 12
Shuffled-token Detection for Refining Pre-trained RoBERTa
Panda, S., Agrawal, A., Ha, J., & Bloch, B. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop (NAACL SRW) · doi:10.18653/v1/2021.naacl-srw.12 · Cited by 10
Hu, H., Zhou, H., Tian, Z., Zhang, Y., Patterson, Y., Li, Y., Nie, Y., & Richardson, K. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.331 · Cited by 11
Lacking the Embedding of a Word? Look it up into a Traditional Dictionary
Ruzzetti, E. S., Ranaldi, L., Mastromattei, M., Fallucchi, F., Scarpato, N., & Zanzotto, F. M. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.208 · Cited by 10
Extracting and Analyzing Inorganic Material Synthesis Procedures in the Literature
Makino, K., Kuniyoshi, F., Ozawa, J., & Miwa, M. (2022) · IEEE Access · doi:10.1109/access.2022.3160201 · Cited by 9
Predicting Human Psychometric Properties Using Computational Language Models
Laverghetta, A., Nighojkar, A., Mirzakhalov, J., & Licato, J. (2022) · Springer proceedings in mathematics & statistics · doi:10.1007/978-3-031-04572-1_12 · Cited by 11
Transformer based ensemble for emotion detection
Kane, A., Patankar, S., Khose, S., & Kirtane, N. (2022) · Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (WASSA) · doi:10.18653/v1/2022.wassa-1.25 · Cited by 14
Ji, Y., Chen, L., Dou, C., Ma, B., & Li, X. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.96 · Cited by 12
Shaheen, M., & Nigam, S. K. (2022) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2022.semeval-1.130 · Cited by 10
Clustering-Aware Negative Sampling for Unsupervised Sentence Representation
Deng, J., Wan, F., Yang, T., Quan, X., & Wang, R. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.555 · Cited by 11
PAD-Net: An Efficient Framework for Dynamic Networks
He, S., Ding, L., Dong, D., Liu, B., Yu, F., & Tao, D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.803 · Cited by 16
Back to the Future: Bidirectional Information Decoupling Network for Multi-turn Dialogue Modeling
Li, Y., Zhao, H. Y., & Zhang, Z. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.177 · Cited by 11
Steinberg, E., Jung, K., Fries, J., Corbin, C. K., Pfohl, S. R., & Shah, N. H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.05295 · Cited by 10
Montella, S., Fabre, B., Urvoy, T., Heinecke, J., & Rojas-Barahona, L. M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.00571 · Cited by 12
Identifying Machine-Paraphrased PlagiarismPreprint
Wahle, J. P., Ruas, T., Foltýnek, T., Meuschke, N., & Gipp, B. (2023) · doi:10.22541/au.167528156.61938925/v1 · Cited by 10
Modeling Adversarial Attack on Pre-trained Language Models as Sequential Decision Making
Fang, X., Cheng, S., Liu, Y., & Wang, W. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.461 · Cited by 10
Koltcov, S., Surkov, A., Filippov, V. A., & Ignatenko, V. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.1758 · Cited by 11
Towards smarter hiring: resume parsing and ranking with YOLOv5 and DistilBERT
Kinger, S., Kinger, D., Thakkar, S., & Bhake, D. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-18778-9 · Cited by 13
Recursive Label Attention Network for Nested Named Entity
Kim, H., & Kim, H. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.123657 · Cited by 9
A few-shot word-structure embedded model for bridge inspection reports learning
Wang, Y., Zhu, Y., Xiong, W., & Cai, C. S. (2024) · Advanced Engineering Informatics · doi:10.1016/j.aei.2024.102664 · Cited by 10
HarmonyNet: Navigating hate speech detection
Raza, S., & Chatrath, V. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2024.100098 · Cited by 11
Kim, K., Park, S., Min, J., Park, S., Kim, J. Y., Eun, J., Jung, K., Park, Y. E., Kim, E. H., Lee, E. Y., et al. (2024) · JMIR Medical Informatics · doi:10.2196/52897 · Cited by 10
Castells, P., & Jannach, D. (2024) · ACM eBooks · doi:10.1145/3674127.3674131 · Cited by 10
Mapping human security strategies to sustainable development goals: a machine learning approach
Koundouri, P., Aslanidis, P. C., Dellis, K., Plataniotis, A., & Feretzakis, G. (2025) · Discover Sustainability · doi:10.1007/s43621-025-00883-w · Cited by 12
Li, X., Li, J., Jiang, J., Pan, X., & Huang, X. (2025) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2025.104471 · Cited by 11
He, Y., Lv, X., Long, W., Zhai, S., Li, M., & Wen, Z. (2025) · Journal of Pharmaceutical Analysis · doi:10.1016/j.jpha.2025.101387 · Cited by 9
Thermostat: A Large Collection of NLP Model Explanations and Analysis Tools
Feldhus, N., Schwarzenberg, R., & Möller, S. (2021) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2021.emnlp-demo.11 · Cited by 9
CS-BERT: a pretrained model for customer service dialogues
Wang, P., Fang, J., & Reinspach, J. (2021) · Workshop on NLP for Conversational AI (NLP4ConvAI) · doi:10.18653/v1/2021.nlp4convai-1.13 · Cited by 9
Pre-trained Language Models for Tagalog with Multi-source Data
Jiang, S., Fu, Y., Lin, X., & Lin, N. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-88480-2_17 · Cited by 9
Both, M., Müller, J. M., & Diedrich, C. (2021) · at - Automatisierungstechnik · doi:10.1515/auto-2021-0050 · Cited by 12
On the effectiveness of pretrained models for API learning
Hadi, M. A., Yusuf, I. N. B., Thung, F., Luong, K., Jiang, L., Fard, F. H., & Lo, D. (2022) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1145/3524610.3527886 · Cited by 13
OPERA: Operation-Pivoted Discrete Reasoning over Text
Zhou, Y., Bao, J., Duan, C., Sun, H., Liang, J., Wang, Y., Zhao, J., Wu, Y., He, X., & Zhao, T. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.119 · Cited by 9
Shedding Light on the Dark Web: Authorship Attribution in Radical Forums
Ranaldi, L., Ranaldi, F., Fallucchi, F., & Zanzotto, F. M. (2022) · Information · doi:10.3390/info13090435 · Cited by 11
RoBERTa: language modelling in building Indonesian question-answering systems
Suwarningsih, W., Pramata, R. A., Rahadika, F. Y., & Purnomo, M. H. A. (2022) · TELKOMNIKA (Telecommunication Computing Electronics and Control) · doi:10.12928/telkomnika.v20i6.24248 · Cited by 14
Pugazhenthi, T., & Liang, H. (2022) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata55660.2022.10020689 · Cited by 9
Improving Pre-trained Language Models
Paaß, G., & Giesselbach, S. (2023) · Artificial intelligence: foundations, theory, and algorithms/Artificial intelligence: Foundations, theory, and algorithms · doi:10.1007/978-3-031-23190-2_3 · Cited by 9
Pre-trained Model Based Feature Envy Detection
Ma, W., Yu, Y., Ruan, X., & Cai, B. (2023) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr59073.2023.00065 · Cited by 9
AliBERT: A Pre-trained Language Model for French Biomedical Text
Berhe, A., Draznieks, G., Martenot, V., Masdeu, V., Davy, L., & Zucker, J. (2023) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2023.bionlp-1.19 · Cited by 12
AraMUS: Pushing the Limits of Data and Model Scale for Arabic Natural Language Processing
Alghamdi, A., Duan, X., Jiang, W., Wang, Z., Wu, Y., Xia, Q., Wang, Z., Zheng, Y., Rezagholizadeh, M., Huai, B., et al. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.181 · Cited by 9
Attributable and Scalable Opinion Summarization
Hosking, T., Tang, H., & Lapata, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.473 · Cited by 16
Zanwar, S., Wiechmann, D., Qiao, Y., & Kerz, E. (2022) · Workshop on Natural Language Processing and Computational Social Science (NLP+CSS) · doi:10.18653/v1/2022.nlpcss-1.1 · Cited by 18
Event-Centric Question Answering via Contrastive Learning and Invertible Event Transformation
Lu, J., Tan, X., Pergola, G., Gui, L., & He, Y. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.176 · Cited by 9
Genetic Prompt Search via Exploiting Language Model Probabilities
Zhao, J., Wang, Z., & Yang, F. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/588 · Cited by 13
An Ensemble Approach to Question Classification: Integrating Electra Transformer, GloVe, and LSTM
Aburass, S., Dorgham, O. M., & Rumman, M. A. (2024) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2024.0150148 · Cited by 14
Tolomei, G., Campagnano, C., Silvestri, F., & Trappolini, G. (2023) · IEEE International Conference on Cognitive Machine Intelligence (CogMI) · doi:10.1109/cogmi58952.2023.00027 · Cited by 11
Attacking Open-domain Question Answering by Injecting Misinformation
Pan, L., Chen, W., Kan, M.-Y., & Wang, W. Y. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.35 · Cited by 9
Xu, L., Zhang, J., Zhang, C., Zheng, X., Du, Z., & Xue, X. (2024) · Geo-spatial Information Science · doi:10.1080/10095020.2024.2354229 · Cited by 10
Comparative Analysis of Transformer based Language Models
Pathak, A. (2021) · doi:10.5121/csit.2021.110111 · Cited by 10
Team “NoConflict” at CASE 2021 Task 1: Pretraining for Sentence-Level Protest Event Detection
Hu, T., & Stoehr, N. (2021) · Workshop on Challenges and Applications of Automated Extraction of Socio-political Events from Text (CASE) · doi:10.18653/v1/2021.case-1.20 · Cited by 10
Probing Pre-trained Language Models for Semantic Attributes and their Values
Beloucif, M., & Biemann, C. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.218 · Cited by 10
ANNA”:" Enhanced Language Representation for Question Answering
Jun, C., Jang, H., Sim, M., Kim, H. J., Choi, J., Min, K., & Bae, K. (2022) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2022.repl4nlp-1.13 · Cited by 10
A Survey on Machine Learning-based Automated Software Bug Report Classification
Köksal, Ö., & Öztürk, C. E. (2022) · International Symposium on Multidisciplinary Studies and Innovative Technologies (ISMSIT) · doi:10.1109/ismsit56059.2022.9932822 · Cited by 12
A Heterogeneous Graph to Abstract Syntax Tree Framework for Text-to-SQL
Cao, R., Chen, L., Li, J., Zhang, H., Xu, H., Zhang, W., & Yu, K. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3298895 · Cited by 12
ANALOGICAL - A Novel Benchmark for Long Text Analogy Evaluation in Large Language Models
Wijesiriwardene, T., Wickramarachchi, R., Gajera, B., Gowaikar, S., Gupta, C., Chadha, A. R., Reganti, A., Sheth, A., & Das, A. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.218 · Cited by 10
Feng, S., Tan, Z., Zhang, W., Lei, Z., & Tsvetkov, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.118 · Cited by 10
Raychawdhary, N., Das, A. K., Dozier, G. V., & Seals, C. (2023) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2023.semeval-1.208 · Cited by 10
Yang, K., Jang, W., & Cho, W. I. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.525 · Cited by 13
An Extended Sequence Tagging Vocabulary for Grammatical Error Correction
Mesham, S., Bryant, C., Rei, M., & Yuan, Z. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.119 · Cited by 9
Biaffine Dependency and Semantic Graph Parsing for EnhancedUniversal Dependencies
Attardi, G., Sartiano, D., & Simi, M. (2021) · International Workshop/Conference on Parsing Technologies · doi:10.18653/v1/2021.iwpt-1.19 · Cited by 9
I like fish, especially dolphins: Addressing Contradictions in Dialogue ModelingPreprint
Nie, Y., Williamson, M., Bansal, M., Kiela, D., & Weston, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.13391 · Cited by 9
LazyFormer: Self Attention with Lazy UpdatePreprint
Ying, C., Ke, G., He, D., & Liu, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.12702 · Cited by 24
Not All Attention Is All You NeedPreprint
Wu, H., Zhao, H., & Zhang, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.04692 · Cited by 13
SemEval-2022 Task 3: PreTENS-Evaluating Neural Networks on Presuppositional Semantic Knowledge
Zamparelli, R., Chowdhury, S. A., Brunato⋄, D., Chesi, C., Dell’Orletta⋄, F., Hasan, M. A., & Venturi⋄, G. (2022) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2022.semeval-1.29 · Cited by 8
Exploring the Power of Cross-Contextual Large Language Model in Mimic Emotion Prediction
Yi, G., Yang, Y., Pan, Y., Cao, Y., Yao, J., Lv, X., Fan, C., Lv, Z., Tao, J., Liang, S., et al. (2023) · doi:10.1145/3606039.3613109 · Cited by 8
ViT-TTS: Visual Text-to-Speech with Scalable Diffusion Transformer
Liu, H., Huang, R., Lin, X., Xu, W., Zheng, M., Chen, H., He, J., & Zhao, Z. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.990 · Cited by 35
Ahanger, M. M., Wani, M. A., & Palade, V. (2024) · Knowledge · doi:10.3390/knowledge4030022 · Cited by 10
Knowledge-Enhanced Conversational Agents
Caffaro, F., & Rizzo, G. F. (2024) · Journal of Computer Science and Technology · doi:10.1007/s11390-024-2883-4 · Cited by 10
Software bug prediction using graph neural networks and graph-based text representations
Siachos, I., Kanakaris, N., & Karacapilidis, N. I. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125290 · Cited by 11
Performance and sustainability of BERT derivatives in dyadic data
Escarda-Fernández, M., Eiras‐Franco, C., Cancela, B., Guijarro‐Berdiñas, B., & Alonso‐Betanzos, A. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125647 · Cited by 11
Metadata
- Authors
- Clark, K., Luong, M.-T., Le, Q. V., & Manning, C. D.
- Affiliations
- Stanford University (US)
- Google (United States) (US)
- ORCID
- Le, Quoc V.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2003.10555
- OpenAlex
- W2996035354
- Cited by
- 1,844
- References
- 48
- Citations by year
- 2026: 13 · 2025: 129 · 2024: 240 · 2023: 572 · 2022: 415 · 2021: 375 · 2020: 100
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Learning Representations
- Funders
- Canadian Institute for Advanced Research
- Programa Operacional Regional do Centro (Portugal 2020)
- Fundação para a Ciência e a Tecnologia (PT2020, -01-0247-, 00326, UID/CEC/00326/2020)
- European Regional Development Fund (PT2020)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- Electra, masked language modeling, text encoder, sample efficiency, context representation, natural language understanding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
