The DOI matches its DataCite record; 48 of the 48 references it cites were found in a registry or an index.
ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
Open accessAbstract
Masked language modeling (MLM) pre-training methods such as BERT corrupt the input by replacing some tokens with [MASK] and then train a model to reconstruct the original tokens. While they produce good results when transferred to downstream NLP tasks, they generally require large amounts of compute to be effective. As an alternative, we propose a more sample-efficient pre-training task called replaced token detection. Instead of masking the input, our approach corrupts it by replacing some tokens with plausible alternatives sampled from a small generator network. Then, instead of training a model that predicts the original identities of the corrupted tokens, we train a discriminative model that predicts whether each token in the corrupted input was replaced by a generator sample or not. Thorough experiments demonstrate this new pre-training task is more efficient than MLM because the task is defined over all input tokens rather than just the small subset that was masked out. As a result, the contextual representations learned by our approach substantially outperform the ones learned by BERT given the same model size, data, and compute. The gains are particularly strong for small models; for example, we train a model on one GPU for 4 days that outperforms GPT (trained using 30x more compute) on the GLUE natural language understanding benchmark. Our approach also works well at scale, where it performs comparably to RoBERTa and XLNet while using less than 1/4 of their compute and outperforms them when using the same amount of compute.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We will use FLOPs, as they allow us to estimate the needed energy relating hardware FLOPS with required FLOPs for a model [Hollemans, 2018, Clark et al., 2020].”
From Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning (2023) · cites this paper as Clark et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In NLP, they count FLOPs as single mathematical operations and not as a single hardware instructions (like in CV).”
From Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning (2023) · cites this paper as Clark et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The authors corrupted the tokens with plausible alternatives sampled from a small generator network that improves the model performance significantly.”
From A Review on Deep-Learning-Based Cyberbullying Detection (2023) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“but we ultimately found out this information explained in [Clark et al., 2020].”
From Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning (2023) · cites this paper as Clark et al., 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Self-citationEfficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
GAN（Generative Adversarial Nets）
淳司, 柴. (2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 20,507
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Unsupervised representation learning with deep convolutional generative adversarial networks
Radford, A., Metz, L., & Chintala, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06434 · Cited by 12,235
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 8,608
Research works printed but not linked: 4
- Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS , 2014.
- Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942 , 2019. · arXiv
- Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. English gigaword, fifth edition. Technical report, Linguistic Data Consortium, Philadelphia, 2011.
- Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. MobileBERT: Task-agnostic compression of bert for resource limited devices, 2019b. URL https: //openreview.net/forum?id=SJxjVaNKwB .
Cited by
Sorted by citations
Items 501–600 of 1,010
Deep learning approaches to lexical simplification: A survey
North, K., Ranasinghe, T., Shardlow, M., & Zampieri, M. (2024) · Journal of Intelligent Information Systems · doi:10.1007/s10844-024-00882-9 · Cited by 31
Asim, M. N., Ibrahim, M. A., Zaib, A., & Dengel, A. R. (2025) · Frontiers in Medicine · doi:10.3389/fmed.2025.1503229 · Cited by 19
Towards Understanding and Answering Comparative Questions
Bondarenko, A., Ajjour, Y., Dittmar, V., Homann, N., Braslavski, P., & Hagen, M. (2022) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3488560.3498534 · Cited by 19
Weller, O., Seppi, K. D., & Gardner, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-short.30 · Cited by 24
Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation BenchmarkingPreprint
Ma, Z., Ethayarajh, K., Thrush, T., Jain, S., Wu, L., Jia, R., Potts, C., Williams, A., & Kiela, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.06052 · Cited by 21
Hoppe-Ludwig, C., Pelkmann, D., Migenda, N., Hötte, D., & Schenck, W. (2021) · IEEE International Conference on Artificial Intelligence and Knowledge Engineering (AIKE) · doi:10.1109/aike52691.2021.00011 · Cited by 19
ILDAE: Instance-Level Difficulty Analysis of Evaluation Data
Varshney, N., Mishra, S. R., & Baral, C. R. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.240 · Cited by 19
Chambon, P., Cook, T. S., & Langlotz, C. P. (2022) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-022-00714-8 · Cited by 18
Feature fusion via multi-target learning for ancient artwork captioning
Liu, F., Zhang, M., Zheng, B., Cui, S., Ma, W., & Liu, Z. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101811 · Cited by 16
Classifying European Court of Human Rights Cases Using Transformer-Based Techniques
Imran, A. S., Hodnefjeld, H., Kastrati, Z., Fatima, N., Daudpota, S. M., & Wani, M. A. (2023) · IEEE Access · doi:10.1109/access.2023.3279034 · Cited by 25
Mayil, V. V., & Jeyalakshmi, T. R. (2023) · International Conference on Artificial Intelligence and Signal Processing (AISP) · doi:10.1109/aisp57993.2023.10134937 · Cited by 16
Hamid, O. H. (2023) · International Conference on Information Technology Trends (ITT) · doi:10.1109/itt59889.2023.10184233 · Cited by 16
Open-domain Question Answering via Chain of Reasoning over Heterogeneous Knowledge
Ma, K., Cheng, H., Liu, X., Nyberg, E., & Gao, J. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.392 · Cited by 23
Downstream Datasets Make Surprisingly Good Pretraining Corpora
Krishna, K., Garg, S., Bigham, J. P., & Lipton, Z. C. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.682 · Cited by 27
Chain-of-Skills: A Configurable Model for Open-Domain Question Answering
Ma, K., Cheng, H., Zhang, Y., Liu, X., Nyberg, E., & Gao, J. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.89 · Cited by 29
Li, Y., Du, M., Wang, X., & Wang, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.797 · Cited by 18
Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systems
Varshney, N., & Baral, C. R. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.756 · Cited by 39
Misra, K., Rayz, J. T., & Ettinger, A. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.213 · Cited by 32
Aggarwal, S., & Vishwakarma, D. K. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124278 · Cited by 17
BiRNA-BERT allows efficient RNA language modeling with adaptive tokenization
Tahmid, M. T., Shahgir, H. S., Mahbub, S., Dong, Y., & Bayzid, M. S. (2025) · Communications Biology · doi:10.1038/s42003-025-08982-0 · Cited by 16
Kang, K., Jin, K., Yang, S., Jang, S., Choo, J., & Kim, Y. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.317 · Cited by 19
A Unified Pretraining Framework for Passage Ranking and Expansion
Yan, M., Li, C., Bi, B., Wang, W., & Huang, S. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i5.16584 · Cited by 16
Marreddy, M., Oota, S. R., Vakada, L. S., Chinni, V. C., & Mamidi, R. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9534382 · Cited by 18
Spurious Correlations in Reference-Free Evaluation of Text Generation
Durmus, E., Ladhak, F., & Hashimoto, T. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.102 · Cited by 16
Molecular Graph Contrastive Learning with Parameterized Explainable Augmentations
Wang, Y., Min, Y., Shao, E., & Wu, J. Y. (2021) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm52615.2021.9669302 · Cited by 16
Xu, H., Chen, Y., Liu, Z., Wen, Y., & Yuan, X. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/615 · Cited by 16
Hassid, M., Peng, H. M., Rotem, D., Kasai, J., Montero, I., Smith, N. A., & Schwartz, R. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.101 · Cited by 23
Discriminative Adversarial Search for Abstractive SummarizationPreprint
Scialom, T., Dray, P.-A., Lamprier, S., Piwowarski, B., & Staiano, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.10375 · Cited by 18
TextFlint: Unified Multilingual Robustness Evaluation Toolkit for Natural Language ProcessingPreprint
Gui, T., Wang, X., Zhang, Q., Liu, Q., Zou, Y., Zhou, X., Zheng, R., Zhang, C., Wu, Q., Ye, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.11441 · Cited by 31
Zhao, X., Li, X., Jiang, R., & Tang, B. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102745 · Cited by 18
AraScore: A deep learning-based system for Arabic short answer scoring
Nael, O., ELmanyalawy, Y., & Sharaf, N. (2022) · Array · doi:10.1016/j.array.2021.100109 · Cited by 16
Wang, X., Xu, X., Tong, W., Liu, Q., & Liu, Z. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.999289 · Cited by 16
Search-engine-augmented dialogue response generation with cheaply supervised query production
Wang, A., Song, L., Liu, Q., Mi, H., Wang, L., Tu, Z., Su, J., & Yu, D. (2023) · Artificial Intelligence · doi:10.1016/j.artint.2023.103874 · Cited by 16
KHAN: Knowledge-Aware Hierarchical Attention Networks for Accurate Political Stance Prediction
Ko, Y., Ryu, S., Han, S., Jeon, Y., Kim, J., Park, S., Han, K., Tong, H., & Kim, S. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583300 · Cited by 22
Reshi, J. A., & Ali, R. (2023) · International Journal of Interactive Multimedia and Artificial Intelligence · doi:10.9781/ijimai.2023.02.007 · Cited by 18
Monolingual, multilingual and cross-lingual code comment classification
Kostić, M., Batanović, V., & Nikolić, B. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106485 · Cited by 15
Sentiment Analysis with Neural Models for Hungarian
Laki, L. J., & Yang, Z. G. (2023) · Acta Polytechnica Hungarica · doi:10.12700/aph.20.5.2023.5.8 · Cited by 21
SeeGULL: A Stereotype Benchmark with Broad Geo-Cultural Coverage Leveraging Generative Models
Jha, A., Davani, A. M., Reddy, C. K. A., Dave, S., Prabhakaran, V., & Dev, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.548 · Cited by 43
Learning Better Masking for Better Language Model Pre-training
Yang, D., Zhang, Z., & Zhao, H. Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.400 · Cited by 19
Pre-training Transformer Models with Sentence-Level Objectives for Answer Sentence Selection
Liello, L. D., Garg, S., Soldaini, L., & Moschitti, A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.810 · Cited by 17
Oliveira, R., Sörensen, K., & Martens, D. (2023) · European Journal of Operational Research · doi:10.1016/j.ejor.2023.08.031 · Cited by 16
Chang, H., Xu, H., van Genabith, J., Xiong, D., & Zan, H. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3310879 · Cited by 16
Understanding Transformer Memorization Recall Through Idioms
Haviv, A., Cohen, I., Gidron, J., Schuster, R., Goldberg, Y., & Geva, M. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.19 · Cited by 35
Deep Modular Co-Attention Shifting Network for Multimodal Sentiment Analysis
Shi, P., Hu, M. Q., Shi, X., & Ren, F. (2023) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3634706 · Cited by 15
Zhang, X., Zeng, Y., Zhang, J., & Li, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.40 · Cited by 21
Hoque, M. N., & Seddiqui, M. H. (2023) · International Conference on Computer and Information Technology (ICCIT) · doi:10.1109/iccit60459.2023.10441412 · Cited by 17
Contrastive Tuning: A Little Help to Make Masked Autoencoders Forget
Lehner, J., Alkin, B., Fürst, A., Rumetshofer, E., Miklautz, L., & Hochreiter, S. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i4.28078 · Cited by 24
Recognition of propaganda techniques in newspaper texts: Fusion of content and style analysis
Horák, A., Sabol, R., Herman, O., & Baisa, V. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124085 · Cited by 20
Machine learning for predicting protein properties: A comprehensive review
Wang, Y., Zhang, Y., Zhan, X., He, Y., Yang, Y., Cheng, L., & Alghazzawi, D. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128103 · Cited by 15
Hong, S.-K., Jang, J.-S., & Kwon, H.-Y. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.112404 · Cited by 18
StyleDGPT: Stylized Response Generation with Pre-trained Language Models
Yang, Z., Wu, W., Xu, C., Liang, X., Bai, J., Wang, L., Wang, W., & Li, Z. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.140 · Cited by 20
An Extensive Study on Pretrained Models for Natural Language Processing Based on Transformers
Ramprasath, M., Dhanasekaran, K., Karthick, T., Velumani, R., & Sudhakaran, P. (2022) · International Conference on Electronics and Renewable Systems (ICEARS) · doi:10.1109/icears53579.2022.9752241 · Cited by 19
GAN-BElectra: Enhanced Multi-class Sentiment Analysis with Limited Labeled Data
Riyadh, M., & Shafiq, M. O. (2022) · Applied Artificial Intelligence · doi:10.1080/08839514.2022.2083794 · Cited by 15
Towards Robust Ranker for Text Retrieval
Zhou, Y., Shen, T., Geng, X., Tao, C., Xu, C., Long, G., Jiao, B., & Jiang, D. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.332 · Cited by 20
Expose Backdoors on the Way: A Feature-Based Efficient Defense against Textual Backdoor Attacks
Chen, S., Yang, W., Zhang, Z., Bi, X., & Sun, X. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.47 · Cited by 17
Coreference Resolution for the Biomedical Domain: A Survey
Lu, P., & Poesio, M. (2021) · Workshop on Computational Models of Reference, Anaphora and Coreference (CRAC) · doi:10.18653/v1/2021.crac-1.2 · Cited by 18
ASR Rescoring and Confidence Estimation with Electra
Futami, H., Inaguma, H., Mimura, M., Sakai, S., & Kawahara, T. (2021) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru51503.2021.9688175 · Cited by 16
An Overview on Language Models: Recent Developments and OutlookPreprint
Wei, C., Wang, Y.-C., Wang, B., & Kuo, C. ‐. J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.05759 · Cited by 15
Improved multi-label hierarchical patent classification using LLMs
Rafieian, B., & Vázquez, P. (2025) · World Patent Information · doi:10.1016/j.wpi.2025.102356 · Cited by 14
Learning Realistic Mutations: Bug Creation for Neural Bug Detectors
Richter, C., & Wehrheim, H. (2022) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst53961.2022.00027 · Cited by 19
Alajrami, A., & Αλέτρας, Ν. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-short.16 · Cited by 23
Token Selection is a Simple Booster for Vision Transformers
Zhou, D., Hou, Q., Yang, L., Jin, X., & Feng, J. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3208922 · Cited by 15
Where did you tweet from? Inferring the origin locations of tweets based on contextual information
Lamsal, R., Harwood, A., & Rodriguez, M. A. (2022) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata55660.2022.10020460 · Cited by 17
Lopes, A., Carbonera, J. L., Schmidt, D., Garcia, L. F., Rodrigues, F. H., & Abel, M. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110385 · Cited by 17
ADMit: Improving NER in automotive domain with domain adversarial training and multi-task learning
Park, C., Jeong, S., & Kim, J. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120007 · Cited by 14
SKDBERT: Compressing BERT via Stochastic Knowledge Distillation
Ding, Z., Guoqing, J., Zhang, S., Guo, L., & Lin, W. T. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i6.25902 · Cited by 14
ELECTRA-based graph network model for multi-hop question answering
Zhu, P., Yuan, Y., & Chen, L. (2023) · Journal of Intelligent Information Systems · doi:10.1007/s10844-023-00800-5 · Cited by 15
CAT: A Contextualized Conceptualization and Instantiation Framework for Commonsense Reasoning
Wang, W., Fang, T., Xu, B., Bo, C. Y. L., Song, Y., & Chen, L. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.733 · Cited by 17
Unsupervised Boundary-Aware Language Model Pretraining for Chinese Sequence Labeling
Jiang, P., Long, D., Zhang, Y., Xie, P., Zhang, M., & Zhang, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.34 · Cited by 14
CQR-SQL: Conversational Question Reformulation Enhanced Context-Dependent Text-to-SQL Parsers
Xiao, D., Chai, L., Zhang, Q., Zhao, Y., Li, Z., & Cao, Y. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.150 · Cited by 15
Doc2Bot: Accessing Heterogeneous Documents via Conversational Bots
Fu, H., Zhang, Y., Yu, H., Sun, J., Huang, F., Si, L., Li, Y., & Nguyen, C. T. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.131 · Cited by 15
ViLEM: Visual-Language Error Modeling for Image-Text Retrieval
Chen, Y., Ma, Z., Zhang, Z., Qi, Z., Yuan, C., Shan, Y., Li, B., Hu, W., Qie, X., & Wu, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01060 · Cited by 15
Learning Representations on Logs for AIOps
Gupta, P., Kumar, H., Kar, D., Bhukar, K., Aggarwal, P., & Mohapatra, P. (2023) · IEEE ... International Conference on Cloud Computing · doi:10.1109/cloud60044.2023.00026 · Cited by 18
On the Zero-Shot Generalization of Machine-Generated Text Detectors
Xiao, P., Zhang, J., Han, X., Tsvetkov, Y., & He, T. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.318 · Cited by 24
SourceP: Detecting Ponzi Schemes on Ethereum with Source Code
Lu, P., Cai, L., & Yin, K. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448439 · Cited by 14
Bengali fake reviews: A benchmark dataset and detection system
Shahariar, G. M., Shawon, M. T. R., Shah, F. M., Alam, M. S., & Mahbub, M. S. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127732 · Cited by 18
Chuang, C.-C., Liu, Y., & Ou, Y. (2024) · International Journal of Biological Macromolecules · doi:10.1016/j.ijbiomac.2024.136252 · Cited by 17
Explainable Pre-Trained Language Models for Sentiment Analysis in Low-Resourced Languages
Mabokela, K. R., Primus, M., & Çelik, T. (2024) · Big Data and Cognitive Computing · doi:10.3390/bdcc8110160 · Cited by 24
Inherent risks identification in a contract document through automated rule generation
Kim, J., Kwon, B., Lee, J., & Mun, D. (2025) · Automation in Construction · doi:10.1016/j.autcon.2025.106044 · Cited by 15
Large language models transform biological research: from architecture to utilization
Wang, T., & Luo, Z. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4466-3 · Cited by 14
Making Punctuation Restoration Robust and Fast with Multi-Task Learning and Knowledge Distillation
Hentschel, M., Tsunoo, E., & Okuda, T. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414518 · Cited by 15
Park, C., Eo, S., Moon, H., & Lim, H. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-industry.13 · Cited by 15
Deciphering Implicit Hate: Evaluating Automated Detection Algorithms for Multimodal Hate
Botelho, A., Hale, S. A., & Vidgen, B. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.166 · Cited by 14
Improving task-agnostic BERT distillation with layer mapping search
Jiao, X., Chang, H., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F., & Liu, Q. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.07.050 · Cited by 14
Siamese BERT-Based Model for Web Search Relevance Ranking Evaluated on a New Czech Dataset
Kocián, M., Náplava, J., Stancl, D., & Kadlec, V. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21502 · Cited by 7
Nassar, M., Rogers, A. B., Talo', F., Sanchez, S., Shafique, Z., Finn, R. D., & McEntyre, J. (2022) · GigaScience · doi:10.1093/gigascience/giac077 · Cited by 13
Introducing UberText 2.0: A Corpus of Modern Ukrainian at Scale
Chaplynskyi, D. (2023) · Ukrainian Natural Language Processing Workshop (UNLP) · doi:10.18653/v1/2023.unlp-1.1 · Cited by 16
Cook-Gen: Robust Generative Modeling of Cooking Actions from Recipes
Venkataramanan, R., Roy, K., Raj, K., Prasad, R., Zi, Y., Narayanan, V., & Sheth, A. (2023) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc53992.2023.10394432 · Cited by 14
Efficient Training of Visual Transformers with Small DatasetsPreprint
Liu, Y., Sangineto, E., Bi, W., Sebe, N., Lepri, B., & Nadai, M. D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.03746 · Cited by 140
Kedia, A., Zaidi, M. A., & Lee, H. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.285 · Cited by 14
Dialogue-oriented Pre-training
Xu, Y., & Zhao, H. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.235 · Cited by 18
Keersmaekers, A. (2021) · Workshop on Computational Approaches to Historical Language Change (LChange) · doi:10.18653/v1/2021.lchange-1.6 · Cited by 13
SemEval-2021 Task 4: Reading Comprehension of Abstract Meaning
Zheng, B., Yang, X., Ruan, Y.-P., Ling, Z.-H., Liu, Q., Wei, S., & Zhu, X. (2021) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2021.semeval-1.4 · Cited by 15
Ön Eğitimli Dil Modelleri Kullanarak Türkçe Tweetlerden Cinsiyet Tespiti
İlhami, S. E. L., & Hanbay, D. (2021) · Fırat Üniversitesi Mühendislik Bilimleri Dergisi · doi:10.35234/fumbd.929133 · Cited by 13
Modarressi, A., Fayyaz, M., Yaghoobzadeh, Y., & Pilehvar, M. T. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.19 · Cited by 20
Comparative Analysis of Transformer based Models for Question Answering
Rawat, A., & Samant, S. S. (2022) · International Conference on Innovative Sustainable Computational Technologies (CISCT) · doi:10.1109/cisct55310.2022.10046525 · Cited by 16
Masked Siamese Prompt Tuning for Few-Shot Natural Language Understanding
Ni, S., & Kao, H. (2023) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3275132 · Cited by 14
On the Blind Spots of Model-Based Evaluation Metrics for Text Generation
He, T., Zhang, J., Wang, T., Kumar, S., Cho, K., Glass, J. R., & Tsvetkov, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.674 · Cited by 37
Beermann, J., Benfer, R., Both, M., Müller, J. M., & Diedrich, C. (2023) · IEEE International Conference on Industrial Informatics (INDIN) · doi:10.1109/indin51400.2023.10218154 · Cited by 14
Metadata
- Authors
- Clark, K., Luong, M.-T., Le, Q. V., & Manning, C. D.
- Affiliations
- Stanford University (US)
- Google (United States) (US)
- ORCID
- Le, Quoc V.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2003.10555
- OpenAlex
- W2996035354
- Cited by
- 1,844
- References
- 48
- Citations by year
- 2026: 13 · 2025: 129 · 2024: 240 · 2023: 572 · 2022: 415 · 2021: 375 · 2020: 100
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Learning Representations
- Funders
- Canadian Institute for Advanced Research
- Programa Operacional Regional do Centro (Portugal 2020)
- Fundação para a Ciência e a Tecnologia (PT2020, -01-0247-, 00326, UID/CEC/00326/2020)
- European Regional Development Fund (PT2020)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- Electra, masked language modeling, text encoder, sample efficiency, context representation, natural language understanding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
