The DOI matches its DataCite record; 48 of the 48 references it cites were found in a registry or an index.
ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
Open accessAbstract
Masked language modeling (MLM) pre-training methods such as BERT corrupt the input by replacing some tokens with [MASK] and then train a model to reconstruct the original tokens. While they produce good results when transferred to downstream NLP tasks, they generally require large amounts of compute to be effective. As an alternative, we propose a more sample-efficient pre-training task called replaced token detection. Instead of masking the input, our approach corrupts it by replacing some tokens with plausible alternatives sampled from a small generator network. Then, instead of training a model that predicts the original identities of the corrupted tokens, we train a discriminative model that predicts whether each token in the corrupted input was replaced by a generator sample or not. Thorough experiments demonstrate this new pre-training task is more efficient than MLM because the task is defined over all input tokens rather than just the small subset that was masked out. As a result, the contextual representations learned by our approach substantially outperform the ones learned by BERT given the same model size, data, and compute. The gains are particularly strong for small models; for example, we train a model on one GPU for 4 days that outperforms GPT (trained using 30x more compute) on the GLUE natural language understanding benchmark. Our approach also works well at scale, where it performs comparably to RoBERTa and XLNet while using less than 1/4 of their compute and outperforms them when using the same amount of compute.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We will use FLOPs, as they allow us to estimate the needed energy relating hardware FLOPS with required FLOPs for a model [Hollemans, 2018, Clark et al., 2020].”
From Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning (2023) · cites this paper as Clark et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In NLP, they count FLOPs as single mathematical operations and not as a single hardware instructions (like in CV).”
From Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning (2023) · cites this paper as Clark et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The authors corrupted the tokens with plausible alternatives sampled from a small generator network that improves the model performance significantly.”
From A Review on Deep-Learning-Based Cyberbullying Detection (2023) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“but we ultimately found out this information explained in [Clark et al., 2020].”
From Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning (2023) · cites this paper as Clark et al., 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Self-citationEfficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
GAN（Generative Adversarial Nets）
淳司, 柴. (2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 20,507
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Unsupervised representation learning with deep convolutional generative adversarial networks
Radford, A., Metz, L., & Chintala, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06434 · Cited by 12,235
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 8,608
Research works printed but not linked: 4
- Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS , 2014.
- Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942 , 2019. · arXiv
- Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. English gigaword, fifth edition. Technical report, Linguistic Data Consortium, Philadelphia, 2011.
- Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. MobileBERT: Task-agnostic compression of bert for resource limited devices, 2019b. URL https: //openreview.net/forum?id=SJxjVaNKwB .
Cited by
Sorted by citations
Items 801–900 of 1,010
Code LLMs: A Taxonomy-based Survey
Raihan, N., Newman, C. D., & Zampieri, M. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10826108 · Cited by 11
Entity-enhanced BERT for medical specialty prediction based on clinical questionnaire data
Lee, S., Han, Y. J., Park, H., Lee, B., Son, D., Kim, S., Yang, H., Han, T., Kim, E., & Han, S. W. (2025) · PLoS ONE · doi:10.1371/journal.pone.0317795 · Cited by 11
Shetty, A. M., Manjaiah, D. H., & Aljunid, M. F. (2025) · ETRI Journal · doi:10.4218/etrij.2024-0318 · Cited by 10
Disclosure Dynamism: TCFD Aligned Climate Claims of UK Corporates
Cortés, D. G., Lodh, S., Akbar, S., & Nandy, M. (2025) · Business Strategy and the Environment · doi:10.1002/bse.4189 · Cited by 10
Graph-aware pre-trained language model for political sentiment analysis in Filipino social media
Aquino, J. A., Liew, D.-J., & Chang, Y. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.110317 · Cited by 10
Pan, Q., Ng, S. T., Peng, F., & Dong, Y.-H. (2025) · Tunnelling and Underground Space Technology · doi:10.1016/j.tust.2025.106710 · Cited by 8
Hu, M., Yang, Y., Pan, D., Guo, Z., Xiao, L., Lin, D., & Zhao, S. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103376 · Cited by 9
Question Answering System with Indic multilingual-BERT
Jha, B. K., Akana, C. M. V. S., & Anand, R. (2021) · International Conference on Computing Methodologies and Communication (ICCMC) · doi:10.1109/iccmc51019.2021.9418387 · Cited by 9
Ensemble ALBERT and RoBERTa for Span Prediction in Question Answering
Bachina, S., Balumuri, S., & Kamath, S. S. (2021) · doi:10.18653/v1/2021.dialdoc-1.9 · Cited by 8
Wang, J., Bao, R., Zhang, Z., & Zhao, H. Y. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.73 · Cited by 9
Comparative Analysis of Transformers to Support Fine-Grained Emotion Detection in Short-Text Data
Frye, R. H., & Wilson, D. C. (2022) · Proceedings of the ... International Florida Artificial Intelligence Research Society Conference · doi:10.32473/flairs.v35i.130612 · Cited by 9
Lin, S.-J., Yeh, W.-C., Chiu, Y.-W., Chang, Y., Hsu, M., Chen, Y.-S., & Hsu, W. (2022) · Database · doi:10.1093/database/baac056 · Cited by 9
GanLM: Encoder-Decoder Pre-training with an Auxiliary Discriminator
Yang, J. Y., Ma, S., Dong, L., Huang, S., Huang, H., Yin, Y., Zhang, D., Yang, L., Wei, F., & Li, Z. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.522 · Cited by 13
Cross-Lingual Knowledge Distillation for Answer Sentence Selection in Low-Resource Languages
Gupta, S., Matsubara, Y., Chadha, A., & Moschitti, A. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.885 · Cited by 9
Wang, X., Weissweiler, L., Schütze, H., & Plank, B. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.157 · Cited by 11
Few-shot Reranking for Multi-hop QA via Language Model Prompting
Khalifa, M., Logeswaran, L., Lee, M., Lee, H., & Wang, L. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.885 · Cited by 18
TABS: Efficient Textual Adversarial Attack for Pre-trained NL Code Model Using Semantic Beam Search
Choi, Y., Kim, H., & Lee, J.-H. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.369 · Cited by 8
Seeberger, P., & Riedhammer, K. (2022) · Workshop on NLP for Positive Impact (NLP4PI) · doi:10.18653/v1/2022.nlp4pi-1.9 · Cited by 8
Robust Integration of Contextual Information for Cross-Target Stance Detection
Beck, T., Waldis, A., & Gurevych, I. (2023) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2023.starsem-1.43 · Cited by 10
Optimizing Large Language Models on Multi-Core CPUs: A Case Study of the BERT Model
Zhao, L., Gao, W., & Fang, J. (2024) · Applied Sciences · doi:10.3390/app14062364 · Cited by 10
Domain-Aware Fine-Tuning: Enhancing Neural Network Adaptability
Ha, S., Jeong, S., & Lee, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i11.29116 · Cited by 8
Software Defect Prediction via Generative Adversarial Networks and Pre-Trained Model
Song, W., Gan, L., & Bao, T. (2024) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2024.01503119 · Cited by 8
Wu, Y. Q., Jin, H., Wang, X., & Liu, N. (2024) · IEEE Transactions on Engineering Management · doi:10.1109/tem.2024.3384260 · Cited by 8
Saleem, S., Hasan, N., Khattar, A., Jain, P. R., Gupta, T. K., & Mehrotra, M. (2024) · IEEE Access · doi:10.1109/access.2024.3478790 · Cited by 11
Infusing Sequential Information into Conditional Masked Translation Model with Self-Review Mechanism
Xie, P., Cui, Z. M., Chen, X., Hu, X., Cui, J., & Wang, B. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.2 · Cited by 9
Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting
Zhou, W., Ge, T., Xu, C., Xu, K., & Wei, F. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.45 · Cited by 13
Keep Learning: Self-supervised Meta-learning for Learning from Inference
Kedia, A., & Chinthakindi, S. C. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.6 · Cited by 7
Pre-training Universal Language Representation
Li, Y., & Zhao, H. (2021) · doi:10.18653/v1/2021.acl-long.398 · Cited by 10
Xie, Y., Lai, Y.-A., Xiong, Y., Zhang, Y., & Soatto, S. (2021) · doi:10.18653/v1/2021.acl-long.515 · Cited by 19
Rethinking Denoised Auto-Encoding in Language Pre-Training
Luo, F., Yang, P., Li, S., Ren, X., Sun, X., Huang, S., & Huang, F. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.232 · Cited by 8
Enriching Language Models with Visually-grounded Word Vectors and the Lancaster Sensorimotor Norms
Kennington, C. (2021) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2021.conll-1.11 · Cited by 10
EPT-X: An Expression-Pointer Transformer model that generates eXplanations for numbers
Kim, B., Ki, K. S., Rhim, S., & Gweon, G. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.305 · Cited by 9
Long-range Sequence Modeling with Predictable Sparse Attention
Zhuang, Y., Zhang, J., & Tu, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.19 · Cited by 10
Identifying Corporate Credit Risk Sentiments from Financial News
Ahbali, N., Liu, X., Nanda, A., Stark, J., Talukder, A., & Khandpur, R. P. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-industry.40 · Cited by 8
Vogler, N., Allen, J., Miller, M. W., & Berg-Kirkpatrick, T. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.15 · Cited by 8
Swati, S., Grobelnik, A. M., Mladenić, D., & Grobelnik, M. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110838 · Cited by 8
Multi-Level Knowledge Distillation for Out-of-Distribution Detection in Text
Wu, Q., Jiang, H., Yin, H., Karlsson, B. F., & Lin, C.-Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.403 · Cited by 11
Efficient Pre-training of Masked Language Model via Concept-based Curriculum Masking
Lee, M., Park, J.-H., Kim, J., Kim, K.-M., & Lee, S. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.502 · Cited by 12
Bipol: A novel multi-axes bias evaluation metric with explainability for NLP
Alkhaled, L., Adewumi, T., & Sabry, S. S. (2023) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100030 · Cited by 8
Comparative Analysis of NLP Models for Detecting Depression on Twitter
Gupta, K., Jinad, R., & Liu, Q. (2023) · International Conference on Communications, Computing and Artificial Intelligence (CCCAI) · doi:10.1109/cccai59026.2023.00013 · Cited by 9
Sentiment Analysis on Banking Feedback and News Data using Synonyms and Antonyms
Mohanty, A., & Cherukuri, R. C. (2023) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2023.0141294 · Cited by 8
Discriminative Self-Training for Punctuation Prediction
Chen, Q., Wang, W., Chen, M., & Zhang, Q. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-246 · Cited by 9
Emotion Classification of COVID-19 Chinese Microblogs Based on the Emotion Category Description
Guo, X., Lai, H., Xiang, Y., Yu, Z., & Huang, Y. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-84186-7_5 · Cited by 8
Extract then Distill: Efficient and Effective Task-Agnostic BERT Distillation
Chen, C., Yin, Y., Shang, L., Wang, Z., Jiang, X., Chen, X., & Liu, Q. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86365-4_46 · Cited by 9
Variance-reduced Language Pretraining via a Mask Proposal NetworkPreprint
Chen, L. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.05333 · Cited by 8
A Brief Overview of Unsupervised Neural Speech Representation LearningPreprint
Borgholt, L., Havtorn, J. D., Edin, J., Maaløe, L., & Igel, C. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.01829 · Cited by 15
Bert-Based Feature Extraction for Long-Lived Bug Prediction in Floss: A Comparative StudyPreprint
Gomes, L. A. F., Côrtes, M. L., & Torres, R. D. S. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4166555 · Cited by 8
Training-free Neural Architecture Search for RNNs and Transformers
Serianni, A., & Kalita, J. K. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.142 · Cited by 10
Iterative framework based on multi-task learning for service recommendation
Yu, T., Yu, D., Wang, D., Yang, Q., & Hu, X. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111873 · Cited by 7
Extending limited datasets with GAN-like self-supervision for SMS spam detection
Anidjar, O. H., Marbel, R., Dubin, R., Dvir, A. Z., & Hajaj, C. (2024) · Computers & Security · doi:10.1016/j.cose.2024.103998 · Cited by 10
Allaway, E., Bhagavatula, C., Hwang, J. D., McKeown, K. R., & Leslie, S. (2024) · Computational Linguistics · doi:10.1162/coli_a_00530 · Cited by 9
Soliman, G., Zaki, H., & Kilany, M. (2025) · Natural Language Processing Journal · doi:10.1016/j.nlp.2025.100131 · Cited by 13
Using cross‐encoders to measure the similarity of short texts in political science
Lin, G. (2025) · American Journal of Political Science · doi:10.1111/ajps.12956 · Cited by 9
DBNLP: detecting bias in natural language processing system for India-centric languages
G, K. K. T., Mendke, S., Parihar, R., Mayya, S., Venkatesh, S., & Koolagudi, S. G. (2025) · International Journal of Information Technology · doi:10.1007/s41870-025-02437-9 · Cited by 7
Yu, Z., Fang, L., Ding, Y., Yan, S., Xu, L., Cai, Y., & Liu, X. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104844 · Cited by 8
Smith, A., Beaney, T. E., Hockham, C., Su, B., Elliott, P., Downey, L. E., Denaxas, S., Barnaghi, P. M., Dehghan, A., & Tzoulaki, I. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01806-9 · Cited by 8
Cho, M., Park, M. S., Park, J., Park, S.-K., Cha, G., & Park, S. (2025) · Automation in Construction · doi:10.1016/j.autcon.2025.106588 · Cited by 13
Temporal positional lexicon expansion for federated learning based on hyperpatism detection
Ahmed, U., Lin, J. C., & Srivastava, G. (2022) · Expert Systems · doi:10.1111/exsy.13183 · Cited by 6
An Efficient Deep Bidirectional Transformer Model for Energy Disaggregation
Sykiotis, S., Kaselimi, M., Doulamis, A. D., & Doulamis, N. D. (2022) · European Signal Processing Conference (EUSIPCO) · doi:10.23919/eusipco55093.2022.9909768 · Cited by 8
AWS-EP: A Multi-Task Prediction Approach for MBTI/Big5 Personality Tests
Elourajini, F., & Aı̈meur, E. (2022) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw58026.2022.00049 · Cited by 7
Learning Cross-Lingual Visual Speech Representations
Zinonos, A., Haliassos, A., Ma, P., Petridis, S., & Pantić, M. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10094585 · Cited by 10
Zekaoui, N. E., Yousfi, S., Rhanoui, M., & Mikram, M. (2023) · IAES International Journal of Artificial Intelligence · doi:10.11591/ijai.v12.i4.pp1995-2010 · Cited by 10
PLUE: Language Understanding Evaluation Benchmark for Privacy Policies in English
Chi, J., Ahmad, W. U., Tian, Y., & Chang, K.-W. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.31 · Cited by 11
Few-Shot Anaphora Resolution in Scientific Protocols via Mixtures of In-Context Experts
Le, T. N., Bai, F., & Ritter, A. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.197 · Cited by 13
Matsubara, Y., Soldaini, L., Lind, E., & Moschitti, A. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.537 · Cited by 9
IMBERT: Making BERT Immune to Insertion-based Backdoor Attacks
He, X., Wang, J., Rubinstein, B. I. P., & Cohn, T. (2023) · Workshop on Trustworthy Natural Language Processing (TrustNLP) · doi:10.18653/v1/2023.trustnlp-1.25 · Cited by 9
Evolutionary learning of selection hyper-heuristics for text classification
Ramírez, J. D. J. E., & Gómez, J. C. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110721 · Cited by 8
Haviana, S. F. C., Mulyono, S., & Badieah, B. (2023) · Proceeding of the Electrical Engineering Computer Science and Informatics · doi:10.1109/eecsi59885.2023.10295797 · Cited by 9
Boudad, N., Faizi, R., & Thami, R. O. H. (2023) · Social Network Analysis and Mining · doi:10.1007/s13278-023-01159-9 · Cited by 9
Wang, X., Gupta, D., Killian, M., & He, Z. (2023) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi57859.2023.00102 · Cited by 6
Beyond Words : Harnessing GPT-2 to Continue Stories with Imagination
Bandyopadhyay, A., Chakraborty, P., Debdas, S., Patra, M., Mohapatra, S., & Guha, D. (2023) · IEEE Silchar Subsection Conference (SILCON) · doi:10.1109/silcon59133.2023.10404413 · Cited by 9
A self-supervised learning model for graph clustering optimization problems
Cai, Q., Guo, X., & Huang, S. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111549 · Cited by 7
SC-Net: Multimodal metaphor detection using semantic conflicts
He, X., Yu, L., Tian, S.-W., Yang, Q.-M., & Long, J. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127825 · Cited by 7
Raj, K., Roy, K., Bonagiri, V. K., Govil, P., Thirunarayan, K., Goswami, R., & Gaur, M. (2024) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v3i1.31203 · Cited by 9
Explanatory argument extraction of correct answers in resident medical exams
Goenaga, I., Atutxa, A., Gojenola, K., Oronoz, M., & Agerri, R. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102985 · Cited by 13
Hussain, M., Chen, C., Albouq, S. S., Shinan, K., Alanazi, F., Iqbal, M. W., & Ashraf, M. U. (2025) · PLoS ONE · doi:10.1371/journal.pone.0312867 · Cited by 9
How did we get there? AI applications to biological networks and sequences
Anteghini, M., Gualdi, F., & Oliva, B. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110064 · Cited by 9
Training ELECTRA Augmented with Multi-word Selection
Shen, J., Liu, J., Liu, T., Yu, C., & Han, J. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.219 · Cited by 9
Lightweight Cross-Lingual Sentence Representation Learning
Mao, Z., Gupta, P., Chu, C., Jaggi, M., & Kurohashi, S. (2021) · doi:10.18653/v1/2021.acl-long.226 · Cited by 7
Zeng, Y., & Nie, J. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.393 · Cited by 7
MVP-BERT: Multi-Vocab Pre-training for Chinese BERT
Zhu, W. (2021) · doi:10.18653/v1/2021.acl-srw.27 · Cited by 7
Research on Chinese Event Extraction Method Based on RoBERTa-WWM-CRF
Li, Z., Cheng, N., & Song, W. (2021) · IEEE International Conference on Software Engineering and Service Science (ICSESS) · doi:10.1109/icsess52187.2021.9522150 · Cited by 9
Diacritics generation and application in hate speech detection on Vietnamese social networks
Le-Hong, P. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107504 · Cited by 7
Combating Word-level Adversarial Text with Robust Adversarial Training
Du, X., Yu, J., Li, S., Yi, Z., Liu, H., & Ma, J. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533725 · Cited by 7
Schmidt, T., Dennerlein, K., & Wolff, C. G. (2021) · Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL) · doi:10.18653/v1/2021.latechclfl-1.8 · Cited by 7
Alrowili, S., & Shanker, V. K. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.108 · Cited by 7
Same Sentence Prediction: A new Pre-training Task for BERT
Sonmezoz, K., & Amasyalı, M. F. (2021) · Innovations in Intelligent Systems and Applications Conference (ASYU) · doi:10.1109/asyu52992.2021.9598954 · Cited by 7
Evaluation of Arabic-Based Contextualized Word Embedding Models
Yagi, S., Mansour, Y., Kamalov, F., & Elnagar, A. (2021) · International Conference on Asian Language Processing · doi:10.1109/ialp54817.2021.9675208 · Cited by 7
Constructing Open Cloze Tests Using Generation and Discrimination Capabilities of Transformers
Felice, M., Taslimipoor, S., & Buttery, P. J. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.100 · Cited by 7
Findings of the Shared Task on Offensive Span Identification fromCode-Mixed Tamil-English Comments
Ravikiran, M., Chakravarthi, B. R., Kumar, M. A., Sangeetha, S., Rajalakshmi, R., Thavareesan, S., Ponnusamy, R., & Mahadevan, S. (2022) · Workshop on Speech, Vision, and Language Technologies for Dravidian Languages (DravidianLangTech) · doi:10.18653/v1/2022.dravidianlangtech-1.40 · Cited by 47
TeluguNER: Leveraging Multi-Domain Named Entity Recognition with Deep Transformers
Duggenpudi, S. R., Oota, S. R., Marreddy, M., & Mamidi, R. (2022) · Annual Meeting of the Association for Computational Linguistics: Student Research Workshop (ACL SRW) · doi:10.18653/v1/2022.acl-srw.20 · Cited by 7
Lee, H., Yoo, K. M., Park, J., Lee, H., & Jung, K. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.76 · Cited by 9
Feng, W. Q., & Nguyen, U. T. (2022) · IEEE Open Journal of the Computer Society · doi:10.1109/ojcs.2022.3213791 · Cited by 12
Text-based neural networks for question intent recognition
Trewhela, A., & Figueroa, A. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.105933 · Cited by 7
Qurashi, A. W., Farhat, Z. A., Holmes, V., & Johnson, A. P. (2023) · IEEE Access · doi:10.1109/access.2023.3272610 · Cited by 9
Fake News Encoder Classifier (FNEC) for Online Published News Related to COVID-19 Vaccines
Qaiser, A., Hina, S., Kazi, A. K., Ahmed, S., & Asif, R. (2023) · Intelligent Automation & Soft Computing · doi:10.32604/iasc.2023.036784 · Cited by 7
Khoo, L. S., Bay, J. Q., Yap, M. L. K., Lim, M. K., Chong, C. Y., Yang, Z., & Lo, D. (2023) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner56733.2023.00066 · Cited by 7
VarMAE: Pre-training of Variational Masked Autoencoder for Domain-adaptive Language Understanding
Hu, D., Hou, X., Du, X., Zhou, M., Jiang, L., Mo, Y., & Shi, X. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.468 · Cited by 12
Metadata
- Authors
- Clark, K., Luong, M.-T., Le, Q. V., & Manning, C. D.
- Affiliations
- Stanford University (US)
- Google (United States) (US)
- ORCID
- Le, Quoc V.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2003.10555
- OpenAlex
- W2996035354
- Cited by
- 1,844
- References
- 48
- Citations by year
- 2026: 13 · 2025: 129 · 2024: 240 · 2023: 572 · 2022: 415 · 2021: 375 · 2020: 100
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Learning Representations
- Funders
- Canadian Institute for Advanced Research
- Programa Operacional Regional do Centro (Portugal 2020)
- Fundação para a Ciência e a Tecnologia (PT2020, -01-0247-, 00326, UID/CEC/00326/2020)
- European Regional Development Fund (PT2020)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- Electra, masked language modeling, text encoder, sample efficiency, context representation, natural language understanding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
