The DOI matches its DataCite record; 48 of the 48 references it cites were found in a registry or an index.
ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
Open accessAbstract
Masked language modeling (MLM) pre-training methods such as BERT corrupt the input by replacing some tokens with [MASK] and then train a model to reconstruct the original tokens. While they produce good results when transferred to downstream NLP tasks, they generally require large amounts of compute to be effective. As an alternative, we propose a more sample-efficient pre-training task called replaced token detection. Instead of masking the input, our approach corrupts it by replacing some tokens with plausible alternatives sampled from a small generator network. Then, instead of training a model that predicts the original identities of the corrupted tokens, we train a discriminative model that predicts whether each token in the corrupted input was replaced by a generator sample or not. Thorough experiments demonstrate this new pre-training task is more efficient than MLM because the task is defined over all input tokens rather than just the small subset that was masked out. As a result, the contextual representations learned by our approach substantially outperform the ones learned by BERT given the same model size, data, and compute. The gains are particularly strong for small models; for example, we train a model on one GPU for 4 days that outperforms GPT (trained using 30x more compute) on the GLUE natural language understanding benchmark. Our approach also works well at scale, where it performs comparably to RoBERTa and XLNet while using less than 1/4 of their compute and outperforms them when using the same amount of compute.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We will use FLOPs, as they allow us to estimate the needed energy relating hardware FLOPS with required FLOPs for a model [Hollemans, 2018, Clark et al., 2020].”
From Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning (2023) · cites this paper as Clark et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In NLP, they count FLOPs as single mathematical operations and not as a single hardware instructions (like in CV).”
From Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning (2023) · cites this paper as Clark et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The authors corrupted the tokens with plausible alternatives sampled from a small generator network that improves the model performance significantly.”
From A Review on Deep-Learning-Based Cyberbullying Detection (2023) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“but we ultimately found out this information explained in [Clark et al., 2020].”
From Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning (2023) · cites this paper as Clark et al., 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Self-citationEfficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
GAN（Generative Adversarial Nets）
淳司, 柴. (2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 20,507
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Unsupervised representation learning with deep convolutional generative adversarial networks
Radford, A., Metz, L., & Chintala, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06434 · Cited by 12,235
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 8,608
Research works printed but not linked: 4
- Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS , 2014.
- Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942 , 2019. · arXiv
- Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. English gigaword, fifth edition. Technical report, Linguistic Data Consortium, Philadelphia, 2011.
- Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. MobileBERT: Task-agnostic compression of bert for resource limited devices, 2019b. URL https: //openreview.net/forum?id=SJxjVaNKwB .
Cited by
Sorted by citations
Items 901–1,000 of 1,010
Deutsch, D., & Roth, D. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.42 · Cited by 8
ShellGPT: Generative Pre-trained Transformer Model for Shell Language Understanding
Shi, J., Jiang, S., Xu, B., Liang, J., Xiao, Y., & Wang, W. (2023) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre59848.2023.00082 · Cited by 10
Lee, S.-S., Cha, S.-M., Ko, B., & Park, J. J. (2023) · IEEE Access · doi:10.1109/access.2023.3339774 · Cited by 7
Pre-training LLMs using human-like development data corpus
Bhardwaj, K., Shah, R. S., & Varma, S. (2023) · BabyLM Challenge · doi:10.18653/v1/2023.conll-babylm.30 · Cited by 7
Critical Analysis of BERT and LSTM Model for Bengali Sentiment Analysis Across Varied Datasets
Hossain, M. M., Akhi, I. A. H., Raisa, S. R. S., Onni, T. S., Rifat, A. I., & Islam, A. (2023) · International Conference on Computational Intelligence and Communication Networks (CICN) · doi:10.1109/cicn59264.2023.10402353 · Cited by 8
Dialogue for Prompting: A Policy-Gradient-Based Discrete Prompt Generation for Few-Shot Learning
Li, C., Liu, X., Wang, Y., Li, D., Lan, Y., & Shen, C. Q. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29809 · Cited by 14
Warped Language Models for Noise Robust Language Understanding
Namazifar, M., Tür, G., & Hakkani-Tur, D. Z. (2021) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt48900.2021.9383493 · Cited by 6
Zero-resource Multi-dialectal Arabic Natural Language Understanding
Khalifa, M., Hassan, H., & Fahmy, A. (2021) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2021.0120369 · Cited by 8
Low Anisotropy Sense Retrofitting (LASeR) : Towards Isotropic and Sense Enriched Representations
Bihani, G., & Rayz, J. T. (2021) · Workshop on Knowledge Extraction and Integration for Deep Learning Architectures: Deep Learning Inside Out (DeeLIO) · doi:10.18653/v1/2021.deelio-1.9 · Cited by 12
Clinical de-identification using sub-document analysis and ELECTRA
Catelli, R., Gargiulo, F., Damiano, E., Esposito, M., & Pietro, G. D. (2021) · IEEE International Conference on Digital Health (ICDH) · doi:10.1109/icdh52753.2021.00050 · Cited by 7
EmoPars: A Collection of 30K Emotion-Annotated Persian Social Media Texts
Sabri, N., Kharazi, R. A., & Bahrak, B. (2021) · Student Research Workshop .../Proceedings of the Student Research Workshop ... · doi:10.26615/issn.2603-2821.2021_023 · Cited by 7
A Pre-trained Language Model for Medical Question Answering Based on Domain Adaption
Liu, L., Ren, J., Wu, Y., Song, R., Cheng, Z., & Wang, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-17189-5_18 · Cited by 7
Model Selection with Model Zoo via Graph Learning
Li, Z., van der Wilk, H., Zhan, D., Khosla, M., Bozzon, A., & Hai, R. (2024) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde60146.2024.00088 · Cited by 9
Item Difficulty Modeling Using Fine-tuned Small and Large Language Models
Li, M., Jiao, H., Zhou, T., Zhang, N., Peters, S., & Lissitz, R. W. (2025) · Educational and Psychological Measurement · doi:10.1177/00131644251344973 · Cited by 10
Transformer spectral optimization: From gradient frequency analysis to adaptive spectral integration
Huang, Z., Chen, M., & Zheng, S. (2025) · Applied Soft Computing · doi:10.1016/j.asoc.2025.113637 · Cited by 6
Cooperative Self-training of Machine Reading Comprehension
Luo, H., Li, S.-W. D., Gao, M., Yu, S., & Glass, J. R. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.18 · Cited by 7
A Feedback-Enhanced Two-Stage Framework for judicial machine reading comprehension
Lin, Z., Yang, F., Wu, X., Su, J., & Wang, X. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106178 · Cited by 6
Ingle, D. A., Tripathi, R., Kumar, A., Patel, K., & Vepa, J. (2022) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2022.blackboxnlp-1.19 · Cited by 6
Unveiling Black-Boxes: Explainable Deep Learning Models for Patent Classification
Shajalal, M., Denef, S., Karim, M. R., Boden, A., & Stevens, G. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-44067-0_24 · Cited by 6
Veeramani, H., Thapa, S., & Naseem, U. (2023) · Arabic Natural Language Processing Conference (ArabicNLP) · doi:10.18653/v1/2023.arabicnlp-1.78 · Cited by 6
FedID: Federated Interactive Distillation for Large-Scale Pretraining Language Models
Ma, X., Liu, J., Wang, J., & Zhang, X. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.529 · Cited by 6
Exploring Transformers as Compact, Data-efficient Language Models
Fields, C., & Kennington, C. (2023) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-1.35 · Cited by 6
Bengali Image Captioning Using Vision Encoder-Decoder Model
Ishan, T. I., Noman, A. A., Rokib, R., Masum, M. I., Ahmed, S., & Shah, F. M. (2023) · International Conference on Computer and Information Technology (ICCIT) · doi:10.1109/iccit60459.2023.10441125 · Cited by 7
DPPMask: Masked Image Modeling with Determinantal Point Processes
Xu, J., Lin, Z., Zhou, D., Yang, Y., Liao, X., Wang, Q., Wu, B., Chen, G., & Heng, P. (2024) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv57701.2024.00226 · Cited by 7
CORRECT? CORECT!: Classification of ESG Ratings with Earnings Call Transcript
Lee, H., Jung, H. S., Park, H., & Kim, J. H. (2024) · KSII Transactions on Internet and Information Systems · doi:10.3837/tiis.2024.04.015 · Cited by 8
Aurpa, T. T., Ahmed, M. S., Rahman, M. M., & Moazzam, M. G. (2024) · PLoS ONE · doi:10.1371/journal.pone.0311161 · Cited by 6
Raj, S., Namdeo, V., Singh, P., & Srivastava, A. (2025) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2025.110320 · Cited by 8
Enhancing Text Classification Through Grammar-Based Feature Engineering and Learning Models
Mohasseb, A., Kanavos, A., & Amer, E. (2025) · Information · doi:10.3390/info16060424 · Cited by 8
Commonsense Knowledge Adversarial Dataset that Challenges ELECTRA
Lin, G., Miao, Y., Yang, X., Ou, W., Cui, L., Guo, W., & Miao, C. (2020) · International Conference on Control, Automation, Robotics and Vision (ICARCV) · doi:10.1109/icarcv50220.2020.9305451 · Cited by 6
An Alignment-Agnostic Model for Chinese Text Error Correction
Zheng, L., Deng, Y., Song, W., Xu, L. J., & Xiao, J. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.30 · Cited by 6
Kementchedjhieva, Y., Anderson, M., & Søgaard, A. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.429 · Cited by 10
Analogy Training Multilingual Encoders
Garneau, N., Hartmann, M., Sandholm, A., Ruder, S., Vulić, I., & Søgaard, A. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17524 · Cited by 7
An Efficient Transformer-Based Model for Vietnamese Punctuation Prediction
Tran, H., Dinh, C. V., Pham, Q., & Nguyen, B. T. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-79463-7_5 · Cited by 6
Towards Multi-class Sentiment Analysis With Limited Labeled Data
Riyadh, M., & Shafiq, M. O. (2021) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata52589.2021.9671692 · Cited by 6
Wen, Z., Powell, G. A., Chafi, I., Buckeridge, D. L., & Li, Y. (2022) · Patterns · doi:10.1016/j.patter.2022.100435 · Cited by 6
The Dark Side of the Language: Pre-trained Transformers in the DarkNet
Ranaldi, L., Nourbakhsh, A., Patrizi, A., Ruzzetti, E. S., Onorati, D., Mastromattei, M., Fallucchi, F., & Zanzotto, F. M. (2023) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-092-2_102 · Cited by 7
Product Answer Generation from Heterogeneous Sources: A New Benchmark and Best Practices
Shen, X., Barlacchi, G., Tredici, M. D., Cheng, W., Byrne, B., & de Gispert, A. (2022) · Workshop on e-Commerce and NLP (ECNLP) · doi:10.18653/v1/2022.ecnlp-1.13 · Cited by 6
An Empirical Study on the Fairness of Pre-trained Word Embeddings
Sesari, E., Hort, M., & Sarro, F. (2022) · Workshop on Gender Bias in Natural Language Processing (GeBNLP) · doi:10.18653/v1/2022.gebnlp-1.15 · Cited by 7
Capturing Conversational Interaction for Question Answering via Global History Reasoning
Qian, J., Zou, B., Dong, M., Li, X., Aw, A. T., & Yu, H. Z. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.159 · Cited by 6
Job Recommendation Based on Extracted Skill Embeddings
Kara, A., Daniş, F. S., Orman, G. K., Turhan, S., & Özlü, Ö. A. (2022) · Lecture notes in networks and systems · doi:10.1007/978-3-031-16075-2_35 · Cited by 6
Alfarizy, G., & Mandala, R. (2022) · International Conference on Advanced Informatics: Concepts, Theory and Applications (ICAICTA) · doi:10.1109/icaicta56449.2022.9932903 · Cited by 6
Deep Learning in Automatic Math Word Problem Solvers
Zhang, D. (2022) · Springer eBooks · doi:10.1007/978-3-031-09687-7_14 · Cited by 8
RNN Variants vs Transformer Variants: Uncertainty in Text Classification with Monte Carlo Dropout
Islam, M. F., Rahman, F. B., Zabeen, S., Islam, M. A., Hossain, M. S., Mehedi, M. H. K., Manab, M. A., & Rasel, A. A. (2022) · International Conference on Computer and Information Technology (ICCIT) · doi:10.1109/iccit57492.2022.10055922 · Cited by 7
GenTAL: Generative Denoising Skip-gram Transformer for Unsupervised Binary Code Similarity Detection
Li, L. T., Ding, S. H. H., & Charland, P. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191550 · Cited by 6
Multitask Pretraining with Structured Knowledge for Text-to-SQL Generation
Giaquinto, R. A., Zhang, D., Kleiner, B., Li, Y., Tan, M., Bhatia, P., Nallapati, R. M., & Ma, X. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.620 · Cited by 7
SocioProbe: What, When, and Where Language Models Learn about Sociodemographics
Lauscher, A., Bianchi, F., Bowman, S. R., & Hovy, D. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.539 · Cited by 7
Prompt Discriminative Language Models for Domain Adaptation
Lu, K., Potash, P., Lin, X., Sun, Y., Qian, Z., Zheng, Y. G., Naumann, T., Cai, T., & Lu, J. (2023) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2023.clinicalnlp-1.30 · Cited by 7
Hexatagging: Projective Dependency Parsing as Tagging
Amini, A., Liu, T., & Cotterell, R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.124 · Cited by 6
Language Agnostic Multilingual Information Retrieval with Contrastive Learning
Hu, X., Chen, X., Qi, P., Kong, D., Liu, K., Wang, W. Y., & Huang, Z. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.581 · Cited by 8
HyPe: Better Pre-trained Language Model Fine-tuning with Hidden Representation Perturbation
Yuan, H., Yuan, Z., Tan, C., Huang, F., & Huang, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.182 · Cited by 9
Dual Class Knowledge Propagation Network for Multi-label Few-shot Intent Detection
Zhang, F., Chen, W., Ding, F., & Wang, T. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.480 · Cited by 7
A Universal Discriminator for Zero-Shot Generalization
Xu, H., Lin, Z., Zhou, J., Zheng, Y., & Yang, Z. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.589 · Cited by 12
Empowering Dual-Encoder with Query Generator for Cross-Lingual Dense Retrieval
Ren, H., Shou, L., Wu, N., Gong, M., & Jiang, D. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.203 · Cited by 7
Alleviating Sparsity of Open Knowledge Graphs with Ternary Contrastive Learning
Li, Q., Joty, S., Wang, D., Shi, F., & Zhang, Y. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.168 · Cited by 6
Gessler, L., & Zeldes, A. (2022) · Workshop on Multilingual Representation Learning (MRL) · doi:10.18653/v1/2022.mrl-1.9 · Cited by 14
ALToolbox: A Set of Tools for Active Learning Annotation of Natural Language Texts
Tsvigun, A., Sanochkin, L., Larionov, D., Kuzmin, G., Vazhentsev, A., Lazichny, I., Khromov, N., Kireev, D., Rubashevskii, A., Shahmatova, O. V., et al. (2022) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2022.emnlp-demos.41 · Cited by 6
Arici, N., Gerevini, A., Olivato, M., Putelli, L., Sigalini, L., & Serina, I. (2023) · Future Internet · doi:10.3390/fi15080268 · Cited by 7
Masked Autoencoding Does Not Help Natural Language Supervision at Scale
Weers, F., Shankar, V., Katharopoulos, A., Yang, Y., & Gunter, T. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02244 · Cited by 8
GMU-WLV at TSAR-2022 Shared Task: Evaluating Lexical Simplification Models
North, K., Dmonte, A., Ranasinghe, T., & Zampieri, M. (2022) · Workshop on Text Simplification, Accessibility and Readability (TSAR) · doi:10.18653/v1/2022.tsar-1.30 · Cited by 7
Zeng, X., & Zubiaga, A. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.14 · Cited by 6
Oliveira, L. L., Jiang, X., Babu, A. N., Karajagi, P., & Daneshkhah, A. (2024) · Forecasting · doi:10.3390/forecast6010013 · Cited by 8
Odio-BERT: Evaluating domain task impact in hate speech detection
Yigezu, M. G., Kolesnikova, O., Gelbukh, A. F., & Sidorov, G. (2024) · Journal of Intelligent & Fuzzy Systems · doi:10.3233/jifs-219349 · Cited by 7
Hoque, M. N., & Seddiqui, M. H. (2024) · International Conference on Advances in Computing, Communication, Electrical, and Smart Systems (iCACCESS) · doi:10.1109/icaccess61735.2024.10499469 · Cited by 7
Chen, D., Soto, N., Tuttle, J. F., & Zong, Z. (2024) · IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum · doi:10.1109/ipdpsw63119.2024.00162 · Cited by 7
Integrating Task Specific Information into Pretrained Language Models for Low Resource Fine Tuning
Wang, R., Si, S., Wang, G., Zhang, L., Carin, L., & Henao, R. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.285 · Cited by 6
SCRIPT: Self-Critic PreTraining of Transformers
Nijkamp, E., Pang, B., Wu, Y. N., & Xiong, C. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.409 · Cited by 7
OoMMix: Out-of-manifold Regularization in Contextual Embedding Space for Text Classification
Lee, S., Lee, D., & Yu, H. (2021) · doi:10.18653/v1/2021.acl-long.49 · Cited by 8
Learning to Sample Replacements for ELECTRA Pre-Training
Hao, Y., Dong, L., Bao, H., Xu, K., & Wei, F. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.394 · Cited by 7
LICHEE: Improving Language Model Pre-training with Multi-grained Tokenization
Guo, W., Zhao, M. M., Zhang, L., Niu, D., Luo, J., Liu, Z., Li, Z., & Tang, J. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.119 · Cited by 6
Burtenshaw, B., & Kestemont, M. (2021) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2021.semeval-1.121 · Cited by 6
GHOST at SemEval-2021 Task 5: Is explanation all you need?
Pluciński, K., & Klimczak, H. (2021) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2021.semeval-1.114 · Cited by 6
Exploring the Capacity of a Large-scale Masked Language Model to Recognize Grammatical Errors
Nagata, R., Kimura, M., & Hanawa, K. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.324 · Cited by 8
Pretrained Neural Models for Turkish Text Classification
OKUR, H. İ., & Sertbaş, A. (2021) · International Conference on Computer Science and Engineering (UBMK) · doi:10.1109/ubmk52708.2021.9558878 · Cited by 6
Kim, H., & Kim, H. (2021) · Applied Sciences · doi:10.3390/app112210795 · Cited by 8
Analysis of natural language processing technology: modern problems and approaches
Kazakova, M. A., & Султанова, А. П. (2022) · Advanced Engineering Research · doi:10.23947/2687-1653-2022-22-2-169-176 · Cited by 8
Suri, M. (2022) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2022.semeval-1.63 · Cited by 6
Putri, D. U. K., & Pratomo, D. N. (2022) · Inform Jurnal Ilmiah Bidang Teknologi Informasi dan Komunikasi · doi:10.25139/inform.v7i2.4686 · Cited by 9
Topic-Aware Response Generation in Task-Oriented Dialogue with Unstructured Knowledge Access
Feng, Y., Λάμπουρας, Γ., & Iacobacci, I. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.533 · Cited by 7
Enriching Biomedical Knowledge for Low-resource Language Through Large-scale Translation
Phan, L., Dang, T., Tran, H., Trinh, T. H., Phan, V., Chau, L. D., & Luong, M.-T. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.228 · Cited by 7
Self-citationParagraph-based Transformer Pre-training for Multi-Sentence Inference
Liello, L. D., Garg, S., Soldaini, L., & Moschitti, A. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.181 · Cited by 7
Improving Tokenisation by Alternative Treatment of Spaces
Gow-Smith, E., Madabushi, H. T., Scarton, C., & Villavicencio, A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.786 · Cited by 9
LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text RetrievalPreprint
Sun, S., Chen, Y.-C., Li, L., Wang, S., Fang, Y., & Liu, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.08784 · Cited by 6
Hybrid Generative-Contrastive Representation LearningPreprint
Kim, S., Kim, S., & Lee, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.06162 · Cited by 11
Proceedings of The 4th Workshop on e-Commerce and NLP
(2021) · doi:10.18653/v1/2021.ecnlp-1 · Cited by 6
Taxi1500: A Multilingual Dataset for Text Classification in 1500 LanguagesPreprint
Ma, C., Imani, A., Ye, H., Pei, R., Asgari, E., & Schütze, H. (2023) · Research Square · doi:10.21203/rs.3.rs-3235946/v1 · Cited by 25
BanglaBait: Semi-Supervised Adversarial Approach for Clickbait Detection on Bangla Clickbait Dataset
Mahtab, M. M., Haque, M., Hasan, M., & Sadeque, F. (2023) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-092-2_081 · Cited by 7
Compliance Framework for Personal Data Protection Law Standards
Alkhamsi, N. N., & Alqahtani, S. S. (2024) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2024.0150751 · Cited by 11
Gao, J., Yu, H., Cheung, Y., Cao, J., Wong, R. C.-W., & Zhang, Y. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107754 · Cited by 6
Comparative Study of Static and Contextual Text Vectorization for Sentiment Analysis
Bhargavi, A. (2025) · International Journal for Research in Applied Science and Engineering Technology · doi:10.22214/ijraset.2025.73045 · Cited by 5
Sentiment-Enhanced Cyberbullying Detection Models on Social Media Platforms
Philipo, A. G., Ding, J., Sarwatt, D. S., Mohamed, J. A., Yusufu, A. S., Daneshmand, M., & Ning, H. (2025) · ACM Transactions on the Web · doi:10.1145/3766075 · Cited by 8
SKT-SQL: multi-decoupling and skeleton prompt framework for text-to-SQL on small-scale LMs
Yi, J., & Chen, G. (2025) · Journal of Intelligent Information Systems · doi:10.1007/s10844-025-00963-3 · Cited by 5
Improving Pretraining Techniques for Code-Switched NLP
Das, R., Ranjan, S., Pathak, S., & Jyothi, P. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.66 · Cited by 5
Multilingual BERT Cross-Lingual Transferability with Pre-trained Representations on Tangut: A Survey
Lu, X.-M., Liu, W., Jiang, S., & Liu, C. (2023) · doi:10.1109/icnlp58431.2023.00048 · Cited by 6
Exploring Data Augmentation for Code Generation Tasks
Chen, P., & Λάμπουρας, Γ. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.114 · Cited by 6
Artificial Intelligence for Impact Assessment of Administrative Burdens
Costa, V., Coelho, P. S., & Castelli, M. (2024) · Emerging Science Journal · doi:10.28991/esj-2024-08-01-019 · Cited by 7
Learning legal text representations via disentangling elements
Miao, Y., Zhou, F. Y., Pavlovski, M., & Qian, W. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.123749 · Cited by 5
Multi-Passage Machine Reading Comprehension Through Multi-Task Learning and Dual Verification
Li, X., Cheng, X., Xia, M., Ren, Q., He, Z., & Su, S. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3383103 · Cited by 5
Comparison of Transformer-Based Turkish Models for Question-Answering Task
Arzu, M., & Aydoğan, M. (2025) · Balkan Journal of Electrical and Computer Engineering · doi:10.17694/bajece.1576976 · Cited by 6
Construction Grammar and Language Models
Madabushi, H. T., Romain, L., Milin, P., & Divjak, D. (2025) · Cambridge University Press eBooks · doi:10.1017/9781009049139.023 · Cited by 16
Metadata
- Authors
- Clark, K., Luong, M.-T., Le, Q. V., & Manning, C. D.
- Affiliations
- Stanford University (US)
- Google (United States) (US)
- ORCID
- Le, Quoc V.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2003.10555
- OpenAlex
- W2996035354
- Cited by
- 1,844
- References
- 48
- Citations by year
- 2026: 13 · 2025: 129 · 2024: 240 · 2023: 572 · 2022: 415 · 2021: 375 · 2020: 100
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Learning Representations
- Funders
- Canadian Institute for Advanced Research
- Programa Operacional Regional do Centro (Portugal 2020)
- Fundação para a Ciência e a Tecnologia (PT2020, -01-0247-, 00326, UID/CEC/00326/2020)
- European Regional Development Fund (PT2020)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- Electra, masked language modeling, text encoder, sample efficiency, context representation, natural language understanding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
