The DOI matches its DataCite record; 48 of the 48 references it cites were found in a registry or an index.
ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
Open accessAbstract
Masked language modeling (MLM) pre-training methods such as BERT corrupt the input by replacing some tokens with [MASK] and then train a model to reconstruct the original tokens. While they produce good results when transferred to downstream NLP tasks, they generally require large amounts of compute to be effective. As an alternative, we propose a more sample-efficient pre-training task called replaced token detection. Instead of masking the input, our approach corrupts it by replacing some tokens with plausible alternatives sampled from a small generator network. Then, instead of training a model that predicts the original identities of the corrupted tokens, we train a discriminative model that predicts whether each token in the corrupted input was replaced by a generator sample or not. Thorough experiments demonstrate this new pre-training task is more efficient than MLM because the task is defined over all input tokens rather than just the small subset that was masked out. As a result, the contextual representations learned by our approach substantially outperform the ones learned by BERT given the same model size, data, and compute. The gains are particularly strong for small models; for example, we train a model on one GPU for 4 days that outperforms GPT (trained using 30x more compute) on the GLUE natural language understanding benchmark. Our approach also works well at scale, where it performs comparably to RoBERTa and XLNet while using less than 1/4 of their compute and outperforms them when using the same amount of compute.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We will use FLOPs, as they allow us to estimate the needed energy relating hardware FLOPS with required FLOPs for a model [Hollemans, 2018, Clark et al., 2020].”
From Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning (2023) · cites this paper as Clark et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In NLP, they count FLOPs as single mathematical operations and not as a single hardware instructions (like in CV).”
From Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning (2023) · cites this paper as Clark et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The authors corrupted the tokens with plausible alternatives sampled from a small generator network that improves the model performance significantly.”
From A Review on Deep-Learning-Based Cyberbullying Detection (2023) · cites this paper as [52]
The cited source says what the claim attributes to it.
In the citing paper
“but we ultimately found out this information explained in [Clark et al., 2020].”
From Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning (2023) · cites this paper as Clark et al., 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Self-citationEfficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
GAN（Generative Adversarial Nets）
淳司, 柴. (2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 20,507
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Unsupervised representation learning with deep convolutional generative adversarial networks
Radford, A., Metz, L., & Chintala, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06434 · Cited by 12,235
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 8,608
Flagged first, then by citations
Research works printed but not linked: 4
- Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS , 2014.
- Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942 , 2019. · arXiv
- Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. English gigaword, fifth edition. Technical report, Linguistic Data Consortium, Philadelphia, 2011.
- Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. MobileBERT: Task-agnostic compression of bert for resource limited devices, 2019b. URL https: //openreview.net/forum?id=SJxjVaNKwB .
Cited by
Sorted by citations
Items 201–300 of 1,010
MasakhaNER 2.0: Africa-centric Transfer Learning for Named Entity Recognition
Adelani, D. I., Neubig, G., Ruder, S., Rijhwani, S., Beukman, M., Palen-Michel, C., Lignos, C., Alabi, J. O., Muhammad, S. H., Nabende, P., et al. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.298 · Cited by 78
Filling the Gap of Utterance-aware and Speaker-aware Representation for Multi-turn Dialogue
Liu, L., Zhang, Z., Zhao, H., Zhou, X., & Zhou, X. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17582 · Cited by 51
Less is More: Pretrain a Strong Siamese Encoder for Dense Text Retrieval Using a Weak Decoder
Lu, S., He, D., Xiong, C., Ke, G., Malik, W., Dou, Z., Bennett, P., Liu, T.-Y., & Overwijk, A. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.220 · Cited by 49
MuLan-Methyl—multiple transformer-based language models for accurate DNA methylation prediction
Zeng, W., Gautam, A., & Huson, D. H. (2022) · GigaScience · doi:10.1093/gigascience/giad054 · Cited by 49
BERT & Family Eat Word Salad: Experiments with Text Understanding
Gupta, A., Kvernadze, G., & Srikumar, V. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17531 · Cited by 57
TaCL: Improving BERT Pre-training with Token-aware Contrastive Learning
Su, Y., Liu, F., Meng, Z., Lan, T., Shu, L., Shareghi, E., & Collier, N. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.191 · Cited by 59
Intelligent recognition of spacecraft components from photorealistic images based on Unreal Engine 4
Zhao, Y., Zhong, R., & Cui, L. (2022) · Advances in Space Research · doi:10.1016/j.asr.2022.09.045 · Cited by 48
DeCEMBERT: Learning from Noisy Instructional Videos via Dense Captions and Entropy Minimization
Tang, Z., Lei, J., & Bansal, M. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.193 · Cited by 48
A study of BERT for context-aware neural machine translation
Wu, X., Xia, Y., Zhu, J., Wu, L., Xie, S., & Qin, T. (2022) · Machine Learning · doi:10.1007/s10994-021-06070-y · Cited by 53
Which *BERT? A Survey Organizing Contextualized Encoders
Xia, P., Wu, S., & Durme, B. V. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.608 · Cited by 57
WebSRC: A Dataset for Web-Based Structural Reading Comprehension
Chen, X., Zhao, Z., Chen, L., Ji, J., Zhang, D., Luo, A., Xiong, Y., & Yu, K. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.343 · Cited by 97
Röttger, P., & Pierrehumbert, J. B. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.206 · Cited by 65
Nguyen, V.-A., Nguyen, D. Q., Nguyen, V., Le, T., Tran, Q. H., & Phung, D. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse-companion55297.2022.9793807 · Cited by 192
NoisyTune: A Little Noise Can Help You Finetune Pretrained Language Models Better
Wu, C., Wu, F., Qi, T., & Huang, Y. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-short.76 · Cited by 59
A Framework and Operational Procedures for Metaverses-Based Industrial Foundation Models
Wang, J., Tian, Y., Wang, Y., Yang, J., Wang, X., Wang, S., & Kwan, O. (2022) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2022.3226755 · Cited by 48
Masked Spectrogram Prediction for Self-Supervised Audio Pre-Training
Chong, D., Wang, H., Zhou, P., & Zeng, Q. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095691 · Cited by 67
Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection
Gururangan, S., Card, D., Dreier, S. K., Gade, E. K., Wang, L. Z., Wang, Z., Zettlemoyer, L., & Smith, N. A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.165 · Cited by 87
SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapters
He, S., Ding, L., Dong, D., Zhang, J., & Tao, D. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.160 · Cited by 95
Learning towards conversational AI: A survey
Fu, T., Gao, S., Zhao, X., Wen, J.-R., & Yan, R. (2022) · AI Open · doi:10.1016/j.aiopen.2022.02.001 · Cited by 46
A Transformers Approach to Detect Depression in Social Media
Malviya, K., Roy, B., & Saritha, S. K. (2021) · International Conference on Artificial Intelligence and Smart Energy (ICAIS) · doi:10.1109/icais50930.2021.9395943 · Cited by 47
A Survey on Arabic Named Entity Recognition: Past, Recent Advances, and Future Trends
Qu, X., Gu, Y., Xia, Q., Li, Z., Wang, Z., & Huai, B. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3303136 · Cited by 58
Aggregating Customer Review Attributes for Online Reputation Generation
Benlahbib, A., & Nfaoui, E. H. (2020) · IEEE Access · doi:10.1109/access.2020.2996805 · Cited by 56
Interpretation of NLP models through input marginalization
Kim, S., Yi, J., Kim, E., & Yoon, S. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.255 · Cited by 56
Machine Reading Comprehension: The Role of Contextualized Language Models and BeyondPreprint
Zhang, Z., Zhao, H., & Wang, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.06249 · Cited by 63
A Collaborative AI-Enabled Pretrained Language Model for AIoT Domain Question Answering
Zhu, H., Tiwari, P., Ghoneim, A., & Hossain, M. S. (2021) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2021.3097183 · Cited by 46
Named Entity Recognition in Cyber Threat Intelligence Using Transformer-based Models
Evangelatos, P., Iliou, C., Mavropoulos, T., Apostolou, K., Tsikrika, T., Vrochidis, S., & Kompatsiaris, I. Y. (2021) · IEEE International Conference on Cyber Security and Resilience (CSR) · doi:10.1109/csr51186.2021.9527981 · Cited by 51
A transformer-based generative adversarial network for brain tumor segmentation
Huang, L., Zhu, E., Chen, L. F., Wang, Z., Chai, S., & Zhang, B. (2022) · Frontiers in Neuroscience · doi:10.3389/fnins.2022.1054948 · Cited by 48
Multimodal Fake News Analysis Based on Image–Text Similarity
Zhang, X., Dadkhah, S., Weismann, A. G., Kanaani, M. A., & Ghorbani, A. A. (2023) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2023.3244068 · Cited by 45
A Comprehensive Survey on Arabic Sarcasm Detection: Approaches, Challenges and Future Trends
Rahma, A., Azab, S. S., & Mohammed, A. (2023) · IEEE Access · doi:10.1109/access.2023.3247427 · Cited by 55
Exploring the Effectiveness of Prompt Engineering for Legal Reasoning Tasks
Yu, F., Quartey, L., & Schilder, F. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.858 · Cited by 58
Ahmed, N., Saha, A. K., Noman, M. A. A., Jim, J. R., Mridha, M. F., & Kabir, M. M. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2024.100112 · Cited by 65
Distilling the Knowledge of BERT for Sequence-to-Sequence ASR
Futami, H., Inaguma, H., Ueno, S., Mimura, M., Sakai, S., & Kawahara, T. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1179 · Cited by 52
LIMIT-BERT : Linguistics Informed Multi-Task BERT
Zhou, J., Zhang, Z., Zhao, H., & Zhang, S. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.399 · Cited by 46
The Evolution of Language Models Applied to Emotion Analysis of Arabic Tweets
Al-Twairesh, N. (2021) · Information · doi:10.3390/info12020084 · Cited by 46
SLM: Learning a Discourse Language Representation with Sentence Unshuffling
Lee, H., Hudson, D. A., Lee, K., & Manning, C. D. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.120 · Cited by 44
Self-citationHindi fake news detection using transformer ensembles
Praseed, A., Rodrigues, J., & Thilagam, P. S. (2022) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2022.105731 · Cited by 46
SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval
Wang, L., Yang, N., Huang, X., Jiao, B., Yang, L., Jiang, D., Majumder, R., & Wei, F. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.125 · Cited by 111
Robust Transfer Learning with Pretrained Language Models through Adapters
Han, W., Pang, B., & Wu, Y. N. (2021) · doi:10.18653/v1/2021.acl-short.108 · Cited by 47
Diet code is healthy: simplifying programs for pre-trained models of code
Zhang, Z., Zhang, H., Shen, B., & Gu, X. (2022) · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3540250.3549094 · Cited by 41
Lian, Z., Sun, H., Sun, L., Wen, Z., Zhang, S., Chen, S., Gu, H., Zhao, J., Ma, Z., Chen, X., et al. (2024) · International Workshop on Multimodal and Responsible Affective Computing (MRAC) · doi:10.1145/3689092.3689959 · Cited by 41
EAPT: An encrypted traffic classification model via adversarial pre-trained transformers
Zhan, M., Yang, J., Jia, D., & Fu, G. (2024) · Computer Networks · doi:10.1016/j.comnet.2024.110973 · Cited by 44
Comprehensive analysis of embeddings and pre-training in NLP
Tripathy, J. K., Sethuraman, S. C., Cruz, M. V., Namburu, A., Mangalraj, P., Nandha Kumar, R., Sudhakar Ilango, S., & Vijayakumar, V. (2021) · Computer Science Review · doi:10.1016/j.cosrev.2021.100433 · Cited by 46
Huang, Y.-J., Lin, Y.-T., Liu, C.-C., Lee, L.-E., Hung, S.-H., Lo, J.-K., & Fu, L.-C. (2022) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2022.3163777 · Cited by 46
Prompt Tuning for Discriminative Pre-trained Language Models
Yao, Y., Dong, B., Zhang, A., Zhang, Z., Xie, R., Liu, Z., Lin, L., Sun, M., & Wang, J. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.273 · Cited by 44
Spatiotemporal self-supervised representation learning from multi-lead ECG signals
Hu, R. M., Chen, J., & Zhou, L. P. (2023) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2023.104772 · Cited by 41
Improving antibody language models with native pairing
Burbach, S. M., & Briney, B. S. (2024) · Patterns · doi:10.1016/j.patter.2024.100967 · Cited by 40
Chowdhury, A. K., Sujon, S. R., Shafi, M. S. S., Ahmmad, T., Ahmed, S., Hasib, K. M., & Shah, F. M. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2024.100075 · Cited by 47
Answering Open-Domain Questions of Varying Reasoning Steps from Text
Qi, P., Lee, H., Sido, O. T., & Manning, C. D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.292 · Cited by 65
Self-citationMulti-Granularity Structural Knowledge Distillation for Language Model Compression
Liu, C., Tao, C., Feng, J., & Zhao, D. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.71 · Cited by 43
SemEval-2021 Task 2: Multilingual and Cross-lingual Word-in-Context Disambiguation (MCL-WiC)
Martelli, F., Kalach, N., Tola, G., & Roberto Navigli, A. (2021) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2021.semeval-1.3 · Cited by 40
Generate, Annotate, and Learn: NLP with Synthetic Text
He, X., Nassar, I., Kiros, J., Haffari, G., & Norouzi, M. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00492 · Cited by 55
HOTTEST: Hate and Offensive content identification in Tamil using Transformers and Enhanced STemming
Rajalakshmi, R., Selvaraj, S., Faerie Mattins, R., Vasudevan, P., & Kumar, M. A. (2022) · Computer Speech & Language · doi:10.1016/j.csl.2022.101464 · Cited by 48
Identifying Machine-Paraphrased Plagiarism
Wahle, J. P., Ruas, T., Foltýnek, T., Meuschke, N., & Gipp, B. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-96957-8_34 · Cited by 65
Deep Learning Meets Software Engineering: A Survey on Pre-Trained Models of Source Code
Niu, C., Li, C., Luo, B., & Ng, V. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/775 · Cited by 63
Revisiting Self-training for Few-shot Learning of Language Model
Chen, Y., Zhang, Y., Zhang, C., Lee, G., Cheng, R., & Li, H. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.718 · Cited by 41
FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations
Ribeiro, L., Liu, M., Gurevych, I., Dreyer, M., & Bansal, M. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.236 · Cited by 54
SynGEC: Syntax-Enhanced Grammatical Error Correction with a Tailored GEC-Oriented Parser
Zhang, Y., Zhang, B., Li, Z., Bao, Z., Li, C., & Zhang, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.162 · Cited by 46
Multitude: Large-scale multilingual machine-generated text detection benchmark
Macko, D., Moro, R., Uchendu, A., Lucas, J., Yamashita, M., Pikuliak, M., Srba, I., Le, T., Lee, D., Simko, J., et al. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.616 · Cited by 38
Hashmi, E., Yayilgan, S. Y., Hameed, I. A., Yamin, M. M., Ullah, M., & Abomhara, M. (2024) · IEEE Access · doi:10.1109/access.2024.3452987 · Cited by 45
Nentidis, A., Katsimpras, G., Vandorou, E., Krithara, A., Gascó, L., Krallinger, M., & Παλιούρας, Γ. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-85251-1_18 · Cited by 41
Wang, S., Liu, J., Ouyang, X., & Sun, Y. (2020) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2020.semeval-1.189 · Cited by 41
GLGE: A New General Language Generation Evaluation Benchmark
Liu, D., Yan, Y., Gong, Y., Qi, W., Zhang, H., Jiao, J., Chen, W., Fu, J., Shou, L., Gong, M., et al. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.36 · Cited by 42
Metaphors in Pre-Trained Language Models: Probing and Generalization Across Datasets and Languages
Aghazadeh, E., Fayyaz, M., & Yaghoobzadeh, Y. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.144 · Cited by 54
Abualhaija, S., Arora, C., Sleimi, A., & Briand, L. (2022) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re54965.2022.00011 · Cited by 46
StableMoE: Stable Routing Strategy for Mixture of Experts
Dai, D., Dong, L., Ma, S., Zheng, B., Sui, Z., Chang, B., & Wei, F. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.489 · Cited by 108
Clinical insights: A comprehensive review of language models in medicine
Neveditsin, N., Lingras, P., & Mago, V. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000800 · Cited by 37
KALA: Knowledge-Augmented Language Model Adaptation
Kang, M., Baek, J., & Hwang, S. J. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.379 · Cited by 44
Defending against Backdoor Attacks in Natural Language Generation
Sun, X., Li, X., Meng, Y., Ao, X., Lyu, L., Li, J., & Zhang, T. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i4.25656 · Cited by 48
Yang, L., Zhang, S., Qin, L., Li, Y., Wang, Y., Liu, H., Wang, J., Xie, X., & Zhang, Y. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.806 · Cited by 81
Lamsal, R., Rodriguez, M. A., & Karunasekera, S. A. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111916 · Cited by 47
Chada, R., & Natarajan, P. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.491 · Cited by 49
Gani, M. O., Ayyasamy, R. K., Sangodiah, A., & Fui, Y. T. (2023) · Education and Information Technologies · doi:10.1007/s10639-023-11842-1 · Cited by 41
Discourse Probing of Pretrained Language Models
Koto, F., Lau, J. H., & Baldwin, T. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.301 · Cited by 42
Your fairness may vary: Pretrained language model fairness in toxic text classification
Baldini, I., Wei, D., Ramamurthy, K. N., Singh, M., & Yurochkin, M. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.176 · Cited by 51
Schema Matching using Pre-Trained Language Models
Zhang, Y., Floratou, A., Cahoon, J., Krishnan, S., Müller, A. C., Banda, D., Psallidas, F., & Patel, J. M. (2023) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde55515.2023.00123 · Cited by 50
UnitedQA: A Hybrid Approach for Open Domain Question Answering
Cheng, H., Shen, Y., Liu, X., He, P., Chen, W., & Gao, J. (2021) · doi:10.18653/v1/2021.acl-long.240 · Cited by 56
ConfliBERT: A Pre-trained Language Model for Political Conflict and Violence
Hu, Y., Hosseini, M., Parolin, E. S., Osorio, J., Khan, L. R., Brandt, P. T., & D'Orazio, V. J. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.400 · Cited by 44
Universal Multimodal Representation for Language Understanding
Zhang, Z., Chen, K., Wang, R., Utiyama, M., Sumita, E., Li, Z., & Zhao, H. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3234170 · Cited by 39
TreeBERT: A Tree-Based Pre-Trained Model for Programming LanguagePreprint
Jiang, X., Zheng, Z., Lyu, C., Li, L., & Lyu, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.12485 · Cited by 43
Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysis
Xiao, Y., Liang, P. P., Bhatt, U., Neiswanger, W., Salakhutdinov, R., & Morency, L. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.538 · Cited by 83
Contemporary Approaches in Evolving Language Models
Оралбекова, Д., Mamyrbayev, O. J., Othman, М., Kassymova, D., & Mukhsina, K. (2023) · Applied Sciences · doi:10.3390/app132312901 · Cited by 43
Cirillo, S., Desiato, D., Polese, G., Solimando, G., Sugumaran, V., & Sundaramurthy, S. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.104043 · Cited by 40
Pruning Redundant Mappings in Transformer Models via Spectral-Normalized Identity Prior
Lin, Z. J., Liu, J. Z., Yang, Z., Hua, N., & Roth, D. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.64 · Cited by 46
Xiao, D., Li, Y., Zhang, H., Sun, Y., Tian, H., Wu, H. R., & Wang, H. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.136 · Cited by 45
ECONET: Effective Continual Pretraining of Language Models for Event Temporal Reasoning
Han, R., Ren, X., & Peng, N. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.436 · Cited by 38
Efficient Domain Adaptation of Language Models via Adaptive Tokenization
Sachidananda, V., Kessler, J. S., & Lai, Y.-A. (2021) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2021.sustainlp-1.16 · Cited by 47
CLUECorpus2020: A Large-scale Chinese Corpus for Pre-training Language ModelPreprint
Xu, L. J., Zhang, X., & Dong, Q. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.01355 · Cited by 66
A Critical Evaluation of Evaluations for Long-form Question Answering
Xu, F., Song, Y., Iyyer, M., & Choi, E. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.181 · Cited by 33
Yu, B., Tang, F., Ergu, D., Zeng, R., Ma, B., & Liu, F. (2024) · IEEE Access · doi:10.1109/access.2024.3357095 · Cited by 39
Balasubramanian, P., Nazari, S., Kholgh, D. K., Mahmoodi, A., Seby, J., & Kostakos, P. (2025) · Decision Analytics Journal · doi:10.1016/j.dajour.2025.100545 · Cited by 46
Improving Low-Resource Named Entity Recognition using Joint Sentence and Token Labeling
Kruengkrai, C., Nguyen, T. H., Aljunied, S. M., & Bing, L. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.523 · Cited by 36
How Certain is Your Transformer?
Shelmanov, A., Tsymbalov, E., Puzyrev, D., Fedyanin, K., Panchenko, A., & Panov, M. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.157 · Cited by 34
A Case Study of the Shortcut Effects in Visual Commonsense Reasoning
Ye, K., & Kovashka, A. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i4.16428 · Cited by 34
Surgical GAN: Towards real-time path planning for passive flexible tools in endovascular surgeries
Zhao, Y. H., Wang, Y., Zhang, J., Liu, X., Li, Y., Guo, S., Xu, Y., & Hong, S. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.05.044 · Cited by 33
Semantic matching in machine reading comprehension: An empirical study
Liu, Q., Mao, R., Geng, X., & Cambria, E. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103145 · Cited by 34
Retraining a BERT Model for Transfer Learning in Requirements Engineering: A Preliminary Study
Ajagbe, M. A., & Zhao, L. (2022) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re54965.2022.00046 · Cited by 35
Roles and Utilization of Attention Heads in Transformer-based Neural Language Models
Jo, J.-Y., & Myaeng, S.-H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.311 · Cited by 33
Open Domain Question Answering with A Unified Knowledge Interface
Ma, K., Cheng, H., Liu, X., Nyberg, E., & Gao, J. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.113 · Cited by 41
Yu, L., Liu, B., Lin, Q., Zhao, X., & Che, C. (2024) · Journal of Advances in Information Technology · doi:10.12720/jait.15.3.446-450 · Cited by 84
Metadata
- Authors
- Clark, K., Luong, M.-T., Le, Q. V., & Manning, C. D.
- Affiliations
- Stanford University (US)
- Google (United States) (US)
- ORCID
- Le, Quoc V.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2003.10555
- OpenAlex
- W2996035354
- Cited by
- 1,844
- References
- 48
- Citations by year
- 2026: 13 · 2025: 129 · 2024: 240 · 2023: 572 · 2022: 415 · 2021: 375 · 2020: 100
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Learning Representations
- Funders
- Canadian Institute for Advanced Research
- Programa Operacional Regional do Centro (Portugal 2020)
- Fundação para a Ciência e a Tecnologia (PT2020, -01-0247-, 00326, UID/CEC/00326/2020)
- European Regional Development Fund (PT2020)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- Electra, masked language modeling, text encoder, sample efficiency, context representation, natural language understanding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
