ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
Open accessAbstract
Masked language modeling (MLM) pre-training methods such as BERT corrupt the input by replacing some tokens with [MASK] and then train a model to reconstruct the original tokens. While they produce good results when transferred to downstream NLP tasks, they generally require large amounts of compute to be effective. As an alternative, we propose a more sample-efficient pre-training task called replaced token detection. Instead of masking the input, our approach corrupts it by replacing some tokens with plausible alternatives sampled from a small generator network. Then, instead of training a model that predicts the original identities of the corrupted tokens, we train a discriminative model that predicts whether each token in the corrupted input was replaced by a generator sample or not. Thorough experiments demonstrate this new pre-training task is more efficient than MLM because the task is defined over all input tokens rather than just the small subset that was masked out. As a result, the contextual representations learned by our approach substantially outperform the ones learned by BERT given the same model size, data, and compute. The gains are particularly strong for small models; for example, we train a model on one GPU for 4 days that outperforms GPT (trained using 30x more compute) on the GLUE natural language understanding benchmark. Our approach also works well at scale, where it performs comparably to RoBERTa and XLNet while using less than 1/4 of their compute and outperforms them when using the same amount of compute.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
FLOPs allow estimating needed energy by relating hardware FLOPS with required FLOPs for a model.
“We will use FLOPs, as they allow us to estimate the needed energy relating hardware FLOPS with required FLOPs for a model [Hollemans, 2018, Clark et al., 2020].”
From Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning (2023) · cites this paper as Clark et al., 2020
Section 3.2 (p. 6) reports model FLOP requirements alongside training times and hardware, and Appendix E (p. 16) explains how the FLOPs are counted. These passages support using FLOPs to estimate computational requirements, but they do not describe estimating energy by relating those requirements to hardware FLOPS. (LLM, conf=0.83)
Experimental — accuracy not yet measuredELECTRA corrupts tokens with plausible alternatives sampled from a small generator network, which improves model performance significantly.
“The authors corrupted the tokens with plausible alternatives sampled from a small generator network that improves the model performance significantly.”
From A Review on Deep-Learning-Based Cyberbullying Detection (2023) · cites this paper as [52]
The information on how FLOPs are counted is explained in Clark et al., 2020.
“but we ultimately found out this information explained in [Clark et al., 2020].”
From Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning (2023) · cites this paper as Clark et al., 2020
In NLP, FLOPs are counted as single mathematical operations and not as single hardware instructions (like in CV).
“In NLP, they count FLOPs as single mathematical operations and not as a single hardware instructions (like in CV).”
From Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning (2023) · cites this paper as Clark et al., 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · see citing works
Self-citationEfficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · see citing works
GAN（Generative Adversarial Nets）
淳司, 柴. (2017) · Journal of Japan Society for Fuzzy Theory and Intelligent Informatics · doi:10.3156/jsoft.29.5_177_2 · Cited by 19,268 · see citing works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
Unsupervised representation learning with deep convolutional generative adversarial networks
Radford, A., Metz, L., & Chintala, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06434 · Cited by 12,235 · see citing works
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
Williams, R. J. (1992) · Machine Learning · doi:10.1007/bf00992696 · Cited by 7,936 · see citing works
Extracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436 · see citing works
Natural Language Processing (almost) from ScratchPreprint
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. P. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1103.0398 · Cited by 6,882 · see citing works
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878 · see citing works
Self-citationXLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687 · see citing works
Self-citationRepresentation Learning with Contrastive Predictive Coding
Oord, A. V. D., Li, Y., & Vinyals, O. (2018) · arXiv · doi:10.48550/arxiv.1807.03748 · Cited by 6,242 · see citing works
(2013) · Cited by 5,726 · see citing works
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · see citing works
Learning a Similarity Metric Discriminatively, with Application to Face Verification
Chopra, S., Hadsell, R., & LeCun, Y. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.202 · Cited by 4,036 · see citing works
Large Scale GAN Training for High Fidelity Natural Image Synthesis
Brock, A., Donahue, J., & Simonyan, K. (2018) · arXiv · doi:10.48550/arxiv.1809.11096 · Cited by 3,717 · see citing works
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342 · see citing works
SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient
Yu, L., Zhang, W., Wang, Y.-X., & Yu, Y. H. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10804 · Cited by 2,342 · see citing works
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., & Fidler, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.11 · Cited by 2,264 · see citing works
SpanBERT: Improving Pre-training by Representing and Predicting Spans
Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., & Levy, O. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00300 · Cited by 1,827 · see citing works
TinyBERT: Distilling BERT for natural language understanding
Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F., & Liu, Q. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.372 · Cited by 1,786 · see citing works
The PASCAL Recognising Textual Entailment Challenge
Dagan, I., Glickman, O., & Magnini, B. (2006) · Lecture notes in computer science · doi:10.1007/11736790_9 · Cited by 1,674 · see citing works
(2018) · International Conference on Learning Representations · Cited by 1,621 · see citing works
Noise-contrastive estimation: A new estimation principle for unnormalized statistical models
Gutmann, M. U., & Hyvärinen, A. (2010) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 1,588 · see citing works
SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation
Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., & Specia, L. (2017) · Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) · doi:10.18653/v1/s17-2001 · Cited by 1,562 · see citing works
Unified Language Model Pre-training for Natural Language Understanding and Generation
Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, Y., Gao, J., Zhou, M., & Hon, H.-W. (2019) · arXiv · doi:10.48550/arxiv.1905.03197 · Cited by 1,225 · see citing works
Automatically Constructing a Corpus of Sentential Paraphrases.
Dolan, W. B., & Brockett, C. (2005) · Cited by 1,101 · see citing works
Unsupervised Learning of Visual Representations Using Videos
Wang, X., & Gupta, A. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.320 · Cited by 1,005 · see citing works
Semi-supervised Sequence Learning
Dai, A. M., & Le, Q. V. (2015) · arXiv · doi:10.48550/arxiv.1511.01432 · Cited by 994 · see citing works
Self-citationNeural Network Acceptability Judgments
Warstadt, A., Singh, A., & Bowman, S. R. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00290 · Cited by 984 · see citing works
ERNIE: Enhanced Representation through Knowledge IntegrationPreprint
Sun, Y., Wang, S., Li, Y., Feng, S., Chen, X., Zhang, H., Tian, X., Zhu, D., Tian, H., & Wu, H. R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.09223 · Cited by 983 · see citing works
MASS: Masked Sequence to Sequence Pre-training for Language GenerationPreprint
Song, K., Tan, X., Qin, T., Lu, J., & Liu, T. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.02450 · Cited by 834 · see citing works
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · arXiv · doi:10.48550/arxiv.1606.05250 · Cited by 678 · see citing works
A Theoretical Analysis of Contrastive Unsupervised Representation LearningPreprint
Arora, S., Khandeparkar, H., Khodak, M., Plevrakis, O., & Saunshi, N. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.09229 · Cited by 426 · see citing works
Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data TasksPreprint
Phang, J., Févry, T., & Bowman, S. R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.01088 · Cited by 417 · see citing works
MaskGAN: Better Text Generation via Filling in the______Preprint
Fedus, W., Goodfellow, I., & Dai, A. M. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.07736 · Cited by 412 · see citing works
Smith, N. A., & Eisner, J. M. (2005) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.3115/1219840.1219884 · Cited by 305 · see citing works
Adversarial Feature Matching for Text GenerationPreprint
Zhang, Y., Gan, Z., Fan, K., Chen, Z., Henao, R., Shen, D., & Carin, L. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.03850 · Cited by 292 · see citing works
BAM! Born-Again Multi-Task Networks for Natural Language Understanding
Clark, K., Luong, M.-T., Khandelwal, U., Manning, C. D., & Le, Q. V. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1595 · Cited by 219 · see citing works
Self-citationCaccia, M., Caccia, L., Fedus, W., Larochelle, H., Pineau, J., & Charlin, L. (2020) · International Conference on Learning Representations · Cited by 103 · see citing works
Reducing BERT Pre-Training Time from 3 Days to 76 MinutesPreprint
You, Y., Li, J., Hseu, J., Song, X., Demmel, J. W., & Hsieh, C. (2019) · arXiv (Cornell University) · Cited by 64 · see citing works
Adversarial Contrastive Estimation
Bose, A. J., Ling, H., & Cao, Y. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1094 · Cited by 48 · see citing works
Time-Contrastive Networks: Self-Supervised Learning from VideoPreprint
Sermanet, P., Lynch, C., Chebotar, Y., Hsu, J., Jang, E., Stefan Schaal, A., & Levine, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.06888 · Cited by 41 · see citing works
Evaluating Text GANs as Language Models
Tevet, G., Habib, G., Shwartz, V., & Berant, J. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1233 · Cited by 12 · see citing works
Flagged first, then by citations
1 of the 48 references couldn’t be matched to a known paper.
Research works printed but not linked: 4
- Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS , 2014.
- Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942 , 2019. · arXiv
- Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. English gigaword, fifth edition. Technical report, Linguistic Data Consortium, Philadelphia, 2011.
- Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. MobileBERT: Task-agnostic compression of bert for resource limited devices, 2019b. URL https: //openreview.net/forum?id=SJxjVaNKwB .
Cited by
Sorted by citations
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741 · see citing works
Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., & Xie, S. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01167 · Cited by 8,494 · see citing works
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824 · see citing works
Ray, P. P. (2023) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2023.04.003 · Cited by 2,463 · see citing works
ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning
Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3095381 · Cited by 2,329 · see citing works
TinyBERT: Distilling BERT for natural language understanding
Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F., & Liu, Q. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.372 · Cited by 1,786 · see citing works
Text Data Augmentation for Deep Learning
Shorten, C., Khoshgoftaar, T. M., & Furht, B. (2021) · Journal Of Big Data · doi:10.1186/s40537-021-00492-0 · Cited by 1,798 · see citing works
Pre-trained models for natural language processing: A survey
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1647-3 · Cited by 1,591 · see citing works
Deep Learning--based Text Classification
Minaee, S., Kalchbrenner, N., Cambria, E., Nikzad-Khasmakhi, N., Asgari-Chenaghlu, M., & Gao, J. (2021) · ACM Computing Surveys · doi:10.1145/3439726 · Cited by 1,967 · see citing works
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · arXiv · doi:10.48550/arxiv.2109.00859 · Cited by 1,495 · see citing works
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.685 · Cited by 1,495 · see citing works
Unifying Large Language Models and Knowledge Graphs: A Roadmap
Pan, S., Luo, L., Wang, Y., Chen, C., Wang, J., & Wu, X. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3352100 · Cited by 1,376 · see citing works
BioGPT: generative pre-trained transformer for biomedical text generation and mining
Luo, R., Sun, L., Xia, Y., Qin, T., Zhang, S., Poon, H., & Liu, T.-Y. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac409 · Cited by 1,150 · see citing works
GLM: General Language Model Pretraining with Autoregressive Blank Infilling
Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., & Tang, J. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.26 · Cited by 1,028 · see citing works
ProteinBERT: a universal deep-learning model of protein sequence and function
Brandes, N., Ofer, D., Peleg, Y., Rappoport, N., & Linial, M. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac020 · Cited by 1,009 · see citing works
Unified Pre-training for Program Understanding and Generation
Ahmad, W., Chakraborty, S., Ray, B., & Chang, K.-W. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.211 · Cited by 642 · see citing works
Explainability for Large Language Models: A Survey
Zhao, H., Chen, H., Yang, F., Liu, N., Deng, H., Cai, H., Wang, S., Yin, D., & Du, M. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3639372 · Cited by 681 · see citing works
Less is More: CLIPBERT for Video-and-Language Learning via Sparse Sampling
Lei, J., Li, L., Zhou, L., Gan, Z., Berg, T. L., Bansal, M., & Liu, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00725 · Cited by 606 · see citing works
Impact of AI assistance on student agency
Darvishi, A., Khosravi, H., Sadiq, S., Gašević, D., & Siemens, G. (2024) · Computers & Education · doi:10.1016/j.compedu.2023.104967 · Cited by 548 · see citing works
DeBERTa: Decoding-enhanced BERT with Disentangled AttentionPreprint
He, P., Liu, X., Gao, J., & Chen, W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03654 · Cited by 1,224 · see citing works
Integrating QSAR modelling and deep learning in drug discovery: the emergence of deep QSAR
Tropsha, A., Isayev, O., Varnek, A. A., Schneider, G., & Cherkasov, A. R. (2023) · Nature Reviews Drug Discovery · doi:10.1038/s41573-023-00832-0 · Cited by 423 · see citing works
ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised LearningPreprint
Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.07.12.199554 · Cited by 409 · see citing works
He, P., Gao, J., & Chen, W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.09543 · Cited by 862 · see citing works
Nyströmformer: A Nyström-based Algorithm for Approximating Self-Attention
Xiong, Y., Zeng, Z., Chakraborty, R., Tan, M., Fung, G., Li, Y., & Singh, V. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17664 · Cited by 376 · see citing works
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations
Giorgi, J., Nitski, O., Wang, B., & Bader, G. D. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.72 · Cited by 432 · see citing works
ChatGPT in healthcare: A taxonomy and systematic review
Li, J., Dada, A., Puladi, B., Kleesiek, J., & Egger, J. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108013 · Cited by 452 · see citing works
An Empirical Study of Training End-to-End Vision-and-Language Transformers
Dou, Z.-Y., Xu, Y., Gan, Z., Wang, J., Wang, S., Wang, L., Zhu, C., Zhang, P., Yuan, L., Peng, N., et al. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01763 · Cited by 366 · see citing works
VirTex: Learning Visual Representations from Textual Annotations
Desai, K., & Johnson, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01101 · Cited by 350 · see citing works
R-Drop: Regularized Dropout for Neural NetworksPreprint
Liang, X., Wu, L., Li, J., Wang, Y., Meng, Q., Qin, T., Chen, W., Zhang, M., & Liu, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.14448 · Cited by 359 · see citing works
The language of proteins: NLP, machine learning & protein sequences
Ofer, D., Brandes, N., & Linial, M. (2021) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2021.03.022 · Cited by 345 · see citing works
A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335 · see citing works
Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges
Matsubara, Y., Levorato, M., & Restuccia, F. (2022) · ACM Computing Surveys · doi:10.1145/3527155 · Cited by 327 · see citing works
Evidence of a predictive coding hierarchy in the human brain listening to speech
Caucheteux, C., Gramfort, A., & King, J.-R. (2023) · Nature Human Behaviour · doi:10.1038/s41562-022-01516-2 · Cited by 308 · see citing works
Log-based Anomaly Detection Without Log Parsing
Le, V.-H., & Zhang, H. (2021) · 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678773 · Cited by 289 · see citing works
Trends in AI inference energy consumption: Beyond the performance-vs-parameter laws of deep learning
Desislavov, R., Martínez-Plumed, F., & Hernández-Orallo, J. (2023) · Sustainable Computing: Informatics and Systems · doi:10.1016/j.suscom.2023.100857 · Cited by 269 · see citing works
MultiMAE: Multi-modal Multi-task Masked Autoencoders
Bachmann, R., Mizrahi, D., Atanov, A., & Zamir, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19836-6_20 · Cited by 268 · see citing works
A Survey on Text Classification Algorithms: From Text to Predictions
Gasparetto, A., Marcuzzo, M., Zangari, A., & Albarelli, A. (2022) · Information · doi:10.3390/info13020083 · Cited by 278 · see citing works
Scaling Language-Image Pre-Training via Masking
Li, Y., Fan, H., Hu, R., Feichtenhofer, C., & He, K. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.02240 · Cited by 255 · see citing works
Revisiting Few-sample BERT Fine-tuningPreprint
Zhang, T., Wu, F., Katiyar, A., Weinberger, K. Q., & Artzi, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.05987 · Cited by 295 · see citing works
A benchmark study of machine learning models for online fake news detection
Khan, J. Y., Khondaker, M. T. I., Afroz, S., Uddin, G., & Iqbal, A. (2021) · Machine Learning with Applications · doi:10.1016/j.mlwa.2021.100032 · Cited by 302 · see citing works
Are All Languages Created Equal in Multilingual BERT?
Wu, S., & Dredze, M. H. (2020) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2020.repl4nlp-1.16 · Cited by 264 · see citing works
Hopfield Networks is All You Need
Ramsauer, H., Schäfl, B., Lehner, J., Seidl, P., Widrich, M., Adler, T., Gruber, L., Holzleitner, M., Pavlović, M., Sandve, G. K., et al. (2020) · arXiv · doi:10.48550/arxiv.2008.02217 · Cited by 236 · see citing works
A Fine-Tuned BERT-Based Transfer Learning Approach for Text Classification
Qasim, R., Bangyal, W. H., Alqarni, M., & Almazroi, A. A. (2022) · Journal of Healthcare Engineering · doi:10.1155/2022/3498123 · Cited by 256 · see citing works
TabTransformer: Tabular Data Modeling Using Contextual EmbeddingsPreprint
Huang, X., Khetan, A., Cvitkovic, M. W., & Karnin, Z. S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.06678 · Cited by 591 · see citing works
Text Classification via Large Language Models
Sun, X., Li, X., Li, J., Wu, F. Q., Guo, S., Zhang, T., & Wang, G. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.603 · Cited by 273 · see citing works
ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information
Sun, Z., Li, X., Sun, X., Meng, Y., Ao, X., He, Q., Wu, F. Q., & Li, J. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.161 · Cited by 214 · see citing works
Self-Attention Attribution: Interpreting Information Interactions Inside Transformer
Hao, Y., Dong, L., Wei, F., & Xu, K. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i14.17533 · Cited by 217 · see citing works
Transformers and large language models in healthcare: A review
Nerella, S., Bandyopadhyay, S., Zhang, J., Contreras, M., Siegel, S., Bumin, A., Silva, B., Sena, J., Shickel, B., Bihorac, A., et al. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102900 · Cited by 243 · see citing works
Masked Autoencoders Are Scalable Vision LearnersPreprint
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., & Girshick, R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.06377 · Cited by 200 · see citing works
Bhattacharjee, A., Hasan, T., Ahmad, W. U., Mubasshir, K. S., Islam, M. S., Iqbal, A., Rahman, M. S., & Shahriyar, R. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.98 · Cited by 201 · see citing works
Chan, B., Schweter, S., & Möller, T. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.598 · Cited by 208 · see citing works
Retrospective Reader for Machine Reading Comprehension
Zhang, Z., Yang, J., & Zhao, H. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17705 · Cited by 200 · see citing works
Rethinking Positional Encoding in Language Pre-trainingPreprint
Ke, G., He, D., & Liu, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.15595 · Cited by 223 · see citing works
Knowledge Neurons in Pretrained Transformers
Dai, D., Dong, L., Hao, Y., Sui, Z., Chang, B., & Wei, F. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.581 · Cited by 187 · see citing works
Ankh ☥: Optimized Protein Language Model Unlocks General-Purpose ModellingPreprint
Elnaggar, A., Essam, H., Salah-Eldin, W., Moustafa, W., Elkerdawy, M., Rochereau, C., & Rost, B. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.01.16.524265 · Cited by 201 · see citing works
Muppet: Massive Multi-task Representations with Pre-Finetuning
Aghajanyan, A., Gupta, A., Shrivastava, A., Chen, X., Zettlemoyer, L., & Gupta, S. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.468 · Cited by 183 · see citing works
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings
Chuang, Y.-S., Dangovski, R., Luo, H., Zhang, Y., Chang, S., Marin Soljacic, A., Li, S.-W. D., Yih, S., Kim, Y., & Glass, J. R. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.311 · Cited by 174 · see citing works
Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning
Henderson, P., Hu, J., Romoff, J., Brunskill, E., Jurafsky, D., & Pineau, J. (2020) · arXiv · doi:10.48550/arxiv.2002.05651 · Cited by 393 · see citing works
Coreferential Reasoning Learning for Language Representation
Ye, D., Lin, Y., Du, J., Liu, Z., Li, P., Sun, M., & Liu, Z. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.582 · Cited by 174 · see citing works
CLIMATEBERT: A Pretrained Language Model for Climate-Related TextPreprint
Webersinke, N., Kraus, M., Bingler, J., & Leippold, M. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4229146 · Cited by 231 · see citing works
Traceability Transformed: Generating More Accurate Links with Pre-Trained BERT Models
Lin, J., Liu, Y., Zeng, Q., Jiang, M., & Cleland‐Huang, J. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00040 · Cited by 167 · see citing works
Cyberbullying detection for low-resource languages and dialects: Review of the state of the art
Mahmud, T., Ptaszynski, M. E., Eronen, J., & Masui, F. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103454 · Cited by 172 · see citing works
The NLP Cookbook: Modern Recipes for Transformer Based Deep Learning Architectures
Singh, S. K., & Mahmood, A. (2021) · IEEE Access · doi:10.1109/access.2021.3077350 · Cited by 164 · see citing works
Automatic Detection of Machine Generated Text: A Critical Survey
Jawahar, G., Abdul-Mageed, M., & Lakshmanan, V. S. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.208 · Cited by 156 · see citing works
N-LTP: An Open-source Neural Language Technology Platform for Chinese
Che, W., Feng, Y., Qin, L., & Liu, T. (2021) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2021.emnlp-demo.6 · Cited by 152 · see citing works
No more fine-tuning? an experimental evaluation of prompt tuning in code intelligence
Wang, C., Yang, Y., Gao, C., Peng, Y. F., Zhang, H., & Lyu, M. R.-T. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549113 · Cited by 153 · see citing works
Recommending Root-Cause and Mitigation Steps for Cloud Incidents using Large Language Models
Ahmed, T., Ghosh, S., Bansal, C., Zimmermann, T., Zhang, X., & Rajmohan, S. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00149 · Cited by 148 · see citing works
Utility is in the Eye of the User: A Critique of NLP Leaderboards
Ethayarajh, K., & Jurafsky, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.393 · Cited by 147 · see citing works
LGESQL: Line Graph Enhanced Text-to-SQL Model with Mixed Local and Non-Local Relations
Cao, R., Chen, L., Chen, Z., Zhao, Y., Zhu, S., & Yu, K. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.198 · Cited by 163 · see citing works
Siino, M., Tinnirello, I., & Cascia, M. L. (2023) · Information Systems · doi:10.1016/j.is.2023.102342 · Cited by 208 · see citing works
A Practical Survey on Faster and Lighter Transformers
Fournier, Q., Caron, G. M., & Aloise, D. (2023) · ACM Computing Surveys · doi:10.1145/3586074 · Cited by 145 · see citing works
Should You Mask 15% in Masked Language Modeling?
Wettig, A., Gao, T., Zhong, Z., & Chen, D. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.217 · Cited by 161 · see citing works
DynaBERT: Dynamic BERT with Adaptive Width and Depth
Hou, L., Huang, Z., Shang, L., Jiang, X., Chen, X., & Liu, Q. (2020) · arXiv · doi:10.48550/arxiv.2004.04037 · Cited by 143 · see citing works
Mole-BERT: Rethinking Pre-training Graph Neural Networks for MoleculesPreprint
Xia, J., Zhao, C., Hu, B., Gao, Z., Tan, C., Liu, Y., Li, S., & Li, S. Z. (2023) · ChemRxiv · doi:10.26434/chemrxiv-2023-dngg4 · Cited by 141 · see citing works
Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning
Xu, R., Luo, F., Zhang, Z., Tan, C., Chang, B., Huang, S., & Huang, F. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.749 · Cited by 137 · see citing works
ConvBERT: Improving BERT with Span-based Dynamic ConvolutionPreprint
Jiang, Z., Yu, W., Zhou, D., Chen, Y., Feng, J., & Yan, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.02496 · Cited by 146 · see citing works
Boukkouri, H. E., Ferret, O., Lavergne, T., Noji, H., Zweigenbaum, P., & Tsujii, J. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.609 · Cited by 152 · see citing works
COCO-LM: Correcting and Contrasting Text Sequences for Language Model PretrainingPreprint
Yu, M., Xiong, C., Bajaj, P., Tiwary, S., Bennett, P. N., Han, J., & Song, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.08473 · Cited by 133 · see citing works
TABBIE: Pretrained Representations of Tabular Data
Iida, H., Thai, D., Manjunatha, V., & Iyyer, M. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.270 · Cited by 130 · see citing works
The Stem Cell Hypothesis: Dilemma behind Multi-Task Learning with Transformer Encoders
He, H. W., & Choi, J. D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.451 · Cited by 130 · see citing works
Deep contextualized text representation and learning for fake news detection
Samadi, M., Mousavian, M., & Momtazi, S. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102723 · Cited by 130 · see citing works
Scientific Large Language Models: A Survey on Biological & Chemical Domains
Zhang, Q., Ding, K., Lv, T., Wang, X., Yin, Q., Zhang, Y., Jing, Y., Wang, Y., Li, X., Xiang, Z., et al. (2025) · ACM Computing Surveys · doi:10.1145/3715318 · Cited by 155 · see citing works
Pix2seq: A Language Modeling Framework for Object DetectionPreprint
Chen, T., Saxena, S., Li, L., Fleet, D. J., & Hinton, G. E. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.10852 · Cited by 159 · see citing works
Hubert: How Much Can a Bad Teacher Benefit ASR Pre-Training?
Hsu, W.-N., Tsai, Y.-H. H., Bolte, B., Salakhutdinov, R., & Mohamed, A. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414460 · Cited by 123 · see citing works
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
Fabbri, A., Wu, C.-S., Liu, W., & Xiong, C. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.187 · Cited by 119 · see citing works
BioELECTRA:Pretrained Biomedical text Encoder using Discriminators
Kanakarajan, K. R., Kundumani, B., & Sankarasubbu, M. (2021) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2021.bionlp-1.16 · Cited by 118 · see citing works
Neural Architecture Search for Transformers: A Survey
Chitty-Venkata, K. T., Emani, M., Vishwanath, V., & Somani, A. K. (2022) · IEEE Access · doi:10.1109/access.2022.3212767 · Cited by 118 · see citing works
A Survey on Contextual EmbeddingsPreprint
Liu, Q., Kusner, M. J., & Blunsom, P. (2020) · arXiv (Cornell University) · doi:10.4230/oasics.slate.2021.12 · Cited by 178 · see citing works
SqueezeBERT: What can computer vision teach NLP about efficient neural networks?
Iandola, F., Shaw, A., Krishna, R., & Keutzer, K. (2020) · Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing · doi:10.18653/v1/2020.sustainlp-1.17 · Cited by 114 · see citing works
Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval
Xiong, W., Li, X. L., Iyer, S., Du, J., Lewis, P., Wang, W. Y., Mehdad, Y., Yih, W.-T., Riedel, S., Kiela, D., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.12756 · Cited by 112 · see citing works
CLEVE: Contrastive Pre-training for Event Extraction
Wang, Z., Wang, X., Han, X., Lin, Y., Hou, L., Liu, Z., Li, P., Li, J., & Zhou, J. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.491 · Cited by 121 · see citing works
LI-EMRSQL: Linking Information Enhanced Text2SQL Parsing on Complex Electronic Medical Records
Li, Q., You, T., Chen, J., Zhang, Y., & Du, C. (2023) · IEEE Transactions on Reliability · doi:10.1109/tr.2023.3336330 · Cited by 111 · see citing works
An Overview on Generative AI at Scale With Edge–Cloud Computing
Wang, Y.-C., Xue, J., Wei, C., & Kuo, C. ‐. J. (2023) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2023.3320646 · Cited by 117 · see citing works
KLUE: Korean Language Understanding EvaluationPreprint
Park, S., Moon, J., Kim, S., Cho, W. I., Han, J., Park, J., Song, C., Kim, J., Song, Y., Oh, T., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.09680 · Cited by 153 · see citing works
Few-Shot Named Entity Recognition: An Empirical Baseline Study
Huang, J., Li, C., Subudhi, K., Jose, D., Balakrishnan, S., Chen, W., Peng, B., Gao, J., & Han, J. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.813 · Cited by 108 · see citing works
E-Branchformer: Branchformer with Enhanced Merging for Speech Recognition
Kim, K., Wu, F., Peng, Y., Pan, J. S., Sridhar, P., Han, K. J., & Watanabe, S. (2023) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt54892.2023.10022656 · Cited by 107 · see citing works
RealFormer: Transformer Likes Residual Attention
He, R., Ravula, A., Kanagal, B., & Ainslie, J. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.81 · Cited by 114 · see citing works
SemEval-2022 Task 11: Multilingual Complex Named Entity Recognition (MultiCoNER)
Malmasi, S., Fang, A., Fetahu, B., Kar, S., & Rokhlenko, O. (2022) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2022.semeval-1.196 · Cited by 104 · see citing works
Multilingual Speech Translation from Efficient Finetuning of Pretrained Models
Li, X., Wang, C., Tang, Y., Tran, C., Tang, Y., Pino, J., Baevski, A., Conneau, A., & Auli, M. (2021) · doi:10.18653/v1/2021.acl-long.68 · Cited by 107 · see citing works
- Authors
- Clark, K., Luong, M.-T., Le, Q. V., & Manning, C. D.
- Affiliations
- Stanford University (US)
- Google (United States) (US)
- ORCID
- Le, Quoc V.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2003.10555
- OpenAlex
- W2996035354
- Cited by
- 1,844
- References
- 48
- Citations by year
- 2026: 13 · 2025: 129 · 2024: 240 · 2023: 572 · 2022: 415 · 2021: 375 · 2020: 100
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Learning Representations
- Funders
- Canadian Institute for Advanced Research
- Programa Operacional Regional do Centro (Portugal 2020)
- Fundação para a Ciência e a Tecnologia (PT2020, -01-0247-, 00326, UID/CEC/00326/2020)
- European Regional Development Fund (PT2020)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- Electra, masked language modeling, text encoder, sample efficiency, context representation, natural language understanding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
