Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach
Open accessThe DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Abstract
Wenpeng Yin, Jamaal Hay, Dan Roth. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This is evidenced by LLMs impressive zero-shot and few-shot learning ability [18,132], as well as the fact that in-context learning does not require weight update, which mechanically prevents unfavorable fine-tuning-related behavior like catastrophic forgetting.”
From Harnessing the power of llms in practice: a survey on chatGPT and beyond (2024) · cites this paper as [132]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs have been shown to outperform previous zero-shot methods [132].”
From Harnessing the power of llms in practice: a survey on chatGPT and beyond (2024) · cites this paper as [132]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 6×
“Regarding this label-fully-unseen evaluation, apart from our entailment models and three unsupervised baselines “Majority”, “Word2Vec” and “ESA”, we also report the following baseline: Wikipedia-based : We train a binary clas-sifier based on BERT on a dataset collected from Wikipedia.”
MethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 6×
“Regarding this label-fully-unseen evaluation, apart from our entailment models and three unsupervised baselines “Majority”, “Word2Vec” and “ESA”, we also report the following baseline: Wikipedia-based : We train a binary clas-sifier based on BERT on a dataset collected from Wikipedia.”
MethodInfluentialDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 2×
“• Word2Vec 5 (Mikolov et al., 2013): Both the representations of the text and the labels are the addition of word embeddings element-wisely.”
BackgroundGLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222 · cited 11×
“For label-partially-unseen setup in which we intentionally provide annotated data, we first pretrain BERT on the MNLI/FEVER/RTE, then fine-tune on the provided training data.”
BackgroundMethodInfluentialA Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342 · cited 2×
“, 2019), and train it on three mainstream entailment datasets: MNLI (Williams et al., 2018), GLUE RTE (Dagan et al.”
BackgroundMethodCharacter-level Convolutional Networks for Text Classification
Zhang, X., Zhao, J., & LeCun, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.01626 · Cited by 3,205 · cited 1×
“We use the large-scale Yahoo dataset released by Zhang et al. (2015).”
MethodComputing semantic relatedness using Wikipedia-based explicit semantic analysis
Gabrilovich, E., & Markovitch, S. (2007) · Cited by 1,981
The PASCAL Recognising Textual Entailment Challenge
Dagan, I., Glickman, O., & Magnini, B. (2006) · Lecture Notes in Computer Science · doi:10.1007/11736790_9 · Cited by 1,628
(2018) · International Conference on Learning Representations · Cited by 1,621
FEVER: a Large-scale Dataset for Fact Extraction and VERification
Thorne, J., Vlachos, A., Christodoulopoulos, C., & Mittal, A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1074 · Cited by 1,343 · cited 1×
“In this work, we make use of the widely-recognized state of the art entailment technique – BERT (Devlin et al., 2019), and train it on three mainstream entailment datasets: MNLI (Williams et al., 2018), GLUE RTE (Dagan et al., 2005; Wang et al., 2019) and FEVER 3 (Thorne et al., 2018), respectively.”
Method
Research works printed but not linked: 2
- [8]8. Chenliang Li, Jian Xing, Aixin Sun, and Zongyang Ma. 2016a. Effective document labeling with very few seed words: A topic model approach. In CIKM, pages 85-94.
- [11]11. Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed repre- sentations of words and phrases and their composi- tionality. In NIPS, pages 3111-3119.
Cited by
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey
Min, B., Ross, H., Sulem, E., Veyseh, A. P. B., Nguyen, T. H., Sainz, O., Agirre, E., Heintz, I., & Roth, D. (2024) · ACM Computing Surveys · doi:10.1145/3605943 · Cited by 1,387
Self-citationExploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Schick, T., & Schütze, H. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.20 · Cited by 1,305
Finetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101
“Our work also falls in the well-established category of zero-shot learning, which has historically been used to refer to classifying instances among a set of unseen categories (Lampert et al., 2009; Romera-Paredes & Torr, 2015; Srivastava et al., 2018; Yin et al., 2019, inter alia).”
BackgroundTop2Vec: Distributed Representations of TopicsPreprint
Angelov, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.09470 · Cited by 944
“The Yahoo Answers dataset [36, 37], contains 1.3 million labelled posts.”
BackgroundInfluentialHarnessing the power of llms in practice: a survey on chatGPT and beyond
Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., Zhong, S., Yin, B., & Hu, X. (2024) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3649506 · Cited by 644
“LLMs have been shown to outperform previous zero-shot methods [132].”
BackgroundMethodHarnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond
Yang, J., Jin, H., Tang, R., Han, X., Feng, Q., Jiang, H., Yin, B., & Hu, X. (2023) · arXiv · doi:10.48550/arxiv.2304.13712 · Cited by 644
“LLMs have been shown to outperform previous zero-shot methods [132].”
BackgroundMethodLiu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352
Evaluating and Inducing Personality in Pre-trained Language Models
Jiang, G., Xu, M., Zhu, S. C., Zhang, C., Zhang, C., & Zhu, Y. (2023) · neural information processing systems · doi:10.52202/075280-0466 · Cited by 65
“Following Yin et al. (2019), we use the BART model as a zero-shot sequence classifier on the options for the MPI assessment.”
MethodGenerating Training Data with Language Models: Towards Zero-Shot Language Understanding
Meng, Y., Huang, J., Zhang, Y. B., & Han, J. (2022) · neural information processing systems · doi:10.52202/068431-0034 · Cited by 336
CODA-Prompt: COntinual Decomposed Attention-Based Prompting for Rehearsal-Free Continual Learning
Smith, J. S., Karlinsky, L., Gutta, V., Cascante-Bonilla, P., Kim, D., Arbelle, A., Panda, R., Feris, R., & Kira, Z. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01146 · Cited by 334
Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification
Hu, S., Ding, N., Wang, H., Liu, Z., Wang, J., Li, J., Wu, W., & Sun, M. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.158 · Cited by 265
LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models
Parmar, M., Patel, N. A., Varshney, N., Nakamura, M., Luo, M., Mashetty, S., Mitra, A., & Baral, C. R. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.739 · Cited by 237
Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections
Zhong, R., Lee, K., Zhang, Z., & Klein, D. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.244 · Cited by 99
Text Classification Using Label Names Only: A Language Model Self-Training Approach
Meng, Y., Zhang, Y., Huang, J., Xiong, C., Ji, H., Zhang, C., & Han, J. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.724 · Cited by 196
Large Language Model Instruction Following: A Survey of Progresses and Challenges
Lou, R., Zhang, K., & Yin, W. (2024) · Computational Linguistics · doi:10.1162/coli_a_00523 · Cited by 198
“Our contributions are three-fold: • Going beyond prompts, we analyze prompt constraints via a user-centric lens, with a focus on discerning the disparity between current instruction following research and real-world needs.”
BackgroundMethodInfluentialSelf-citationLabel Verbalization and Entailment for Effective Zero and Few-Shot Relation Extraction
Sainz, O., de Lacalle, O. L., Labaka, G., Barrena, A., & Agirre, E. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.92 · Cited by 179
“Closer to our work, NLI has been shown too to be a successful pivoting task for text classification (Yin et al., 2019, 2020; Wang et al., 2021; Sainz and Rigau, 2021).”
BackgroundMethodEvaluating the effectiveness of large language models in abstract screening: a comparative analysis
Li, M., Sun, J., & Tan, X. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02609-x · Cited by 125
Laurer, M., van Atteveldt, W., Casas, A., & Welbers, K. (2024) · Political Analysis · doi:10.1017/pan.2023.20 · Cited by 166
“This paper uses one approach, based on natural language inference (NLI), first proposed by Yin, Hay, and Roth (2019) and later refined, for example, by Wang et al .”
Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text ClassificationPreprint
Hu, S., Ding, N., Wang, H., Liu, Z., Wang, J., Li, J., Wu, W., & Sun, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.02035 · Cited by 168
“However, since the extra classifier requires adequate training instances to tune, it is still challenging to apply fine-tuning in few-shot learning (Brown et al., 2020) and zero-shot learning (Yin et al., 2019) scenarios.”
BackgroundAdvances in Sentiment Analysis - Techniques, Applications, and Challenges
Li, J. (2023) · Artificial intelligence · doi:10.5772/intechopen.111293 · Cited by 145
Evaluating Unsupervised Text Classification: Zero-shot and Similarity-based Approaches
Schopf, T., Braun, D., & Matthes, F. (2022) · International Conference on Natural Language Processing and Information Retrieval (NLPIR) · doi:10.1145/3582768.3582795 · Cited by 143
“The highly successful deep learning performances of recent years have also stimulated research initiatives for 0SHOT-TC [17, 24, 27, 42, 44].”
InfluentialZero-shot learning for requirements classification: An exploratory study
Alhoshan, W., Ferrari, A., & Zhao, L. (2023) · Information and Software Technology · doi:10.1016/j.infsof.2023.107202 · Cited by 143
Legal Prompt Engineering for Multilingual Legal Judgement PredictionPreprint
Trautmann, D., Petrova, A., & Schilder, F. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.02199 · Cited by 143
“1, the Completion) in a zero-shot fashion (Yin et al., 2019; Sanh et al., 2021).”
BackgroundTransZero++: Cross Attribute-Guided Transformer for Zero-Shot Learning
Chen, S., Hong, Z., Hou, W., Xie, G.-S., Song, Y., Zhao, J. C., You, X., Yan, S., & Shao, L. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3229526 · Cited by 133
“ZSL is typically denoted as zero-shot image classification or object recognition [3], [4], which is different to zero-shot text classification [20] or zero-shot transfer of model [21].”
BackgroundExtracting Cultural Commonsense Knowledge at Scale
Nguyen, T.-P., Razniewski, S., Varde, A. S., & Weikum, G. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583535 · Cited by 132
Zero-shot Event Extraction via Transfer Learning: Challenges and Insights
Lyu, Q., Zhang, H., Sulem, E., & Roth, D. (2021) · doi:10.18653/v1/2021.acl-short.42 · Cited by 129
“Then, for each text piece, we pass it to the TE model as the premise, coupled with a hypothesis in the format of “ This text is about ... ” for each event type, inspired by Yin et al. (2019).”
MethodSelf-citationZero-Shot and Few-Shot Learning With Knowledge Graphs: A Comprehensive Survey
Chen, J., Geng, Y., Chen, Z., Pan, J. Z., He, Y., Zhang, W., Horrocks, I. R., & Chen, H. (2023) · Proceedings of the IEEE · doi:10.1109/jproc.2023.3279374 · Cited by 119
“Although there have been some benchmarking studies for ZSL and FSL [26, 192], systematic evaluation and comparison of KG-aware methods is still not enough.”
BackgroundRespectful or Toxic? Using Zero-Shot Learning with Language Models to Detect Hate Speech
Plaza-del-Arco, F. M., Nozza, D., & Hovy, D. (2023) · doi:10.18653/v1/2023.woah-1.6 · Cited by 114
“(Goldzycher and Schneider, 2022) develop NLI-based zero-shot hate speech detection approaches using prompts as a hypothesis as proposed by Yin et al. (2019).”
MethodManias, G., Mavrogiorgou, A., Kiourtis, A., Symvoulidis, C., & Kyriazis, D. (2023) · Neural Computing and Applications · doi:10.1007/s00521-023-08629-3 · Cited by 113
True Few-Shot Learning with Prompts—A Real-World Perspective
Schick, T., & Schütze, H. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00485 · Cited by 111
“While most approaches rephrase tasks as a language modeling problem, some use prompts to reformulate them as different tasks for which large amounts of training data are available (Levy et al., 2017; McCann et al., 2018; Yin et al., 2019; Sun et al., 2021; Sainz et al., 2021).”
BackgroundZero-Shot Text Classification with Self-Training
Gera, A., Halfon, A., Shnarch, E., Perlitz, Y., Ein‐Dor, L., & Slonim, N. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.73 · Cited by 106
Language Models for Text Classification: Is In-Context Learning Enough?
Edwards, A., & Camacho-Collados, J. (2024) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/3dnziwdd5bh2 · Cited by 106
Large Language Models for Text Classification: From Zero-Shot Learning to Instruction-Tuning
Chae, Y., & Davidson, T. (2025) · Sociological Methods & Research · doi:10.1177/00491241251325243 · Cited by 102
Zero-shot Text Classification via Reinforced Self-training
Ye, Z., Geng, Y., Chen, J., Chen, J., Xu, X., Zheng, S., Wang, F., Zhang, J., & Chen, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.272 · Cited by 76
“The first comes from the recently released benchmarks for zero-shot text classification (Yin et al., 2019), including 3 datasets: topic, emotion and situation classification.”
BackgroundMethodInfluentialCRASS: A Novel Data Set and Benchmark to Test Counterfactual Reasoning of Large Language Models
Frohberg, J., & Binder, F. (2022) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/34bje4jp4zxa · Cited by 100
Learning from Task Descriptions
Weller, O., Lourie, N., Gardner, M., & Peters, M. E. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.105 · Cited by 58
TaxoClass: Hierarchical Multi-Label Text Classification Using Only Class Names
Shen, J., Qiu, W., Yu, M., Shang, J., Ren, X., & Han, J. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.335 · Cited by 98
“1 Document-Class Similarity Calculation We take a textual entailment approach (Yin et al., 2019) to calculate the semantic similarity between each 〈document, class〉 pair.”
InfluentialKnowledge-aware Zero-Shot Learning: Survey and Perspective
Chen, J., Geng, Y., Chen, Z., Horrocks, I., Z. Pan, J., & Chen, H. (2021) · Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/597 · Cited by 52
Building Efficient Universal Classifiers with Natural Language InferencePreprint
Laurer, M., van Atteveldt, W., Casas, A., & Welbers, K. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.17543 · Cited by 98
“Several papers have used the universal NLI task for zero-and fewshot classification, but stopped short of mixing NLI data with multiple other non-NLI datasets to build more universal classifiers (Yin et al., 2019, 2020; Wang et al., 2021; Laurer et al., 2023a).”
MethodPellert, M., Lechner, C. M., Wagner, C., Rammstedt, B., & Strohmaier, M. (2024) · Perspectives on Psychological Science · doi:10.1177/17456916231214460 · Cited by 97
“Different from few-shot classification, models fine-tuned on NLI corpora are able to classify sequences into categories chosen on the fly or to summarize arbitrary texts (Yin et al., 2019) without the need to be presented with any labeled examples before.”
BackgroundX-Class: Text Classification with Extremely Weak Supervision
Wang, Z., Mekala, D., & Shang, J. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.242 · Cited by 97
Tesfagergish, S. G., Kapočiūtė-Dzikienė, J., & Damaševičius, R. (2022) · Applied Sciences · doi:10.3390/app12178662 · Cited by 97
Estimating the Personality of White-Box Language ModelsPreprint
Karra, S. R., Nguyen, S. T., & Tulabandhula, T. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4598766 · Cited by 94
Forecasting container freight rates using the Prophet forecasting method
Saeed, N., Nguyen, S., Cullinane, K., Gekara, V. O., & Chhetri, P. (2023) · Transport Policy · doi:10.1016/j.tranpol.2023.01.012 · Cited by 91
Zhang, Y., Yang, R., Xu, X., Li, R., Xiao, J., Shen, J., & Han, J. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3696410.3714940 · Cited by 91
“We only report Example-F1 and P@k, because it is not straightforward to get ranking of classes predicted by LLMs for MRR calculation.”
InfluentialMayer, C. W. F., Ludwig, S., & Brandt, S. (2023) · Journal of Research on Technology in Education · doi:10.1080/15391523.2022.2142872 · Cited by 57
“Prompts are short pieces of texts in natural language that can include examples of the task to be completed (few-shot learning) or just an instruction without examples (zero-shot learning; see Yin et al., 2019).”
BackgroundMethodInfluentialRAFT: A Real-World Few-Shot Text Classification BenchmarkPreprint
Alex, N., Lifland, E., Tunstall, L., Thakur, A., Maham, P., Riedel, C. J., Hine, E., Ashurst, C., Sedille, P., Carlier, A., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.14076 · Cited by 83
“[38] and Davison [9] as an effective zero-shot classification approach”
Yin, W., Rajani, N. F., Radev, D., Socher, R., & Xiong, C. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.660 · Cited by 55
Self-citationOpen Aspect Target Sentiment Classification with Natural Language Prompts
Seoh, R., Birle, I., Tak, M., Chang, H.-S., Pinette, B., & Hough, A. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.509 · Cited by 74
“Our method enables zeroshot ATSC, which previously had not considered by previous efforts that leveraged NLI for sentiment analysis and text classification tasks (Yin et al., 2019; Sun et al., 2019; Wang et al., 2021).”
MethodEfficient (Soft) Q-Learning for Text Generation with Limited Good Data
Guo, H., Tan, B., Liu, Z., Xing, E. P., & Hu, Z. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.518 · Cited by 72
“Prominent examples of such low-data problems include generating prompts to control the massive LMs (Yin et al., 2019; Shin et al., 2020; Zhong et al., 2021; Liu et al., 2021), learning text generation from noisy or even negative data, generating adversarial text attacks for robustness study (Wallace et al., 2019; Atan…”
BackgroundInfluentialaeroBERT-Classifier: Classification of Aerospace Requirements Using BERT
Ray, A. T., Cole, B. F., Fischer, O. J. P., White, R. T., & Mavris, D. N. (2023) · Aerospace · doi:10.3390/aerospace10030279 · Cited by 71
Large Language Models for Text Classification: From Zero-Shot Learning to Instruction-TuningPreprint
Chae, Y., & Davidson, T. (2023) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/sthwk · Cited by 71
Hadi, M. A., & Fard, F. H. (2023) · Empirical Software Engineering · doi:10.1007/s10664-023-10314-x · Cited by 71
“We follow the steps in [19] to prepare the datasets and models and set up our study for ZSL.”
BackgroundMethodInfluentialZero-Shot Text Classification via Knowledge Graph Embedding for Social Media Data
Chen, Q., Wang, W., Huang, K., & Coenen, F. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3093065 · Cited by 70
“• BART-NLI: The study presented in [28] showed that zero-shot text classification can be modeled in a natural language inference architecture, where the hypothesis is”
BackgroundMethodZero-shot Label-Aware Event Trigger and Argument Classification
Zhang, H., Wang, H., & Roth, D. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.114 · Cited by 72
“Solving NLP problems without using annotations has been explored in many NLP tasks including text classification (Chang et al., 2008; Yin et al., 2019), entity typing (Zhou et al., 2018), sequence classification (Rei and S o gaard, 2018), and intent detection (Xia et al., 2018).”
MethodSelf-citationSutter, T. S., Bozkır, A. S., Gehring, B., & Berlich, P. (2022) · IEEE Access · doi:10.1109/access.2022.3207272 · Cited by 67
“In this work, we leveraged the approach suggested by [108], which employs a pre-trained Multi Natural Lan-938 guage Inference sequence-pair classifier as a zero-shot text classifier.”
MethodWeakly-supervised Text Classification Based on Keyword Graph
Zhang, L., Ding, J., Xu, Y., Liu, Y., & Zhou, S. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.222 · Cited by 66
“…supervision signals used by existing methods includes external knowledge base (Gabrilovich et al., 2007; Chang et al., 2008; Song and Roth, 2014; Yin et al., 2019), key-words (Agichtein and Gravano, 2000; Riloff et al., 2003; Kuipers et al., 2006; Tao et al., 2015; Meng et al., 2018, 2019,…”
MethodStance detection: a practical guide to classifying political beliefs in text
Burnham, M. (2024) · Political Science Research and Methods · doi:10.1017/psrm.2024.35 · Cited by 67
“As shown in Table 4, performance comparable to supervised classifiers only emerges in larger models.”
Evaluating and Benchmarking the System One Model JevPreprint
Deußer, T., Ramamurthy, R., Sparrenberg, L., & Sifa, R. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2609.37647 · Cited by 66
“Framing classification as natural language inference [8] and prompting generative LLMs [9] made it possible to solve many classification and multiple-choice tasks without task-specific training, a setting known as “zero-shot”.”
Building trustworthy NeuroSymbolic AI Systems: Consistency, reliability, explainability, and safety
Gaur, M., & Sheth, A. (2024) · AI Magazine · doi:10.1002/aaai.12149 · Cited by 65
Xia, C., Yin, W., Feng, Y., & Yu, P. S. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.106 · Cited by 62
Self-citationAdaPrompt: Adaptive Model Training for Prompt-based NLP
Chen, Y., Liu, Y., Li, D., Wang, S., Zhu, C., Zeng, M., & Zhang, Y. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.448 · Cited by 61
Pi, X., Wang, B., Gao, Y., Guo, J., Li, Z., & Lou, J. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.142 · Cited by 61
Sun, D., Boudouaia, A., Yang, J., & Xu, J. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-03991-6 · Cited by 61
Leveraging Type Descriptions for Zero-shot Named Entity Recognition and Classification
Aly, R., Vlachos, A., & McDonald, R. (2021) · doi:10.18653/v1/2021.acl-long.120 · Cited by 58
“Similar observation applies to similarity scores based on word2vec embeddings (Mikolov et al., 2013) as used in (Yin et al., 2019), highlighting the difficulty of this task.”
BackgroundMethodInfluentialMapRE: An Effective Semantic Mapping Approach for Low-resource Relation Extraction
Dong, M., Pan, C., & Luo, Z. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.212 · Cited by 58
“, 2019), which assumes the label vectors take an equally crucial role as the representations of the support instances (Yin et al., 2019).”
Few-Shot Learning with Siamese Networks and Label Tuning
Müller, T., Pérez-Torró, G., & Franco-Salvador, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.584 · Cited by 58
Textual Entailment for Event Argument Extraction: Zero- and Few-Shot with Multi-Source Learning
Sainz, O., González-Dios, I., de Lacalle, O. L., Min, B., & Agirre, E. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.187 · Cited by 57
Handbook of Computational Social Science for Policy
Bertoni, E., Fontana, M., Gabrielli, L., Signorelli, S., & Vespe, M. (2023) · Springer eBooks · doi:10.1007/978-3-031-16624-2 · Cited by 57
“There are many off-the-shelf existing tools for text classification (Yin et al., 2019; Brown et al., 2020; Loria, 2018) such as the implementation1 using the Python package transformers (Wolf et al., 2020).”
BackgroundMethodInfluentialSpeedy activists: How firm response time to sociopolitical events influences consumer behavior
Nam, J., Balakrishnan, M., Freitas, J. D., & Brooks, A. W. (2023) · Journal of Consumer Psychology · doi:10.1002/jcpy.1380 · Cited by 56
EPIC: Multi-Perspective Annotation of a Corpus of Irony
Frenda, S., Pedrani, A., Basile, V., Lo, S. M., Cignarella, A. T., Panizzon, R., Marco, C. S., Scarlini, B., Patti, V., Bosco, C., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.774 · Cited by 56
“Then, we followed the approach described by Yin et al. (2019) and used a pre-trained Natural Language Inference model as a zero-shot sequence classifier to classify our instances into the above list of topics.”
MethodA Search Engine for Discovery of Scientific Challenges and Directions
Lahav, D., Saad-Falcon, J., Kuehl, B., Johnson, S., Parasa, S., Shomron, N., Chau, D. H., Yang, D., Horvitz, E., Weld, D. S., et al. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21456 · Cited by 55
“• Zero-shot inference : In zero-shot classification, models predict labels they were not trained on (Yin, Hay, and Roth 2019).”
MethodIssues with Entailment-based Zero-shot Text Classification
Ma, T., Yao, J.-G., Lin, C.-Y., & Zhao, T. (2021) · doi:10.18653/v1/2021.acl-short.99 · Cited by 54
Learning from Language Description: Low-shot Named Entity Recognition via Decomposed Framework
Wang, Y., Chu, H., Zhang, C., & Gao, J. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.139 · Cited by 53
Logical Fallacy DetectionPreprint
Jin, Z., Lalwani, A., Vaidhya, T., Shen, X., Ding, Y., Zhiheng, L., Sachan, M., Mihalcea, R. F., & Schölkopf, B. (2022) · arXiv (Cornell University) · doi:10.3929/ethz-b-000592493 · Cited by 53
Jin, Z., Lalwani, A., Vaidhya, T., Shen, X., Ding, Y., Lyu, Z., Sachan, M., Mihalcea, R., & Schölkopf, B. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.532 · Cited by 52
Lessons from the Use of Natural Language Inference (NLI) in Requirements Engineering Tasks
Fazelnia, M., Koscinski, V., Herzog, S., & Mirakhorli, M. T. (2024) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re59067.2024.00020 · Cited by 53
“For the probability-based approach, we tested the following values as the classification thresholds: [1, 3, 5, 10, 15, 20, 25, 30, 35, 40].”
MethodZero-shot Node Classification with Decomposed Graph Prototype Network
Wang, Z., Wang, J., Guo, Y., & Gong, Z. (2021) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3447548.3467230 · Cited by 52
“However, existing ZSL methods are mainly limited to computer vision [32] or natural language processing [41].”
MethodText Classification in the LLM Era -- Where do we stand?Preprint
Vajjala, S., & Shimangaud, S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.11830 · Cited by 53
NewsClaims: A New Benchmark for Claim Detection from News with Attribute Knowledge
Reddy, R. G., Chinthakindi, S. C., Wang, Z., Fung, Y. R., Conger, K., ELsayed, A., Palmer, M. S., Nakov, P., Hovy, E. H., Small, K., et al. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.403 · Cited by 51
“ClaimBuster+Zero-shot NLI: Following Yin et al. (2019), we use pre-trained NLI models as zero-shot text classifiers: we pose the claim sentence to be classified as the NLI premise and construct a hypothesis from each candidate topic.”
MethodChandra, R., & Ranjan, M. (2022) · PLoS ONE · doi:10.1371/journal.pone.0273476 · Cited by 50
“Most of the topic modelling research [52, 116,117] involves the bench-marking model results on pre-existing datasets such as the 20 News Groups dataset [118], the Yahoo Answers dataset [119,120], Web Snippets dataset [121], W2E datasets [122].”
BackgroundGovernance of artificial intelligence in Southeast Asia
Keith, A. J. (2024) · Global Policy · doi:10.1111/1758-5899.13458 · Cited by 50
Are We Really Making Much Progress in Text Classification? A Comparative Review
Scherp, A., Diera, A., Karl, F., Lin, B. X., Khera, B., Meuser, T., Singhal, T., & Galke, L. P. A. (2026) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2026.3735248 · Cited by 50
Labels in a haystack: Approaches beyond supervised learning in biomedical applications
Yakimovich, A., Beaugnon, A., Huang, Y., & Özkırımlı, E. (2021) · Patterns · doi:10.1016/j.patter.2021.100383 · Cited by 50
“the data or labels in the high-dimensional ‘‘semantic space’’, measure the similarity of their relationship to those in the training set, and provide proper predictions accordingly.(129,20) The embedding approach has proved its effectiveness in multiple biomedical domains, such as affinity prediction,(58) drug discov-”
BackgroundMoreno-Garcia, C. F., Jayne, C., Elyan, E., & Aceves-Martins, M. (2023) · Decision Analytics Journal · doi:10.1016/j.dajour.2023.100162 · Cited by 35
A Survey of Transformer-based Language Models with Focus on EfficiencyPreprint
Ansar, W., Goswami, S., & Chakrabarti, A. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.16893 · Cited by 49
tasksource: A Large Collection of NLP tasks with a Structured Dataset Preprocessing Framework
Sileo, D. (2024) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2h4pkr685mkq · Cited by 48
“One of the most popular techniques to achieve this is entailment-based classification (Yin et al., 2019).”
MethodPolyglot Prompt: Multilingual Multitask Prompt Training
Fu, J., Ng, S.-K., & Liu, P. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.674 · Cited by 48
Truth-O-Meter: Collaborating with LLM in fighting its hallucinations
Galitsky, B. A. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-29246-0.00004-3 · Cited by 50
Definitions Matter: Guiding GPT for Multi-label Classification
Peskine, Y., Korenčić, D., Grubišić, I., Papotti, P., Troncy, R., & Rosso, P. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.267 · Cited by 47
“Approaches that use class definitions for constructing zero-shot classifiers and prompts have been proposed, both for CNNs and transformers (Zhang et al., 2019b; Yin et al., 2019), and for LLMs (Brown et al., 2020; Gilardi et al., 2023).”
ZeroBERTo: Leveraging Zero-Shot Text Classification by Topic Modeling
Alcoforado, A., Ferraz, T. P., Gerber, R., Bustos, E., Oliveira, A. S., Veloso, B., Siqueira, F. L., & Costa, A. H. R. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-98305-5_12 · Cited by 48
Language Models for Text Classification: Is In-Context Learning Enough?Preprint
Edwards, A., & Camacho-Collados, J. (2024) · ORCA Online Research @Cardiff (Cardiff University) · doi:10.48550/arxiv.2403.17661 · Cited by 47
“A standard approach for supervised text classification is fine-tuning language models such as BERT using an additional classifier head (Radford et al., 2018; Dong et al., 2019; Devlin et al., 2018; Yin et al., 2019; Viswanathan et al., 2023; Mos-bach et al., 2023).”
BackgroundMethodFew-Shot Stance Detection via Target-Aware Prompt Distillation
Jiang, Y., Gao, J., Shen, H., & Cheng, X. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531979 · Cited by 46
“[43] used a prompt such as “the topic of this document is [Z].”
MethodIntroduction to Neural Transfer Learning With Transformers for Social Science Text Analysis
Wankmüller, S. (2022) · Sociological Methods & Research · doi:10.1177/00491241221134527 · Cited by 47
“In the zero-shot classification framework of Yin et al. (2019), a model is presented with the input text (taking the role of the premise) and a hypothesis that asks whether the input text belongs to a particular class.”
InfluentialEvaluating Gender Bias in Natural Language InferencePreprint
Sharma, S., Dey, M., & Sinha, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.05541 · Cited by 46
“Table 1: Statistics of evaluation sets designed for evaluating gender bias Premise: To measure the bias, we select 38 different occupations to include a variety of gender distribution characteristics and occupation types, in correspondence with US Current Population Survey 1 (CPS) 2019 data and prior literature (Zhao…”
MethodHas It All Been Solved? Open NLP Research Questions Not Solved by Large Language Models
Ignat, O., Jin, Z., Abzaliev, A., Biester, L., Castro, S., Deng, N., Gao, X., Gunal, A. E., He, J., Kazemi, A., et al. (2024) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2bw7xfcwyeq3 · Cited by 45
Austin, D. E., Korikov, A., Toroghi, A., & Sanner, S. (2024) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3640457.3688142 · Cited by 44
“Recent work on ConvRec uses language models 2 to facilitate NL dialogue while integrating calls to a recommender module which generates item recommendations based on user-item interaction history [4, 22, 30, 32].”
BackgroundMethodInfluentialZero-Shot Learning with Common Sense Knowledge Graphs
Nayak, N. V., & Bach, S. H. (2022) · Transactions on machine learning research · doi:10.48550/arxiv.2006.10713 · Cited by 44
“Other notable works in zero-shot learning include text classification (Chang et al., 2008; Yin et al., 2019), video action recognition (Gan et al., 2015), machine translation (Johnson et al., 2017), and more (Wang et al., 2019).”
The ACL OCL Corpus: Advancing Open Science in Computational Linguistics
Rohatgi, S., Qin, Y., Aw, B., Unnithan, N., & Kan, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.640 · Cited by 44
Metadata
- Authors
- Yin, W., Hay, J., & Roth, D.
- Affiliations
- University of Pennsylvania (US)
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 3912-3921
- OpenAlex
- W2970200208
- Cited by
- 537
- References
- 30
- Citations by year
- 2026: 66 · 2025: 86 · 2024: 75 · 2023: 141 · 2022: 94 · 2021: 64 · 2020: 11
- FWCI
- 16.17
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Funders
- U.S. Department of Defense
- Defense Advanced Research Projects Agency (HR0011-15-C-0113)
- Advanced Research Projects Agency
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- zero-shot text classification, benchmarking, evaluation metrics, logical consequence, cross-domain classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.

