Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach
Open accessThe DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Abstract
Wenpeng Yin, Jamaal Hay, Dan Roth. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This is evidenced by LLMs impressive zero-shot and few-shot learning ability [18,132], as well as the fact that in-context learning does not require weight update, which mechanically prevents unfavorable fine-tuning-related behavior like catastrophic forgetting.”
From Harnessing the power of llms in practice: a survey on chatGPT and beyond (2024) · cites this paper as [132]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs have been shown to outperform previous zero-shot methods [132].”
From Harnessing the power of llms in practice: a survey on chatGPT and beyond (2024) · cites this paper as [132]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 6×
“Regarding this label-fully-unseen evaluation, apart from our entailment models and three unsupervised baselines “Majority”, “Word2Vec” and “ESA”, we also report the following baseline: Wikipedia-based : We train a binary clas-sifier based on BERT on a dataset collected from Wikipedia.”
MethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 6×
“Regarding this label-fully-unseen evaluation, apart from our entailment models and three unsupervised baselines “Majority”, “Word2Vec” and “ESA”, we also report the following baseline: Wikipedia-based : We train a binary clas-sifier based on BERT on a dataset collected from Wikipedia.”
MethodInfluentialDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 2×
“• Word2Vec 5 (Mikolov et al., 2013): Both the representations of the text and the labels are the addition of word embeddings element-wisely.”
BackgroundGLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222 · cited 11×
“For label-partially-unseen setup in which we intentionally provide annotated data, we first pretrain BERT on the MNLI/FEVER/RTE, then fine-tune on the provided training data.”
BackgroundMethodInfluentialA Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342 · cited 2×
“, 2019), and train it on three mainstream entailment datasets: MNLI (Williams et al., 2018), GLUE RTE (Dagan et al.”
BackgroundMethodCharacter-level Convolutional Networks for Text Classification
Zhang, X., Zhao, J., & LeCun, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.01626 · Cited by 3,205 · cited 1×
“We use the large-scale Yahoo dataset released by Zhang et al. (2015).”
MethodComputing semantic relatedness using Wikipedia-based explicit semantic analysis
Gabrilovich, E., & Markovitch, S. (2007) · Cited by 1,981
The PASCAL Recognising Textual Entailment Challenge
Dagan, I., Glickman, O., & Magnini, B. (2006) · Lecture Notes in Computer Science · doi:10.1007/11736790_9 · Cited by 1,628
(2018) · International Conference on Learning Representations · Cited by 1,621
FEVER: a Large-scale Dataset for Fact Extraction and VERification
Thorne, J., Vlachos, A., Christodoulopoulos, C., & Mittal, A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1074 · Cited by 1,343 · cited 1×
“In this work, we make use of the widely-recognized state of the art entailment technique – BERT (Devlin et al., 2019), and train it on three mainstream entailment datasets: MNLI (Williams et al., 2018), GLUE RTE (Dagan et al., 2005; Wang et al., 2019) and FEVER 3 (Thorne et al., 2018), respectively.”
MethodZero-Shot Relation Extraction via Reading Comprehension
Levy, O., Seo, M., Choi, E., & Zettlemoyer, L. (2017) · Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017) · doi:10.18653/v1/k17-1034 · Cited by 482 · cited 1×
“Levy et al. (2017) and Obamuyide and Vlachos (2018) study zero-shot relation extraction by converting it into a machine comprehension and textual entailment problem respectively.”
BackgroundZero-Shot Relation Extraction via Reading Comprehension
Levy, O., Seo, M., Choi, E., & Zettlemoyer, L. (2017) · arXiv · doi:10.48550/arxiv.1706.04115 · Cited by 482 · cited 1×
“Levy et al. (2017) and Obamuyide and Vlachos (2018) study zero-shot relation extraction by converting it into a machine comprehension and textual entailment problem respectively.”
BackgroundAn Analysis of Annotated Corpora for Emotion Classification in Text
Bostan, L. A. M., & Klinger, R. (2018) · International Conference on Computational Linguistics · Cited by 289
Importance of semantic representation: dataless classification
Chang, M., Ratinov, L., Roth, D., & Srikumar, V. (2008) · National Conference on Artificial Intelligence · Cited by 229
Self-citationZero-shot User Intent Detection via Capsule Neural Networks
Xia, C., Zhang, C., Yan, X., Chang, Y., & Yu, P. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1348 · Cited by 195 · cited 4×
“Xia et al. (2018) study the zero-shot intent detection problem.”
BackgroundInfluentialFew-Shot and Zero-Shot Multi-Label Learning for Structured Label Spaces
Rios, A., & Kavuluru, R. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1352 · Cited by 181 · cited 2×
“For example, Rios and Kavuluru (2018) work on medical data while reporting R@K as metric; Xia et al. (2018) work on SNIPS-NLU intent detection data while only unseen intents are in the label-searching space in evaluation.”
BackgroundIntegrating Semantic Knowledge to Tackle Zero-shot Text Classification
Zhang, J., Lertvittayakumjorn, P., & Guo, Y. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1108 · Cited by 159 · cited 2×
“Zhang et al. (2019) enrich the embedding representations by incorporating class descriptions, class hierarchy, and the word-to-label paths in ConceptNet.”
BackgroundMethodOn Dataless Hierarchical Text Classification
Song, Y., & Roth, D. (2014) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v28i1.8938 · Cited by 152 · cited 1×
“Then this idea was further developed in (Song and Roth, 2014; Chen et al., 2015; Li et al., 2016a,b; Song et al., 2016).”
MethodSelf-citationZero-shot Relation Classification as Textual Entailment
Obamuyide, A., & Vlachos, A. (2018) · Fact Extraction and VERification Workshop (FEVER) · doi:10.18653/v1/w18-5511 · Cited by 116 · cited 1×
“Levy et al. (2017) and Obamuyide and Vlachos (2018) study zero-shot relation extraction by converting it into a machine comprehension and textual entailment problem respectively.”
BackgroundGenerative and Discriminative Text Classification with Recurrent Neural Networks
Yogatama, D., Dyer, C., Wang, L., & Blunsom, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.01898 · Cited by 100 · cited 3×
“Yogatama et al. (2017) build generative LSTM to generate text given the embedded labels.”
BackgroundLi, Y., Zheng, R., Tian, T., Hu, Z., Iyer, R., & Sycara, K. P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.03924 · Cited by 81
Dataless Text Classification with Descriptive LDA
Chen, X., Xia, Y., Jin, P., & Carroll, J. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9506 · Cited by 76 · cited 1×
“Then this idea was further developed in (Song and Roth, 2014; Chen et al., 2015; Li et al., 2016a,b; Song et al., 2016).”
MethodTrain Once, Test Anywhere: Zero-Shot Learning for Text Classification
Pushp, P. K., & Srivastava, M. M. (2017) · arXiv · doi:10.48550/arxiv.1712.05972 · Cited by 74
Zero-Shot Open Entity Typing as Type-Compatible Grounding
Zhou, B., Khashabi, D., Tsai, C.-T., & Roth, D. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1231 · Cited by 72 · cited 2×
“Collecting categorized articles from Wikipedia is popular way of creating training data for text categorization, such as (Zhou et al., 2018).”
BackgroundSelf-citationLi, Y., Zheng, R., Tian, T., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.07956 · Cited by 64
Zero-Shot Sequence Labeling: Transferring Knowledge from Sentences to Tokens
Rei, M., & Søgaard, A. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1027 · Cited by 57 · cited 1×
“Other zero-shot problems studied in NLP involve entity typing (Zhou et al., 2018), sequence labeling (Rei and Søgaard, 2018), etc.”
BackgroundZero-shot Learning of Classifiers from Natural Language Quantification
Srivastava, S., Labutov, I., & Mitchell, T. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1029 · Cited by 45 · cited 1×
“Srivastava et al. (2018) assume that some natural language explanations about new labels are available.”
BackgroundUsing Semantic Similarity for Multi-Label Zero-Shot Classification of Text Documents
Sappadla, P. V., Nam, J., Mencía, E. L., & Fürnkranz, J. (2016) · The European Symposium on Artificial Neural Networks · Cited by 34
Cross-lingual dataless classification for many languages
Song, Y., Upadhyay, S., Peng, H., & Roth, D. (2016) · International Joint Conference on Artificial Intelligence · Cited by 25
Self-citationSituational Awareness for Low Resource Languages: the LORELEI Situation Frame Annotation Task.
Strassel, S. M., Ann Bies, A., & Tracey, J. (2017) · Cited by 22
Integrating Semantic Knowledge to Tackle Zero-shot Text ClassificationPreprint
Zhang, J., Lertvittayakumjorn, P., & Guo, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.12626 · Cited by 17 · cited 2×
“Zhang et al. (2019) enrich the embedding representations by incorporating class descriptions, class hierarchy, and the word-to-label paths in ConceptNet.”
BackgroundMethodLow-resource Cross-lingual Event Type Detection via Distant Supervision with Minimal Effort
Muis, A. O., Otani, N., Vyas, N., Xu, R., Yang, Y., Mitamura, T., & Hovy, E. H. (2018) · Cited by 11
Research works printed but not linked: 2
- [8]8. Chenliang Li, Jian Xing, Aixin Sun, and Zongyang Ma. 2016a. Effective document labeling with very few seed words: A topic model approach. In CIKM, pages 85-94.
- [11]11. Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed repre- sentations of words and phrases and their composi- tionality. In NIPS, pages 3111-3119.
Cited by
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey
Min, B., Ross, H., Sulem, E., Veyseh, A. P. B., Nguyen, T. H., Sainz, O., Agirre, E., Heintz, I., & Roth, D. (2024) · ACM Computing Surveys · doi:10.1145/3605943 · Cited by 1,387
Self-citationExploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Schick, T., & Schütze, H. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.20 · Cited by 1,305
Finetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101
“Our work also falls in the well-established category of zero-shot learning, which has historically been used to refer to classifying instances among a set of unseen categories (Lampert et al., 2009; Romera-Paredes & Torr, 2015; Srivastava et al., 2018; Yin et al., 2019, inter alia).”
BackgroundTop2Vec: Distributed Representations of TopicsPreprint
Angelov, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.09470 · Cited by 944
“The Yahoo Answers dataset [36, 37], contains 1.3 million labelled posts.”
BackgroundInfluential
Metadata
- Authors
- Yin, W., Hay, J., & Roth, D.
- Affiliations
- University of Pennsylvania (US)
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 3912-3921
- OpenAlex
- W2970200208
- Cited by
- 537
- References
- 30
- Citations by year
- 2026: 66 · 2025: 86 · 2024: 75 · 2023: 141 · 2022: 94 · 2021: 64 · 2020: 11
- FWCI
- 16.17
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Funders
- U.S. Department of Defense
- Defense Advanced Research Projects Agency (HR0011-15-C-0113)
- Advanced Research Projects Agency
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- zero-shot text classification, benchmarking, evaluation metrics, logical consequence, cross-domain classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.

