CoChat
Sign Up
PreprintCited by 4Domain Adaptation and Few-Shot LearningTopic ModelingMultimodal Machine Learning Applications

The DOI matches its DataCite record.

SemSup: Semantic Supervision for Simple and Scalable Zero-shot Generalization

Open access
Hanjie, Austin W., Deshpande, Ameet, Narasimhan, Karthik·2022·arXiv
DOI 10.48550/arxiv.2202.13100arXiv 2202.13100
Open this paper in CoChat — free accountOpen PDF

Abstract

Zero-shot learning is the problem of predicting instances over classes not seen during training. One approach to zero-shot learning is providing auxiliary class information to the model. Prior work along this vein have largely used expensive per-instance annotation or singular class-level descriptions, but per-instance descriptions are hard to scale and single class descriptions may not be rich enough. Furthermore, these works have used natural-language descriptions exclusively, simple bi-encoders models, and modality or task-specific methods. These approaches have several limitations: text supervision may not always be available or optimal and bi-encoders may only learn coarse relations between inputs and class descriptions. In this work, we present SemSup, a novel approach that uses (1) a scalable multiple description sampling method which improves performance over single descriptions, (2) alternative description formats such as JSON that are easy to generate and outperform text on certain settings, and (3) hybrid lexical-semantic similarity to leverage fine-grained information in class descriptions. We demonstrate the effectiveness of SemSup across four datasets, two modalities, and three generalization settings. For example, across text and image datasets, SemSup increases unseen class generalization accuracy by 15 points on average compared to the closest baseline.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2202.13100

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Llama-2-7B's toxicity generation decreased from 22% to near 0 percentage on the ToxiGen benchmark with RLHF.

    “For instance, Llama-2-7B's performance on TruthfulQA [100] improved from 33% to 57% with RLHF and toxicity generation decreased from 22% to near 0 percentage on ToxiGen benchmark [63].”

    From RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs (2026) · cites this paper as [63]

    The Results (p. 5) report SemSup classification performance, not Llama-2-7B toxicity generation. Searches for ToxiGen, RLHF, and the claimed 22%-to-near-zero change found no passage in our copy that addresses the claim or identifies this paper as the ToxiGen benchmark. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 74

  1. Acquaviva, S., Pu, Y., Kryven, M., Wong, C., Ecanow, G. E., Nye, M., Sechopoulos, T., Tessler, M. H., and Tenenbaum, J. B. Communicating natural programs to humans and machines. arXiv preprint arXiv:2106.07824, 2021. · arXiv
  2. Akata, Z., Perronnin, F., Harchaoui, Z., and Schmid, C. Label-embedding for image classification. IEEE transactions on pattern analysis and machine intelligence, 38(7):1425-1438, 2015.
  3. Andreas, J., Klein, D., and Levine, S. Learning with latent language. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 2166-2179, 2018.
  4. Bai, B., Weston, J., Grangier, D., Collobert, R., Sadamasa, K., Qi, Y., Chapelle, O., and Weinberger, K. Supervised semantic indexing. In Proceedings of the 18th ACM conference on Information and knowledge management, pp. 187-196, 2009.
  5. Branavan, S., Zettlemoyer, L., and Barzilay, R. Reading between the lines: Learning to map high-level instructions to commands. In Proceedings of the 48th annual meeting of the association for computational linguistics, pp. 1268-1277, 2010.
  6. Branavan, S., Silver, D., and Barzilay, R. Learning to win by reading manuals in a monte-carlo framework. Journal of Artificial Intelligence Research, 43:661-704, 2012.
  7. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https://proceedings. neurips.cc/paper/2020/hash/
  8. Bujwid, S. and Sullivan, J. Large-scale zero-shot image classification from rich and diverse textual descriptions. arXiv preprint arXiv:2103.09669, 2021. · arXiv
  9. Chang, M.-W., Ratinov, L.-A., Roth, D., and Srikumar, V. Importance of semantic representation: Dataless classification. In Aaai, volume 2, pp. 830-835, 2008.
  10. Clarke, J., Goldwasser, D., Chang, M.-W., and Roth, D. Driving semantic parsing from the world's response. In Proceedings of the fourteenth conference on computational natural language learning, pp. 18-27, 2010.
  11. Dauphin, Y. N., T¨ ur, G., Hakkani-T¨ ur, D., and Heck, L. P. Zero-shot learning and clustering for semantic utterance classification. In ICLR (Poster), 2014.
  12. DeJong, G. and Mooney, R. Explanation-based learning: An alternative view. Machine learning, 1(2):145-176, 1986.
  13. Demirel, B., Gokberk Cinbis, R., and Ikizler-Cinbis, N. Attributes2classname: A discriminative model for attributebased unsupervised zero-shot learning. In Proceedings of the IEEE international conference on computer vision, pp. 1232-1241, 2017.
  14. Deng, Y., Kanervisto, A., Ling, J., and Rush, A. M. Imageto-markup generation with coarse-to-fine attention. In International Conference on Machine Learning, pp. 980989. PMLR, 2017.
  15. Denil, M., Colmenarejo, S. G., Cabi, S., Saxton, D., and de Freitas, N. Programmable agents. arXiv preprint arXiv:1706.06383, 2017. · arXiv
  16. Desai, A., Gulwani, S., Hingorani, V., Jain, N., Karkare, A., Marron, M., and Roy, S. Program synthesis using natural language. In Proceedings of the 38th International Conference on Software Engineering, pp. 345-356, 2016.
  17. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. · arXiv
  18. Fidler, S. et al. Teaching machines to describe images with natural language feedback. Advances in Neural Information Processing Systems, 30, 2017.
  19. Frome, A., Corrado, G. S., Shlens, J., Bengio, S., Dean, J., Ranzato, M., and Mikolov, T. Devise: a deep visual-semantic embedding model. In Proceedings of the 26th International Conference on Neural Information Processing Systems-Volume 2, pp. 2121-2129, 2013.
  20. Gao, L., Dai, Z., and Callan, J. Coil: Revisit exact lexical match in information retrieval with contextualized inverted list. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3030-3042, 2021a.
  21. Gao, T., Fisch, A., and Chen, D. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 3816-3830, 2021b.
  22. Hancock, B., Bringmann, M., Varma, P., Liang, P., Wang, S., and R´ e, C. Training classifiers with natural language explanations. In Proceedings of the conference. Association for Computational Linguistics. Meeting, volume 2018, pp. 1884. NIH Public Access, 2018.
  23. Hanjie, A. W., Zhong, V. Y., and Narasimhan, K. Grounding language to entities and dynamics for generalization in reinforcement learning. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 4051-4062. PMLR, 1824 Jul 2021. URL https://proceedings.mlr. press/v139/hanjie21a.html .
  24. Humeau, S., Shuster, K., Lachaux, M.-A., and Weston, J. Poly-encoders: Architectures and pre-training strategies for fast and accurate multi-sentence scoring. In International Conference on Learning Representations, 2019.
  25. Khattab, O. and Zaharia, M. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACMSIGIR conference on research and development in Information Retrieval, pp. 39-48, 2020.
  26. Koh, P. W., Nguyen, T., Tang, Y. S., Mussmann, S., Pierson, E., Kim, B., and Liang, P. Concept bottleneck models. In International Conference on Machine Learning, pp. 5338-5348. PMLR, 2020.
  27. Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., and Houlsby, N. Big transfer (bit): General visual representation learning. In European conference on computer vision, pp. 491-507. Springer, 2020.
  28. Lampert, C. H., Nickisch, H., and Harmeling, S. Learning to detect unseen object classes by between-class attribute transfer. In 2009 IEEE conference on computer vision and pattern recognition, pp. 951-958. IEEE, 2009.
  29. Lang, K. Newsweeder: Learning to filter netnews. In Machine Learning Proceedings 1995, pp. 331-339. Elsevier, 1995.
  30. Larochelle, H., Erhan, D., and Bengio, Y. Zero-data learning of new tasks. In AAAI, volume 1, pp. 3, 2008.
  31. Lei Ba, J., Swersky, K., Fidler, S., et al. Predicting deep zero-shot convolutional neural networks using textual descriptions. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4247-4255, 2015.
  32. Lewis, D. D., Yang, Y., Russell-Rose, T., and Li, F. Rcv1: A new benchmark collection for text categorization research. Journal of machine learning research, 5(Apr):361-397, 2004.
  33. Liang, W., Zou, J., and Yu, Z. Alice: Active learning with contrastive natural language explanations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4380-4391, 2020.
  34. Lison, P., Barnes, J., Hubin, A., and Touileb, S. Named entity recognition without labelled data: A weak supervision approach. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 15181533, 2020.
  35. Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., and Neubig, G. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586, 2021. · arXiv
  36. Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. · arXiv
  37. Mayhew, S., Chaturvedi, S., Tsai, C.-T., and Roth, D. Named entity recognition with partially annotated training data. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pp. 645-655, 2019.
  38. Miller, G. A. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39-41, 1995.
  39. Mitchell, T. M., Keller, R. M., and Kedar-Cabelli, S. T. Explanation-based generalization: A unifying view. Machine learning, 1(1):47-80, 1986.
  40. Mittal, A., Dahiya, K., Agrawal, S., Saini, D., Agarwal, S., Kar, P., and Varma, M. Decaf: Deep extreme classification with label features. In Proceedings of the 14th ACMInternational Conference on Web Search and Data Mining, pp. 49-57, 2021.
  41. Mu, J., Liang, P., and Goodman, N. Shaping visual representations with language for few-shot classification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 48234830, 2020.
  42. Murty, S., Koh, P. W., and Liang, P. Expbert: Representation engineering with natural language explanations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 21062113, 2020.
  43. Nam, J., Menc´ ıa, E. L., and F¨ urnkranz, J. All-in text: Learning document, label, and word representations jointly. In Thirtieth AAAI Conference on Artificial Intelligence, 2016.
  44. Narasimhan, K., Barzilay, R., and Jaakkola, T. Grounding language for transfer in deep reinforcement learning. Journal of Artificial Intelligence Research, 63:849-874, 2018.
  45. Palatucci, M., Pomerleau, D., Hinton, G. E., and Mitchell, T. M. Zero-shot learning with semantic output codes. In NIPS, 2009.
  46. Pappas, N. and Henderson, J. Gile: A generalized inputlabel embedding for text classification. Transactions of the Association for Computational Linguistics, 7:139155, 2019.
  47. Pennington, J., Socher, R., and Manning, C. D. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
  48. Qiao, R., Liu, L., Shen, C., and Van Den Hengel, A. Less is more: zero-shot learning from online textual documents with noise suppression. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2249-2257, 2016.
  49. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 87488763. PMLR, 2021. URL http://proceedings. mlr.press/v139/radford21a.html .
  50. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21: 1-67, 2020.
  51. Ratner, A., Bach, S. H., Ehrenberg, H., Fries, J., Wu, S., and R´ e, C. Snorkel: Rapid training data creation with weak supervision. In Proceedings of the VLDB Endowment. International Conference on Very Large Data Bases, volume 11, pp. 269. NIH Public Access, 2017.
  52. Reed, S., Akata, Z., Lee, H., and Schiele, B. Learning deep representations of fine-grained visual descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 49-58, 2016.
  53. Safranchik, E., Luo, S., and Bach, S. Weakly supervised sequence tagging from noisy rules. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5570-5578, 2020.
  54. Sanh, V., Webson, A., Raffel, C., Bach, S., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Le Scao, T., Raja, A., et al. Multitask prompted training enables zero-shot task generalization. In The Tenth International Conference on Learning Representations, 2022.
  55. Schick, T. and Sch¨ utze, H. It's not just size that matters: Small language models are also few-shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 23392352, 2021.
  56. Sharma, P., Torralba, A., and Andreas, J. Skill induction and planning with latent language. arXiv preprint arXiv:2110.01517, 2021. · arXiv
  57. Snell, J., Swersky, K., and Zemel, R. Prototypical networks for few-shot learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 4080-4090, 2017.
  58. Socher, R., Ganjoo, M., Manning, C. D., and Ng, A. Zeroshot learning through cross-modal transfer. In Advances in Neural Information Processing Systems, pp. 935-943, 2013a.
  59. Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A. Y., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631-1642, 2013b.
  60. Srivastava, S., Labutov, I., and Mitchell, T. Joint concept learning and semantic parsing from natural language explanations. In Proceedings of the 2017 conference on empirical methods in natural language processing, pp. 1527-1536, 2017.
  61. Srivastava, S., Labutov, I., and Mitchell, T. Zero-shot learning of classifiers from natural language quantification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 306-316, 2018.
  62. Torralba, A., Fergus, R., and Freeman, W. T. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE transactions on pattern analysis and machine intelligence, 30(11):1958-1970, 2008.
  63. Turc, I., Chang, M.-W., Lee, K., and Toutanova, K. Well-read students learn better: On the importance of pre-training compact models. arXiv preprint arXiv:1908.08962v2, 2019. · arXiv
  64. Varma, P. and R´ e, C. Snuba: Automating weak supervision to label training data. In Proceedings of the VLDB Endowment. International Conference on Very Large Data Bases, volume 12, pp. 223. NIH Public Access, 2018.
  65. Vinyals, O., Blundell, C., Lillicrap, T., Wierstra, D., et al. Matching networks for one shot learning. Advances in neural information processing systems, 29:3630-3638, 2016.
  66. Wang, G., Li, C., Wang, W., Zhang, Y., Shen, D., Zhang, X., Henao, R., and Carin, L. Joint embedding of words and labels for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2321-2331, 2018.
  67. Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned language models are zero-shot learners. CoRR, abs/2109.01652, 2021. URL https://arxiv.org/ abs/2109.01652 . · arXiv
  68. Williams, A., Nangia, N., and Bowman, S. A broadcoverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1112-1122. Association for Computational Linguistics, 2018. URL http://aclweb.org/anthology/ N18-1101 .
  69. Wong, C., Ellis, K. M., Tenenbaum, J., and Andreas, J. Leveraging language to learn program abstractions and search heuristics. In International Conference on Machine Learning, pp. 11193-11204. PMLR, 2021.
  70. Xian, Y., Lampert, C. H., Schiele, B., and Akata, Z. Zero-shot learning-a comprehensive evaluation of the good, the bad and the ugly. IEEE transactions on pattern analysis and machine intelligence, 41(9):22512265, 2018.
  71. Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., and Raffel, C. mt5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 483498, 2021.
  72. Zeng, A., Wong, A., Welker, S., Choromanski, K., Tombari, F., Purohit, A., Ryoo, M., Sindhwani, V., Lee, J., Vanhoucke, V., et al. Socratic models: Composing zeroshot multimodal reasoning with language. arXiv preprint arXiv:2204.00598, 2022. · arXiv
  73. Zhang, H., Xiao, L., Chen, W., Wang, Y., and Jin, Y. Multi-task label embedding for text classification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 45454553, 2018.
  74. Zhong, V., Rockt¨ aschel, T., and Grefenstette, E. Rtfm: Generalising to new environment dynamics via reading. In International Conference on Learning Representations, 2019.

Cited by

Sorted by citations

Metadata

Authors
Hanjie, A. W., Deshpande, A., & Narasimhan, K.
Affiliations
Princeton University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2022
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2202.13100
arXiv
2202.13100
OpenAlex
W4226064231
Cited by
4
References
0
Citations by year
2024: 1 · 2023: 1 · 2022: 2
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
zero-shot learning, semantic supervision, course descriptions, unseen class generalization, cross-modal generalization, image-text retrieval

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review