UNIFIEDQA: Crossing Format Boundaries with a Single QA System
Open accessAbstract
Question answering (QA) tasks have been posed using a variety of formats, such as extractive span selection, multiple choice, etc.This has led to format-specialized models, and even to an implicit division in the QA community.We argue that such boundaries are artificial and perhaps unnecessary, given the reasoning abilities we seek to teach are not governed by the format.As evidence, we use the latest advances in language modeling to build a single pre-trained QA model, UNIFIEDQA, that performs well across 20 QA datasets spanning 4 diverse formats.UNIFIEDQA performs on par with 8 different models that were trained on individual datasets themselves.Even when faced with 12 unseen datasets of observed formats, UNIFIEDQA performs surprisingly well, showing strong generalization from its out-offormat training data.Finally, fine-tuning this pre-trained QA model into specialized models results in a new state of the art on 10 factoid and commonsense QA datasets, establishing UNIFIEDQA as a strong starting point for building QA systems. 1 1 https://github.com/allenai/unifiedqaExtractive [SQuAD] Question: At what speed did the turbine operate?Context: (Nikola_Tesla) On his 50th birthday in 1906, Tesla demonstrated his 200 horsepower (150 kilowatts) 16,000 rpm bladeless turbine.... Gold answer: 16,000 rpm Multiple-Choice [ARC-challenge] Question: What does photosynthesis produce that helps plants grow?Candidate Answers: (A) water (B) oxygen (C) protein (D) sugar Gold answer: sugar Yes/No [BoolQ]Question: Was America the first country to have a president?Context: (President) The first usage of the word president to denote the highest official in a government was during the Commonwealth of England ... Gold answer: no Abstractive [NarrativeQA]Question: What does a drink from narcissus's spring cause the drinker to do? Context: Mercury has awakened Echo, who weeps for Narcissus, and states that a drink from Narcissus's spring causes the drinkers to "Grow dotingly enamored of themselves."... Gold answer: fall in love with themselves
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
UnifiedQA models generally do better on truthfulness than the three GPT families but are the least informative.
“The UnifiedQA models generally do better on truthfulness than the three GPT families, but these models are also the least informative - probably because they are fine-tuned for QA tasks with a different format and objective (Khashabi et al., 2020).”
From TruthfulQA: Measuring How Models Mimic Human Falsehoods (2022) · cites this paper as Khashabi et al., 2020
Section 3.2 identifies UnifiedQA as a model trained on question-answering datasets spanning four formats, supporting the cited explanation of its training. The paper does not compare UnifiedQA with GPT families on truthfulness or informativeness, or establish that its training causes either result. No page number is available in our copy. (LLM, conf=0.83)
Experimental — accuracy not yet measuredLonger phrasal or sentential answers are often used in language generation tasks and in multiple-choice question answering.
“Longer phrasal or sentential answers are often used in language generation tasks [105] and in other tasks such as multiple-choice question answering (where the scores of multiple phrases are compared against each-other; Khashabi et al. [55]).”
From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [55]
Section 3.1 describes multiple-choice QA as choosing among candidate answers, and Table 2 reports multi-token average answer lengths for several multiple-choice datasets. The cited paper therefore supports the use of phrasal answers in multiple-choice QA, but the passages do not establish the parenthetical claim that scores of multiple phrases are compared. The language-generation part is attached to a different citation. (LLM, conf=0.83)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · see citing works
Caruana, R. (1997) · Machine Learning · doi:10.1023/a:1007379606734 · Cited by 6,636 · see citing works
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · see citing works
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308 · see citing works
Dense passage retrieval for open-domain question answering
Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.550 · Cited by 3,575 · see citing works
Self-citationCan a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
Mihaylov, T., Clark, P., Khot, T., & Sabharwal, A. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1260 · Cited by 2,991 · see citing works
Self-citationCommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge
Talmor, A., Herzig, J., Lourie, N., & Berant, J. (2019) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n19-1421 · Cited by 2,805 · see citing works
Know What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240 · see citing works
Natural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069 · see citing works
RACE: Large-scale ReAding Comprehension Dataset From Examinations
Lai, G., Xie, Q., Liu, H., Yang, Y., & Hovy, E. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1082 · Cited by 1,044 · see citing works
Levesque, H. J., Davis, E., & Morgenstern, L. (2011) · Cited by 879 · see citing works
MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text
Richardson, M., Burges, C., & Renshaw, E. (2013) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1020 · Cited by 778 · see citing works
NewsQA: A Machine Comprehension Dataset
Trischler, A., Wang, T., Yuan, X., Harris, J., Sordoni, A., Bachman, P., & Suleman, K. (2017) · Proceedings of the 2nd Workshop on Representation Learning for NLP · doi:10.18653/v1/w17-2623 · Cited by 706 · see citing works
How Much Knowledge Can You Pack into the Parameters of a Language Model?
Roberts, A., Raffel, C., & Shazeer, N. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.437 · Cited by 638 · see citing works
The NarrativeQA Reading Comprehension Challenge
Kočiský, T., Schwarz, J., Blunsom, P., Dyer, C., Hermann, K. M., Melis, G., & Grefenstette, E. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00023 · Cited by 596 · see citing works
Social IQa: Commonsense Reasoning about Social Interactions
Sap, M., Rashkin, H., Chen, D., LeBras, R., & Choi, Y. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1454 · Cited by 588 · see citing works
WinoGrande: An Adversarial Winograd Schema Challenge at Scale
Sakaguchi, K., Le Bras, R., Bhagavatula, C., & Choi, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6399 · Cited by 579 · see citing works
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., & Tafjord, O. (2018) · arXiv · doi:10.48550/arxiv.1803.05457 · Cited by 462 · see citing works
Self-citationThe Natural Language Decathlon: Multitask Learning as Question Answering
McCann, B., Keskar, N. S., Xiong, C., & Socher, R. (2018) · arXiv · doi:10.48550/arxiv.1806.08730 · Cited by 448 · see citing works
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., & Gardner, M. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1246 · Cited by 393 · cited 5× · see citing works
“The datasets used from this format are: NarrativeQA/NarQA (Kociský et al., 2018), the open-domain version of Natu-ralQuestions/NatQA (Kwiatkowski et al., 2019), and DROP (Dua et al., 2019b).”
BackgroundMethodInfluentialBoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., & Toutanova, K. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1300 · Cited by 391 · see citing works
Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences
Khashabi, D., Chaturvedi, S., Roth, M., Upadhyay, S., & Roth, D. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-1023 · Cited by 374 · see citing works
Self-citationFreeLB: Enhanced Adversarial Training for Natural Language UnderstandingPreprint
Zhu, C., Cheng, Y., Gan, Z., Sun, S., Goldstein, T., & Liu, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.11764 · Cited by 352 · see citing works
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
Arivazhagan, N., Bapna, A., Firat, O., Lepikhin, D., Johnson, M., Krikun, M., Chen, M. X., Cao, Y., Foster, G., Cherry, C., et al. (2019) · arXiv · doi:10.48550/arxiv.1907.05019 · Cited by 279 · see citing works
MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension
Fisch, A., Talmor, A., Jia, R., Seo, M., Choi, E., & Chen, D. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5801 · Cited by 220 · see citing works
BAM! Born-Again Multi-Task Networks for Natural Language Understanding
Clark, K., Luong, M.-T., Khandelwal, U., Manning, C. D., & Le, Q. V. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1595 · Cited by 219 · see citing works
AmbigQA: Answering Ambiguous Open-domain Questions
Min, S., Michael, J., Hajishirzi, H., & Zettlemoyer, L. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.466 · Cited by 219 · see citing works
Self-citationRetrospective Reader for Machine Reading Comprehension
Zhang, Z., Yang, J., & Zhao, H. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17705 · Cited by 200 · see citing works
MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension
Talmor, A., & Berant, J. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1485 · Cited by 157 · see citing works
Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning
Dasigi, P., Liu, N. F., Marasović, A., Smith, N. A., & Gardner, M. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1606 · Cited by 153 · see citing works
Combining Retrieval, Statistics, and Inference to Answer Elementary Science Questions
Clark, P. E., Etzioni, O., Khot, T., Sabharwal, A., Tafjord, O., Turney, P. D., & Khashabi, D. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10325 · Cited by 139 · see citing works
Self-citationA Discrete Hard EM Approach for Weakly Supervised Question Answering
Min, S., Chen, D., Hajishirzi, H., & Zettlemoyer, L. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1284 · Cited by 137 · see citing works
Self-citationAnswering while Summarizing: Multi-task Learning for Multi-hop QA with Evidence Extraction
Nishida, K., Nishida, K., Nagata, M., Otsuka, A., Saito, I., Asano, H., & Tomita, J. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1225 · Cited by 97 · see citing works
An information-theoretic approach to automatic evaluation of summaries
Lin, C.-Y., Cao, G., Gao, J., & Nie, J. (2006) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/1220835.1220894 · Cited by 94 · see citing works
Reasoning Over Paragraph Effects in Situations
Lin, K., Tafjord, O., Clark, P. E., & Gardner, M. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5808 · Cited by 92 · see citing works
Self-citationFrom 'F' to 'A' on the N.Y. Regents Science Exams: An Overview of the Aristo ProjectPreprint
Clark, P. E., Etzioni, O., Khashabi, D., Khot, T., Mishra, B. D., Richardson, K., Sabharwal, A., Schoenick, C., Tafjord, O., Tandon, N., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.01958 · Cited by 92 · see citing works
Self-citationSolving Hard Coreference Problems
Peng, H., Khashabi, D., & Roth, D. (2015) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/v1/n15-1082 · Cited by 84 · see citing works
Self-citationEvaluating Models' Local Decision Boundaries via Contrast SetsPreprint
Gardner, M., Artzi, Y., Basmova, V., Berant, J., Bogin, B., Chen, S., Dasigi, P., Dua, D., Elazar, Y., Gottumukkala, A., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.02709 · Cited by 83 · see citing works
Self-citationEasy Questions First? A Case Study on Curriculum Learning for Question Answering
Sachan, M., & Xing, E. P. (2016) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.18653/v1/p16-1043 · Cited by 76 · see citing works
A Simple and Effective Model for Answering Multi-span Questions
Segal, E., Efrat, A., Shoham, M., Globerson, A., & Berant, J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.248 · Cited by 76 · see citing works
Evaluating NLP Models via Contrast SetsPreprint
Gardner, M., Artzi, Y., Basmova, V., Berant, J., Bogin, B., Chen, S., Dasigi, P., Dua, D., Elazar, Y., Gottumukkala, A., et al. (2020) · arXiv (Cornell University) · Cited by 65 · see citing works
Self-citationMore Bang for Your Buck: Natural Perturbation for Robust Question Answering
Khashabi, D., Khot, T., & Sabharwal, A. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.12 · Cited by 60 · see citing works
Self-citationPIQA: Reasoning about Physical Commonsense in Natural Language
Bisk, Y., Zellers, R., Le bras, R., Gao, J., & Choi, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6239 · Cited by 37 · see citing works
How Additional Knowledge can Improve Natural Language Commonsense Question Answering?Preprint
Mitra, A., Banerjee, P., Pal, K. K., Mishra, S. R., & Baral, C. R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.08855 · Cited by 30 · see citing works
Retrospective Reader for Machine Reading ComprehensionPreprint
Zhang, Z., Yang, J., & Zhao, H. Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.09694 · Cited by 22 · see citing works
Unifying Question Answering and Text Classification via Span ExtractionPreprint
Keskar, N., McCann, B., Xiong, C., & Socher, R. (2019) · arXiv.org · doi:10.48550/arxiv.1904.09286 · Cited by 21 · see citing works
QASC: A Dataset for Question Answering via Sentence Composition
Khot, T., Clark, P., Guerquin, M., Jansen, P., & Sabharwal, A. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6319 · Cited by 20 · see citing works
Self-citationKnowledge Fusion and Semantic Knowledge Ranking for Open Domain Question AnsweringPreprint
Banerjee, P., & Baral, C. R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.03101 · Cited by 20 · see citing works
Comprehensive Multi-Dataset Evaluation of Reading Comprehension
Dua, D., Gottumukkala, A., Talmor, A., Gardner, M., & Singh, S. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5820 · Cited by 7 · see citing works
Natural Perturbation for Robust Question AnsweringPreprint
Khashabi, D., Khot, T., & Sabharwal, A. (2020) · arXiv (Cornell University) · Cited by 5 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 3
- Dheeru Dua, Ananth Gottumukkala, Alon Talmor, Matt Gardner, and Sameer Singh. 2019a. Compre- hensive multi-dataset evaluation of reading compre- hension. In 2nd Workshop on Machine Reading for Question Answering.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019b. Lan- guage models are unsupervised multitask learners. OpenAI Blog, 1(8):9.
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. JMLR.
Cited by
Sorted by citations
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · see citing works
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer
Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., & Raffel, C. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.41 · Cited by 1,629 · see citing works
Finetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101 · see citing works
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022) · arXiv · doi:10.48550/arxiv.2202.12837 · Cited by 752 · see citing works
Self-citationTruthfulQA: Measuring How Models Mimic Human Falsehoods
Lin, S., Hilton, J., & Evans, O. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.229 · Cited by 904 · see citing works
“UnifiedQA (Khashabi et al., 2020) is a T5 model (Raffel et al., 2019) fine-tuned on diverse QA tasks.”
BackgroundMethodInfluentialRethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.759 · Cited by 755 · see citing works
Self-citationMultitask Prompted Training Enables Zero-Shot Task Generalization
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.08207 · Cited by 612 · see citing works
mT5: A massively multilingual pre-trained text-to-text transformerPreprint
Xue, L., Constant, N., Roberts, A., Kale, M., Al‐Rfou, R., Siddhant, A., Barua, A., & Raffel, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11934 · Cited by 852 · see citing works
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569 · see citing works
Self-citationQA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering
Yasunaga, M., Ren, H., Bosselut, A., Liang, P., & Leskovec, J. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.45 · Cited by 620 · see citing works
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · see citing works
(2021) · International Conference on Learning Representations · Cited by 435 · see citing works
PPT: Pre-trained Prompt Tuning for Few-shot Learning
Gu, Y., Han, X., Liu, Z., & Huang, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.576 · Cited by 392 · see citing works
WinoGrande: an adversarial winograd schema challenge at scale
Sakaguchi, K., Bras, R. L., Bhagavatula, C., & Choi, Y. (2021) · Communications of the ACM · doi:10.1145/3474381 · Cited by 356 · see citing works
LinkBERT: Pretraining Language Models with Document Links
Yasunaga, M., Leskovec, J., & Liang, P. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.551 · Cited by 374 · see citing works
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus
Dodge, J., Sap, M., Marasović, A., Agnew, W., Ilharco, G., Groeneveld, D., Mitchell, M., & Gardner, M. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.98 · Cited by 306 · see citing works
Measuring Mathematical Problem Solving With the MATH Dataset
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., & Steinhardt, J. (2021) · arXiv · doi:10.48550/arxiv.2103.03874 · Cited by 306 · see citing works
Measuring Massive Multitask Language Understanding
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020) · arXiv · doi:10.48550/arxiv.2009.03300 · Cited by 291 · see citing works
♫ MuSiQue: Multihop Questions via Single-hop Question Composition
Trivedi, H., Balasubramanian, N., Khot, T., & Sabharwal, A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00475 · Cited by 283 · see citing works
Self-citationCross-Task Generalization via Natural Language Crowdsourcing Instructions
Mishra, S., Khashabi, D., Baral, C., & Hajishirzi, H. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.244 · Cited by 269 · see citing works
Self-citationSuper-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
Wang, Y., Mishra, S., Alipoormolabashi, P., Kordi, Y., Mirzaei, A., Naik, A., Ashok, A., Dhanasekaran, A. S., Arunkumar, A., Stap, D., et al. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.340 · Cited by 247 · see citing works
(2023) · doi:10.18653/v1/2023.emnlp-main.574 · Cited by 249 · see citing works
Xie, T., Wu, C. H., Shi, P., Zhong, R., Scholak, T., Yasunaga, M., Wu, C.-S., Zhong, M., Yin, P., Wang, S. I., et al. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.39 · Cited by 271 · see citing works
Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question Answering
Feng, Y., Chen, X., Lin, B. Y., Wang, P., Yan, J., & Ren, X. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.99 · Cited by 231 · see citing works
Generated Knowledge Prompting for Commonsense Reasoning
Liu, J., Liu, A., Lu, X., Welleck, S., West, P., Bras, R. J. L., Choi, Y., & Hajishirzi, H. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.225 · Cited by 206 · see citing works
Self-citationKRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQA
Marino, K., Chen, X., Parikh, D., Gupta, A., & Rohrbach, M. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01389 · Cited by 204 · see citing works
Parameter-efficient fine-tuning in large language models: a survey of methodologies
Wang, L., Chen, S., Jiang, L., Pan, S., Cai, R., Yang, S., & Yang, F. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11236-4 · Cited by 186 · see citing works
Muppet: Massive Multi-task Representations with Pre-Finetuning
Aghajanyan, A., Gupta, A., Shrivastava, A., Chen, X., Zettlemoyer, L., & Gupta, S. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.468 · Cited by 183 · see citing works
Jiang, Z., Araki, J., Ding, H., & Neubig, G. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00407 · Cited by 177 · see citing works
BBQ: A hand-built bias benchmark for question answering
Parrish, A., Chen, A., Nangia, N., Padmakumar, V., Phang, J., Thompson, J., Htut, P. M., & Bowman, S. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.165 · Cited by 186 · see citing works
MetaICL: Learning to Learn In Context
Min, S., Lewis, M., Zettlemoyer, L., & Hajishirzi, H. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.201 · Cited by 169 · see citing works
Self-citationHuman Language Understanding & Reasoning
Manning, C. D. (2022) · Daedalus · doi:10.1162/daed_a_01905 · Cited by 173 · see citing works
QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension
Rogers, A., Gardner, M., & Augenstein, I. (2022) · ACM Computing Surveys · doi:10.1145/3560260 · Cited by 177 · see citing works
Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformer
Powalski, R., Borchmann, Ł., Jurkiewicz, D., Dwojak, T., Pietruszka, M., & Pałka, G. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86331-9_47 · Cited by 163 · see citing works
Guo, J., Li, J., Li, D., Tiong, A. M. H., Li, B., Tao, D., & Hoi, S. C. H. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01046 · Cited by 164 · see citing works
Text-to-Text Pre-Training for Data-to-Text Tasks
Kale, M., & Rastogi, A. (2020) · International Conference on Natural Language Generation · doi:10.18653/v1/2020.inlg-1.14 · Cited by 145 · see citing works
TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering
Hu, Y., Liu, B., Kasai, J., Wang, Y., Mari Ostendorf, E., Krishna, R., & Smith, N. A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01866 · Cited by 151 · see citing works
A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers
Dasigi, P., Lo, K., Beltagy, I., Cohan, A., Smith, N. A., & Gardner, M. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.365 · Cited by 138 · see citing works
Pretrained Transformers for Text Ranking: BERT and Beyond
Lin, J., Nogueira, R. F., & Yates, A. (2021) · Synthesis lectures on human language technologies · doi:10.2200/s01123ed1v01y202108hlt053 · Cited by 273 · see citing works
No Need to Lift a Finger Anymore? Assessing the Quality of Code Generation by ChatGPT
Liu, Z., Tang, Y., Luo, X., Zhou, Y., & Zhang, L. F. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3392499 · Cited by 128 · see citing works
NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks
Mishra, S. R., Mitra, A., Varshney, N., Sachdeva, B., Clark, P. A., Baral, C. R., & Kalyan, A. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.246 · Cited by 122 · see citing works
Self-citationSurface Form Competition: Why the Highest Probability Answer Isn’t Always Right
Holtzman, A., West, P., Shwartz, V., Choi, Y., & Zettlemoyer, L. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.564 · Cited by 127 · see citing works
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
Fabbri, A., Wu, C.-S., Liu, W., & Xiong, C. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.187 · Cited by 119 · see citing works
End-to-end generation of multiple-choice questions using text-to-text transfer transformer models
Rodriguez-Torrealba, R., Garcia-Lopez, E., & Garcia-Cabot, A. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118258 · Cited by 118 · see citing works
ZeroGen: Efficient Zero-shot Learning via Dataset Generation
Ye, J., Gao, J., Li, Q., Xu, H., Feng, J., Wu, Z., Yu, C., & Kong, L. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.801 · Cited by 143 · see citing works
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP
Ye, Q., Lin, B. Y., & Ren, X. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.572 · Cited by 120 · see citing works
Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections
Zhong, R., Lee, K., Zhang, Z., & Klein, D. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.244 · Cited by 118 · see citing works
Pretrained Transformers for Text Ranking: BERT and BeyondPreprint
Lin, J., Nogueira, R. F., & Yates, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.06467 · Cited by 101 · see citing works
ChatGPT vs SBST: A Comparative Assessment of Unit Test Suite Generation
Tang, Y., Liu, Z., Zhou, Z., & Luo, X. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3382365 · Cited by 109 · see citing works
Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations
Jung, J., Qin, L., Welleck, S., Brahman, F., Bhagavatula, C., Bras, R. J. L., & Choi, Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.82 · Cited by 117 · see citing works
Explaining Answers with Entailment Trees
Dalvi, B. B., Jansen, P. A., Tafjord, O., Xie, Z., Smith, H., Pipatanangkura, L., & Clark, P. E. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.585 · Cited by 96 · see citing works
Self-citationDo Transformer Modifications Transfer Across Implementations and Applications?
Narang, S., Chung, H. W., Tay, Y., Fedus, L., Févry, T., Matena, M. S., Malkan, K., Fiedel, N., Shazeer, N., Lan, Z., et al. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.465 · Cited by 79 · see citing works
ExT5: Towards Extreme Multi-Task Scaling for Transfer LearningPreprint
Aribandi, V., Tay, Y., Schuster, T., Rao, J., Zheng, H., Mehta, S. V., Zhuang, H., Tran, V. Q., Bahri, D., Ni, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.10952 · Cited by 94 · see citing works
Meta-learning via Language Model In-context Tuning
Chen, Y., Zhong, R., Zha, S., Karypis, G., & He, H. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.53 · Cited by 84 · see citing works
ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts
Asai, A., Salehi, M., Peters, M. E., & Hajishirzi, H. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.446 · Cited by 78 · see citing works
Self-citationConnecting the Dots: A Knowledgeable Path Generator for Commonsense Question Answering
Wang, P., Peng, N., Ilievski, F., Szekely, P. A., & Ren, X. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.369 · Cited by 67 · see citing works
Sun, Y., Shi, Q., Qi, L., & Zhang, Y. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.372 · Cited by 69 · see citing works
A Survey of Deep Learning for Mathematical Reasoning
Lu, P., Qiu, L., Yu, W., Welleck, S., & Chang, K.-W. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.817 · Cited by 84 · see citing works
Fusing Context Into Knowledge Graph for Commonsense Question Answering
Xu, Y., Zhu, C., Xu, R., Liu, Y., Zeng, M., & Huang, X. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.102 · Cited by 74 · see citing works
FLEX: Unifying Evaluation for Few-Shot NLPPreprint
Bragg, J., Cohan, A., Lo, K., & Beltagy, I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.07170 · Cited by 64 · see citing works
Explanations for CommonsenseQA: New Dataset and Models
Aggarwal, S., Mandowara, D., Agrawal, V., Khandelwal, D., Singla, P., & Garg, D. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.238 · Cited by 66 · see citing works
Erdem, E., Kuyu, M., Yagcioglu, S., Frank, A., Parcalabescu, L., Plank, B., Babii, A., Turuta, O., Erdem, A., Calixto, I., et al. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12918 · Cited by 54 · see citing works
Deep Transfer Learning & Beyond: Transformer Language Models in Information Systems Research
Gruetzemacher, R., & Paradice, D. (2022) · ACM Computing Surveys · doi:10.1145/3505245 · Cited by 61 · see citing works
Large Language Model Instruction Following: A Survey of Progresses and Challenges
Lou, R., Zhang, K., & Yin, W. (2024) · Computational Linguistics · doi:10.1162/coli_a_00523 · Cited by 92 · see citing works
Few-Shot Self-Rationalization with Natural Language Prompts
Marasović, A., Beltagy, I., Downey, D., & Peters, M. E. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.31 · Cited by 64 · see citing works
Memory-efficient Transformers via Top-k Attention
Gupta, A., Dar, G., Goodman, S., Ciprut, D., & Berant, J. (2021) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2021.sustainlp-1.5 · Cited by 54 · see citing works
He, Q., Sun, X., Diao, W., Yan, Z., Yin, D., & Fu, K. (2022) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2022.08.010 · Cited by 51 · see citing works
RetractedMarasović, A., Bhagavatula, C., Park, J. S., Bras, R. J. L., Smith, N. A., & Choi, Y. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.253 · Cited by 52 · see citing works
What do Models Learn from Question Answering Datasets?
Sen, P., & Saffari, A. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.190 · Cited by 51 · see citing works
Chain-of-Thought Reasoning in Tabular Language Models
Zheng, M., Yang, H., Jiang, W., Lin, Z., Lyu, Y., She, Q., & Wang, W. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.734 · Cited by 50 · see citing works
Still no lie detector for language models: probing empirical and conceptual roadblocks
Levinstein, B. A., & Herrmann, D. A. (2025) · Philosophical Studies · doi:10.1007/s11098-023-02094-3 · Cited by 50 · see citing works
QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question AnsweringPreprint
Yasunaga, M., Ren, H., Bosselut, A., Liang, P., & Leskovec, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.06378 · Cited by 50 · see citing works
Natural Language to Code Translation with Execution
Shi, F., Fried, D., Ghazvininejad, M., Zettlemoyer, L., & Wang, S. I. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.231 · Cited by 65 · see citing works
Alawwad, H. A., Alhothali, A., Naseem, U., Alkhathlan, A., & Jamal, A. T. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2024.111332 · Cited by 78 · see citing works
Successive Prompting for Decomposing Complex Questions
Dua, D., Gupta, S., Singh, S., & Gardner, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.81 · Cited by 46 · see citing works
Large Language Models with Controllable Working Memory
Li, D., Rawat, A. S., Zaheer, M., Wang, X., Lukasik, M., Veit, A., Yu, F., & Kumar, S. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.112 · Cited by 46 · see citing works
ScienceWorld: Is your Agent Smarter than a 5th Grader?
Wang, R., Jansen, P., Côté, M.-A., & Ammanabrolu, P. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.775 · Cited by 45 · see citing works
Zero-Shot Dialogue State Tracking via Cross-Task Transfer
Lin, Z., Liu, B., Madotto, A., Moon, S., Zhou, Z., Crook, P., Wang, Z., Zhou, Y., Cho, E., Subba, R., et al. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.622 · Cited by 52 · see citing works
KAM-CoT: Knowledge Augmented Multimodal Chain-of-Thoughts Reasoning
Mondal, D., Modi, S., Panda, S., Singh, R., & Rao, G. S. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29844 · Cited by 51 · see citing works
Domain-matched Pre-training Tasks for Dense Retrieval
Oğuz, B., Lakhotia, K., Gupta, A., Lewis, P., Karpukhin, V., Piktus, A., Chen, X., Riedel, S., Yih, S., Gupta, S., et al. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.114 · Cited by 46 · see citing works
A Multi-Modal Neural Geometric Solver with Textual Clauses Parsed from Diagram
Zhang, M.-L., Yin, F., & Liu, C. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/376 · Cited by 44 · see citing works
SPARTQA: A Textual Question Answering Benchmark for Spatial Reasoning
Mirzaee, R., Faghihi, H. R., Qiang, N., & Kordjamshidi, P. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.364 · Cited by 44 · see citing works
Pre-training Text-to-Text Transformers for Concept-centric Common SensePreprint
Zhou, W., Lee, D., Selvam, R. K., Lee, S., & Ren, X. (2021) · arXiv (Cornell University) · Cited by 42 · see citing works
A taxonomy and review of generalization research in NLPPreprint
Hupkes, D., Giulianelli, M., Dankers, V., Artetxe, M., Elazar, Y., Pimentel, T., Christodoulopoulos, C., Lasri, K., Saphra, N., Sinclair, A., et al. (2022) · arXiv (Cornell University) · doi:10.3929/ethz-b-000638992 · Cited by 41 · see citing works
Contrastive Learning of User Behavior Sequence for Context-Aware Document Ranking
Zhu, Y., Nie, J., Dou, Z., Ma, Z., Zhang, X., Du, P., Zuo, X., & Jiang, H. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482243 · Cited by 40 · see citing works
Jiang, Z., Araki, J., Ding, H., & Neubig, G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.00955 · Cited by 39 · see citing works
Testing Your Question Answering Software via Asking Recursively
Chen, S., Jin, S., & Xie, X. (2021) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678670 · Cited by 38 · see citing works
Huang, X., Liu, Z. Y., Zhang, M., 桂, 韬., 韩, 先., 户, 保., 黄, 民., 黄, 萱., 刘, 康., 刘, 挺., et al. (2023) · Scientia Sinica Informationis · doi:10.1360/ssi-2023-0113 · Cited by 40 · see citing works
Differentiable Open-Ended Commonsense Reasoning
Lin, B. Y., Sun, H., Dhingra, B., Zaheer, M., Ren, X., & Cohen, W. W. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.366 · Cited by 40 · see citing works
Zhou, Y., Shen, T., Geng, X., Long, G., & Jiang, D. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.183 · Cited by 41 · see citing works
Investigating machine moral judgement through the Delphi experiment
Jiang, L., Hwang, J. D., Bhagavatula, C., Bras, R. J. L., Liang, J. T., Levine, S., Dodge, J., Sakaguchi, K., Forbes, M., Hessel, J., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-024-00969-6 · Cited by 40 · see citing works
LOREN: Logic-Regularized Reasoning for Interpretable Fact Verification
Chen, J., Bao, Q., Sun, C., Zhang, X., Chen, J., Zhou, H., Xiao, Y., & Li, L. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21291 · Cited by 39 · see citing works
Hierarchy-Aware Multi-Hop Question Answering over Knowledge Graphs
Dong, J., Zhang, Q., Huang, X. X., Duan, K., Tan, Q., & Jiang, Z. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583376 · Cited by 37 · see citing works
A Survey for Efficient Open Domain Question Answering
Zhang, Q., Chen, S., Xu, D., Cao, Q., Chen, X., Cohn, T., & Fang, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.808 · Cited by 46 · see citing works
It’s not Rocket Science: Interpreting Figurative Language in Narratives
Chakrabarty, T., Choi, Y., & Shwartz, V. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00478 · Cited by 37 · see citing works
Roy, R. S., & Anand, A. (2021) · Synthesis lectures on information concepts, retrieval, and services · doi:10.2200/s0113ed1v01y202109icr076 · Cited by 36 · see citing works
RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval
Sarthi, P., Abdullah, S., Tuli, A., Khanna, S., Goldie, A., & Manning, C. D. (2024) · arXiv · doi:10.48550/arxiv.2401.18059 · Cited by 36 · see citing works
Yoran, O., Talmor, A., & Berant, J. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.416 · Cited by 34 · see citing works
Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering
Liu, J., Hallinan, S., Lu, X., He, P., Welleck, S., Hajishirzi, H., & Choi, Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.611 · Cited by 41 · see citing works
Self-citationLow-shot learning and class imbalance: a survey
Billion-Polak, P., Prusa, J. D., & Khoshgoftaar, T. M. (2024) · Journal Of Big Data · doi:10.1186/s40537-023-00851-z · Cited by 39 · see citing works
- Authors
- Khashabi, D., Min, S., Khot, T., Sabharwal, A., Tafjord, O., Clark, P., & Hajishirzi, H.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- University of Washington (US)
- Venue
- Findings of the Association for Computational Linguistics: EMNLP 2020
- Published
- 2020
- Pages
- 1896-1907
- OpenAlex
- W3020908159
- Cited by
- 514
- References
- 51
- Citations by year
- 2026: 23 · 2025: 52 · 2024: 63 · 2023: 129 · 2022: 113 · 2021: 101 · 2020: 33
- FWCI
- 35.91
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Alfred P. Sloan Foundation
- Allen Institute for Artificial Intelligence
- Defense Advanced Research Projects Agency (N66001, N66001-19-2-403, N66001-19)
- Office of Naval Research (N00014-18, N00014-18-1-2826, ONR-N00014, N00014)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- question answering, pre-trained language models, multiple-choice question answering, yes/no question answering, fine-tuning, factoid question answering, commonsense question answering, SQuAD, ARC-Challenge
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
