The DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index.
UNIFIEDQA: Crossing Format Boundaries with a Single QA System
Open accessAbstract
Question answering (QA) tasks have been posed using a variety of formats, such as extractive span selection, multiple choice, etc.This has led to format-specialized models, and even to an implicit division in the QA community.We argue that such boundaries are artificial and perhaps unnecessary, given the reasoning abilities we seek to teach are not governed by the format.As evidence, we use the latest advances in language modeling to build a single pre-trained QA model, UNIFIEDQA, that performs well across 20 QA datasets spanning 4 diverse formats.UNIFIEDQA performs on par with 8 different models that were trained on individual datasets themselves.Even when faced with 12 unseen datasets of observed formats, UNIFIEDQA performs surprisingly well, showing strong generalization from its out-offormat training data.Finally, fine-tuning this pre-trained QA model into specialized models results in a new state of the art on 10 factoid and commonsense QA datasets, establishing UNIFIEDQA as a strong starting point for building QA systems. 1 1 https://github.com/allenai/unifiedqaExtractive [SQuAD] Question: At what speed did the turbine operate?Context: (Nikola_Tesla) On his 50th birthday in 1906, Tesla demonstrated his 200 horsepower (150 kilowatts) 16,000 rpm bladeless turbine.... Gold answer: 16,000 rpm Multiple-Choice [ARC-challenge] Question: What does photosynthesis produce that helps plants grow?Candidate Answers: (A) water (B) oxygen (C) protein (D) sugar Gold answer: sugar Yes/No [BoolQ]Question: Was America the first country to have a president?Context: (President) The first usage of the word president to denote the highest official in a government was during the Commonwealth of England ... Gold answer: no Abstractive [NarrativeQA]Question: What does a drink from narcissus's spring cause the drinker to do? Context: Mercury has awakened Echo, who weeps for Narcissus, and states that a drink from Narcissus's spring causes the drinkers to "Grow dotingly enamored of themselves."... Gold answer: fall in love with themselves
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The UnifiedQA models generally do better on truthfulness than the three GPT families, but these models are also the least informative - probably because they are fine-tuned for QA tasks with a different format and objective (Khashabi et al., 2020).”
From TruthfulQA: Measuring How Models Mimic Human Falsehoods (2022) · cites this paper as Khashabi et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Longer phrasal or sentential answers are often used in language generation tasks [105] and in other tasks such as multiple-choice question answering (where the scores of multiple phrases are compared against each-other; Khashabi et al. [55]).”
From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [55]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
Caruana, R. (1997) · Machine Learning · doi:10.1023/a:1007379606734 · Cited by 6,636
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019) · arXiv · doi:10.48550/arxiv.1909.11942 · Cited by 4,308
Dense passage retrieval for open-domain question answering
Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., & Yih, W.-T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.550 · Cited by 3,575
Self-citationKnow What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240
Natural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069
Research works printed but not linked: 3
- Dheeru Dua, Ananth Gottumukkala, Alon Talmor, Matt Gardner, and Sameer Singh. 2019a. Compre- hensive multi-dataset evaluation of reading compre- hension. In 2nd Workshop on Machine Reading for Question Answering.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019b. Lan- guage models are unsupervised multitask learners. OpenAI Blog, 1(8):9.
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. JMLR.
Cited by
Sorted by citations
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer
Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., & Raffel, C. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.41 · Cited by 1,629
Finetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022) · arXiv · doi:10.48550/arxiv.2202.12837 · Cited by 752
Self-citationTruthfulQA: Measuring How Models Mimic Human Falsehoods
Lin, S., Hilton, J., & Evans, O. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.229 · Cited by 904
“UnifiedQA (Khashabi et al., 2020) is a T5 model (Raffel et al., 2019) fine-tuned on diverse QA tasks.”
BackgroundMethodInfluentialRethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.759 · Cited by 755
Self-citationMultitask Prompted Training Enables Zero-Shot Task Generalization
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.08207 · Cited by 612
mT5: A massively multilingual pre-trained text-to-text transformerPreprint
Xue, L., Constant, N., Roberts, A., Kale, M., Al‐Rfou, R., Siddhant, A., Barua, A., & Raffel, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11934 · Cited by 1,324
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569
Self-citationQA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering
Yasunaga, M., Ren, H., Bosselut, A., Liang, P., & Leskovec, J. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.45 · Cited by 726
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352
(2021) · International Conference on Learning Representations · Cited by 435
PPT: Pre-trained Prompt Tuning for Few-shot Learning
Gu, Y., Han, X., Liu, Z., & Huang, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.576 · Cited by 468
WinoGrande: an adversarial winograd schema challenge at scale
Sakaguchi, K., Bras, R. L., Bhagavatula, C., & Choi, Y. (2021) · Communications of the ACM · doi:10.1145/3474381 · Cited by 356
LinkBERT: Pretraining Language Models with Document Links
Yasunaga, M., Leskovec, J., & Liang, P. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.551 · Cited by 474
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus
Dodge, J., Sap, M., Marasović, A., Agnew, W., Ilharco, G., Groeneveld, D., Mitchell, M., & Gardner, M. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.98 · Cited by 306
Measuring Mathematical Problem Solving With the MATH Dataset
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., & Steinhardt, J. (2021) · arXiv · doi:10.48550/arxiv.2103.03874 · Cited by 306
Measuring Massive Multitask Language Understanding
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020) · arXiv · doi:10.48550/arxiv.2009.03300 · Cited by 291
♫ MuSiQue: Multihop Questions via Single-hop Question Composition
Trivedi, H., Balasubramanian, N., Khot, T., & Sabharwal, A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00475 · Cited by 283
Self-citationCross-Task Generalization via Natural Language Crowdsourcing Instructions
Mishra, S., Khashabi, D., Baral, C., & Hajishirzi, H. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.244 · Cited by 269
Self-citationSuper-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
Wang, Y., Mishra, S., Alipoormolabashi, P., Kordi, Y., Mirzaei, A., Naik, A., Ashok, A., Dhanasekaran, A. S., Arunkumar, A., Stap, D., et al. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.340 · Cited by 247
(2023) · doi:10.18653/v1/2023.emnlp-main.574 · Cited by 249
Xie, T., Wu, C. H., Shi, P., Zhong, R., Scholak, T., Yasunaga, M., Wu, C.-S., Zhong, M., Yin, P., Wang, S. I., et al. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.39 · Cited by 271
Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question Answering
Feng, Y., Chen, X., Lin, B. Y., Wang, P., Yan, J., & Ren, X. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.99 · Cited by 248
Generated Knowledge Prompting for Commonsense Reasoning
Liu, J., Liu, A., Lu, X., Welleck, S., West, P., Bras, R. J. L., Choi, Y., & Hajishirzi, H. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.225 · Cited by 229
Self-citationKRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQA
Marino, K., Chen, X., Parikh, D., Gupta, A., & Rohrbach, M. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01389 · Cited by 211
Parameter-efficient fine-tuning in large language models: a survey of methodologies
Wang, L., Chen, S., Jiang, L., Pan, S., Cai, R., Yang, S., & Yang, F. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11236-4 · Cited by 186
Muppet: Massive Multi-task Representations with Pre-Finetuning
Aghajanyan, A., Gupta, A., Shrivastava, A., Chen, X., Zettlemoyer, L., & Gupta, S. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.468 · Cited by 183
Jiang, Z., Araki, J., Ding, H., & Neubig, G. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00407 · Cited by 177
BBQ: A hand-built bias benchmark for question answering
Parrish, A., Chen, A., Nangia, N., Padmakumar, V., Phang, J., Thompson, J., Htut, P. M., & Bowman, S. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.165 · Cited by 186
MetaICL: Learning to Learn In Context
Min, S., Lewis, M., Zettlemoyer, L., & Hajishirzi, H. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.201 · Cited by 169
Self-citationHuman Language Understanding & Reasoning
Manning, C. D. (2022) · Daedalus · doi:10.1162/daed_a_01905 · Cited by 193
QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension
Rogers, A., Gardner, M., & Augenstein, I. (2022) · ACM Computing Surveys · doi:10.1145/3560260 · Cited by 213
Going Full-TILT Boogie on Document Understanding with Text-Image-Layout Transformer
Powalski, R., Borchmann, Ł., Jurkiewicz, D., Dwojak, T., Pietruszka, M., & Pałka, G. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86331-9_47 · Cited by 174
Guo, J., Li, J., Li, D., Tiong, A. M. H., Li, B., Tao, D., & Hoi, S. C. H. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01046 · Cited by 194
Text-to-Text Pre-Training for Data-to-Text Tasks
Kale, M., & Rastogi, A. (2020) · International Conference on Natural Language Generation · doi:10.18653/v1/2020.inlg-1.14 · Cited by 145
TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering
Hu, Y., Liu, B., Kasai, J., Wang, Y., Mari Ostendorf, E., Krishna, R., & Smith, N. A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01866 · Cited by 267
A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers
Dasigi, P., Lo, K., Beltagy, I., Cohan, A., Smith, N. A., & Gardner, M. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.365 · Cited by 138
Pretrained Transformers for Text Ranking: BERT and Beyond
Lin, J., Nogueira, R. F., & Yates, A. (2021) · Synthesis lectures on human language technologies · doi:10.2200/s01123ed1v01y202108hlt053 · Cited by 328
No Need to Lift a Finger Anymore? Assessing the Quality of Code Generation by ChatGPT
Liu, Z., Tang, Y., Luo, X., Zhou, Y., & Zhang, L. F. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3392499 · Cited by 128
NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks
Mishra, S. R., Mitra, A., Varshney, N., Sachdeva, B., Clark, P. A., Baral, C. R., & Kalyan, A. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.246 · Cited by 168
Self-citationSurface Form Competition: Why the Highest Probability Answer Isn’t Always Right
Holtzman, A., West, P., Shwartz, V., Choi, Y., & Zettlemoyer, L. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.564 · Cited by 127
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
Fabbri, A., Wu, C.-S., Liu, W., & Xiong, C. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.187 · Cited by 119
End-to-end generation of multiple-choice questions using text-to-text transfer transformer models
Rodriguez-Torrealba, R., Garcia-Lopez, E., & Garcia-Cabot, A. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118258 · Cited by 118
ZeroGen: Efficient Zero-shot Learning via Dataset Generation
Ye, J., Gao, J., Li, Q., Xu, H., Feng, J., Wu, Z., Yu, C., & Kong, L. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.801 · Cited by 200
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP
Ye, Q., Lin, B. Y., & Ren, X. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.572 · Cited by 154
Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections
Zhong, R., Lee, K., Zhang, Z., & Klein, D. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.244 · Cited by 152
Pretrained Transformers for Text Ranking: BERT and BeyondPreprint
Lin, J., Nogueira, R. F., & Yates, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.06467 · Cited by 101
ChatGPT vs SBST: A Comparative Assessment of Unit Test Suite Generation
Tang, Y., Liu, Z., Zhou, Z., & Luo, X. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3382365 · Cited by 136
Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations
Jung, J., Qin, L., Welleck, S., Brahman, F., Bhagavatula, C., Bras, R. J. L., & Choi, Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.82 · Cited by 176
Explaining Answers with Entailment Trees
Dalvi, B. B., Jansen, P. A., Tafjord, O., Xie, Z., Smith, H., Pipatanangkura, L., & Clark, P. E. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.585 · Cited by 148
Self-citationDo Transformer Modifications Transfer Across Implementations and Applications?
Narang, S., Chung, H. W., Tay, Y., Fedus, L., Févry, T., Matena, M. S., Malkan, K., Fiedel, N., Shazeer, N., Lan, Z., et al. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.465 · Cited by 88
ExT5: Towards Extreme Multi-Task Scaling for Transfer LearningPreprint
Aribandi, V., Tay, Y., Schuster, T., Rao, J., Zheng, H., Mehta, S. V., Zhuang, H., Tran, V. Q., Bahri, D., Ni, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.10952 · Cited by 132
Meta-learning via Language Model In-context Tuning
Chen, Y., Zhong, R., Zha, S., Karypis, G., & He, H. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.53 · Cited by 143
ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts
Asai, A., Salehi, M., Peters, M. E., & Hajishirzi, H. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.446 · Cited by 103
Self-citationConnecting the Dots: A Knowledgeable Path Generator for Commonsense Question Answering
Wang, P., Peng, N., Ilievski, F., Szekely, P. A., & Ren, X. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.369 · Cited by 72
Sun, Y., Shi, Q., Qi, L., & Zhang, Y. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.372 · Cited by 69
A Survey of Deep Learning for Mathematical Reasoning
Lu, P., Qiu, L., Yu, W., Welleck, S., & Chang, K.-W. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.817 · Cited by 145
Fusing Context Into Knowledge Graph for Commonsense Question Answering
Xu, Y., Zhu, C., Xu, R., Liu, Y., Zeng, M., & Huang, X. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.102 · Cited by 79
FLEX: Unifying Evaluation for Few-Shot NLPPreprint
Bragg, J., Cohan, A., Lo, K., & Beltagy, I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.07170 · Cited by 69
Explanations for CommonsenseQA: New Dataset and Models
Aggarwal, S., Mandowara, D., Agrawal, V., Khandelwal, D., Singla, P., & Garg, D. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.238 · Cited by 85
Erdem, E., Kuyu, M., Yagcioglu, S., Frank, A., Parcalabescu, L., Plank, B., Babii, A., Turuta, O., Erdem, A., Calixto, I., et al. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12918 · Cited by 54
Deep Transfer Learning & Beyond: Transformer Language Models in Information Systems Research
Gruetzemacher, R., & Paradice, D. (2022) · ACM Computing Surveys · doi:10.1145/3505245 · Cited by 68
Large Language Model Instruction Following: A Survey of Progresses and Challenges
Lou, R., Zhang, K., & Yin, W. (2024) · Computational Linguistics · doi:10.1162/coli_a_00523 · Cited by 133
Few-Shot Self-Rationalization with Natural Language Prompts
Marasović, A., Beltagy, I., Downey, D., & Peters, M. E. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.31 · Cited by 87
Memory-efficient Transformers via Top-k Attention
Gupta, A., Dar, G., Goodman, S., Ciprut, D., & Berant, J. (2021) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2021.sustainlp-1.5 · Cited by 87
He, Q., Sun, X., Diao, W., Yan, Z., Yin, D., & Fu, K. (2022) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2022.08.010 · Cited by 51
RetractedMarasović, A., Bhagavatula, C., Park, J. S., Bras, R. J. L., Smith, N. A., & Choi, Y. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.253 · Cited by 58
What do Models Learn from Question Answering Datasets?
Sen, P., & Saffari, A. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.190 · Cited by 62
Chain-of-Thought Reasoning in Tabular Language Models
Zheng, M., Yang, H., Jiang, W., Lin, Z., Lyu, Y., She, Q., & Wang, W. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.734 · Cited by 52
Still no lie detector for language models: probing empirical and conceptual roadblocks
Levinstein, B. A., & Herrmann, D. A. (2025) · Philosophical Studies · doi:10.1007/s11098-023-02094-3 · Cited by 50
QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question AnsweringPreprint
Yasunaga, M., Ren, H., Bosselut, A., Liang, P., & Leskovec, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.06378 · Cited by 50
Natural Language to Code Translation with Execution
Shi, F., Fried, D., Ghazvininejad, M., Zettlemoyer, L., & Wang, S. I. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.231 · Cited by 105
Alawwad, H. A., Alhothali, A., Naseem, U., Alkhathlan, A., & Jamal, A. T. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2024.111332 · Cited by 96
Successive Prompting for Decomposing Complex Questions
Dua, D., Gupta, S., Singh, S., & Gardner, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.81 · Cited by 106
Large Language Models with Controllable Working Memory
Li, D., Rawat, A. S., Zaheer, M., Wang, X., Lukasik, M., Veit, A., Yu, F., & Kumar, S. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.112 · Cited by 46
ScienceWorld: Is your Agent Smarter than a 5th Grader?
Wang, R., Jansen, P., Côté, M.-A., & Ammanabrolu, P. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.775 · Cited by 45
Zero-Shot Dialogue State Tracking via Cross-Task Transfer
Lin, Z., Liu, B., Madotto, A., Moon, S., Zhou, Z., Crook, P., Wang, Z., Zhou, Y., Cho, E., Subba, R., et al. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.622 · Cited by 57
KAM-CoT: Knowledge Augmented Multimodal Chain-of-Thoughts Reasoning
Mondal, D., Modi, S., Panda, S., Singh, R., & Rao, G. S. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29844 · Cited by 67
Domain-matched Pre-training Tasks for Dense Retrieval
Oğuz, B., Lakhotia, K., Gupta, A., Lewis, P., Karpukhin, V., Piktus, A., Chen, X., Riedel, S., Yih, S., Gupta, S., et al. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.114 · Cited by 50
A Multi-Modal Neural Geometric Solver with Textual Clauses Parsed from Diagram
Zhang, M.-L., Yin, F., & Liu, C. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/376 · Cited by 71
SPARTQA: A Textual Question Answering Benchmark for Spatial Reasoning
Mirzaee, R., Faghihi, H. R., Qiang, N., & Kordjamshidi, P. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.364 · Cited by 66
Pre-training Text-to-Text Transformers for Concept-centric Common SensePreprint
Zhou, W., Lee, D., Selvam, R. K., Lee, S., & Ren, X. (2021) · arXiv (Cornell University) · Cited by 42
A taxonomy and review of generalization research in NLPPreprint
Hupkes, D., Giulianelli, M., Dankers, V., Artetxe, M., Elazar, Y., Pimentel, T., Christodoulopoulos, C., Lasri, K., Saphra, N., Sinclair, A., et al. (2022) · arXiv (Cornell University) · doi:10.3929/ethz-b-000638992 · Cited by 41
Contrastive Learning of User Behavior Sequence for Context-Aware Document Ranking
Zhu, Y., Nie, J., Dou, Z., Ma, Z., Zhang, X., Du, P., Zuo, X., & Jiang, H. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482243 · Cited by 43
Jiang, Z., Araki, J., Ding, H., & Neubig, G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.00955 · Cited by 39
Testing Your Question Answering Software via Asking Recursively
Chen, S., Jin, S., & Xie, X. (2021) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase51524.2021.9678670 · Cited by 43
Huang, X., Liu, Z. Y., Zhang, M., 桂, 韬., 韩, 先., 户, 保., 黄, 民., 黄, 萱., 刘, 康., 刘, 挺., et al. (2023) · Scientia Sinica Informationis · doi:10.1360/ssi-2023-0113 · Cited by 42
Differentiable Open-Ended Commonsense Reasoning
Lin, B. Y., Sun, H., Dhingra, B., Zaheer, M., Ren, X., & Cohen, W. W. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.366 · Cited by 46
Zhou, Y., Shen, T., Geng, X., Long, G., & Jiang, D. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.183 · Cited by 47
Investigating machine moral judgement through the Delphi experiment
Jiang, L., Hwang, J. D., Bhagavatula, C., Bras, R. J. L., Liang, J. T., Levine, S., Dodge, J., Sakaguchi, K., Forbes, M., Hessel, J., et al. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-024-00969-6 · Cited by 58
LOREN: Logic-Regularized Reasoning for Interpretable Fact Verification
Chen, J., Bao, Q., Sun, C., Zhang, X., Chen, J., Zhou, H., Xiao, Y., & Li, L. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21291 · Cited by 42
Hierarchy-Aware Multi-Hop Question Answering over Knowledge Graphs
Dong, J., Zhang, Q., Huang, X. X., Duan, K., Tan, Q., & Jiang, Z. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583376 · Cited by 41
A Survey for Efficient Open Domain Question Answering
Zhang, Q., Chen, S., Xu, D., Cao, Q., Chen, X., Cohn, T., & Fang, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.808 · Cited by 54
It’s not Rocket Science: Interpreting Figurative Language in Narratives
Chakrabarty, T., Choi, Y., & Shwartz, V. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00478 · Cited by 42
Roy, R. S., & Anand, A. (2021) · Synthesis lectures on information concepts, retrieval, and services · doi:10.2200/s0113ed1v01y202109icr076 · Cited by 37
RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval
Sarthi, P., Abdullah, S., Tuli, A., Khanna, S., Goldie, A., & Manning, C. D. (2024) · arXiv · doi:10.48550/arxiv.2401.18059 · Cited by 36
Yoran, O., Talmor, A., & Berant, J. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.416 · Cited by 40
Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering
Liu, J., Hallinan, S., Lu, X., He, P., Welleck, S., Hajishirzi, H., & Choi, Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.611 · Cited by 48
Self-citation
Metadata
- Authors
- Khashabi, D., Min, S., Khot, T., Sabharwal, A., Tafjord, O., Clark, P., & Hajishirzi, H.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- University of Washington (US)
- Venue
- Findings of the Association for Computational Linguistics: EMNLP 2020
- Published
- 2020
- Pages
- 1896-1907
- OpenAlex
- W3020908159
- Cited by
- 514
- References
- 51
- Citations by year
- 2026: 23 · 2025: 52 · 2024: 63 · 2023: 129 · 2022: 113 · 2021: 101 · 2020: 33
- FWCI
- 35.91
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Alfred P. Sloan Foundation
- Allen Institute for Artificial Intelligence
- Defense Advanced Research Projects Agency (N66001, N66001-19-2-403, N66001-19)
- Office of Naval Research (N00014-18, N00014-18-1-2826, ONR-N00014, N00014)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- question answering, pre-trained language models, multiple-choice question answering, yes/no question answering, fine-tuning, factoid question answering, commonsense question answering, SQuAD, ARC-Challenge
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
