Compositional Questions Do Not Necessitate Multi-hop Reasoning
Open accessAbstract
Multi-hop reading comprehension (RC) questions are challenging because they require reading and reasoning over multiple paragraphs.We argue that it can be difficult to construct large multi-hop RC datasets.For example, even highly compositional questions can be answered with a single hop if they target specific entity types, or the facts needed to answer them are redundant.Our analysis is centered on HOTPOTQA, where we show that single-hop reasoning can solve much more of the dataset than previously thought.We introduce a single-hop BERT-based RC model that achieves 67 F1-comparable to state-of-theart multi-hop models.We also design an evaluation setting where humans are not shown all of the necessary paragraphs for the intended multi-hop reasoning but can still answer over 80% of questions.Together with detailed error analysis, these results suggest there should be an increasing focus on the role of evidence in multi-hop reasoning and possibly even a shift towards information retrieval style evaluations with large and diverse evidence collections.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Dense retrievers like DPR and late-interaction models like ColBERT improve recall on single-hop queries but often struggle with multi-hop reasoning.
“While dense retrievers like Dense Passage Retrieval (DPR) [14] and late-interaction models like ColBERT [15] improve recall on single-hop queries, they often struggle with multi-hop reasoning, where the answer depends on composing information from multiple disjoint documents [20, 6].”
From Replace, Don't Expand: Mitigating Context Dilution in Multi-Hop RAG via Fixed-Budget Evidence Assembly (2025) · cites this paper as [20]
The Open-domain Questions section reports that TF-IDF retrieval often fails to find the paragraphs needed for multi-hop questions. The Related Work section describes multi-hop questions as involving reasoning across multiple pieces of text. The paper does not evaluate DPR or ColBERT, or establish that those models struggle with multi-hop reasoning. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
(2017) · Cited by 11,625 · see citing works
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · see citing works
Teaching Machines to Read and ComprehendPreprint
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,779 · see citing works
Bidirectional Attention Flow for Machine ComprehensionPreprint
Seo, M., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1611.01603 · Cited by 2,322 · see citing works
Self-citationSemantic Parsing on Freebase from Question-Answer Pairs
Berant, J., Chou, A., Frostig, R., & Liang, P. (2013) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1160 · Cited by 2,318 · see citing works
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., & Manning, C. D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1259 · Cited by 1,804 · see citing works
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690 · see citing works
Self-citationReading Wikipedia to answer open-domain questions
Chen, D., Fisch, A., Weston, J., & Bordes, A. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1171 · Cited by 1,546 · see citing works
QANet: Combining Local Convolution with Global Self-Attention for Reading ComprehensionPreprint
Yu, A. W., Dohan, D., Luong, M.-T., Zhao, R., Chen, K., Norouzi, M., & Le, Q. V. (2018) · International Conference on Learning Representations · doi:10.48550/arxiv.1804.09541 · Cited by 1,169 · see citing works
The NarrativeQA Reading Comprehension Challenge
Kočiský, T., Schwarz, J., Blunsom, P., Dyer, C., Hermann, K. M., Melis, G., & Grefenstette, E. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00023 · Cited by 596 · cited 1× · see citing works
“To foster research on reasoning across multiple pieces of text, multi-hop QA has been introduced (Kočiskỳ et al., 2018; Talmor and Berant, 2018; Yang et al., 2018).”
MethodThe Web as a Knowledge-Base for Answering Complex Questions
Talmor, A., & Berant, J. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1059 · Cited by 520 · see citing works
Welbl, J., Stenetorp, P., & Riedel, S. (2018) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.1162/tacl_a_00021 · Cited by 459 · see citing works
Simple and Effective Multi-Paragraph Reading Comprehension
Clark, C., & Gardner, M. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1078 · Cited by 402 · see citing works
Self-citationWhat Makes Reading Comprehension Questions Easier?
Sugawara, S., Inui, K., Sekine, S., & Aizawa, A. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1453 · Cited by 105 · see citing works
DCN+: Mixed Objective and Deep Residual Coattention for Question AnsweringPreprint
Xiong, C., Zhong, V., & Socher, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.00106 · Cited by 90 · see citing works
Understanding Dataset Design Choices for Multi-hop Reasoning
Chen, J., & Durrett, G. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1405 · Cited by 82 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
2 of the 21 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2017. Constructing Datasets for Multi-hop Reading Comprehension Across Documents. In TACL.
Cited by
Sorted by citations
Don’t Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases
Clark, C., Yatskar, M., & Zettlemoyer, L. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1418 · Cited by 381 · see citing works
“Recent concern about dataset bias has led researchers to re-examine many popular datasets, resulting in the discovery of a wide variety of biases (Agrawal et al., 2018; Anand et al., 2018; Min et al., 2019; Schwartz et al., 2017).”
BackgroundSelf-citationPretrained Transformers Improve Out-of-Distribution Robustness
Hendrycks, D., Liu, X., Wallace, E., Dziedzic, A., Krishnan, R., & Song, D. X. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.244 · Cited by 312 · see citing works
Self-citation♫ MuSiQue: Multihop Questions via Single-hop Question Composition
Trivedi, H., Balasubramanian, N., Khot, T., & Sabharwal, A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00475 · Cited by 283 · see citing works
Evaluating Models’ Local Decision Boundaries via Contrast Sets
Gardner, M., Artzi, Y., Basmov, V., Berant, J., Bogin, B., Chen, S., Dasigi, P., Dua, D., Elazar, Y., Gottumukkala, A., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.117 · Cited by 265 · see citing works
Self-citationConstructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps
Ho, X., Duong Nguyen, A.-K., Sugawara, S., & Aizawa, A. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.580 · Cited by 287 · see citing works
“Min et al. (2019) clas-sified the comparison questions in HotpotQA in three types: multi-hop, context-dependent multi-hop, and single-hop.”
BackgroundMethodInfluentialHierarchical Graph Network for Multi-hop Question Answering
Fang, Y., Sun, S., Gan, Z., Pillai, R., Wang, S., & Liu, J. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.710 · Cited by 176 · see citing works
Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering
Asai, A., Hashimoto, K., Hajishirzi, H., Socher, R., & Xiong, C. (2019) · arXiv · doi:10.48550/arxiv.1911.10470 · Cited by 152 · see citing works
Self-citationA Discrete Hard EM Approach for Weakly Supervised Question Answering
Min, S., Chen, D., Hajishirzi, H., & Zettlemoyer, L. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1284 · Cited by 137 · see citing works
Self-citationSelective Question Answering under Domain Shift
Kamath, A., Jia, R., & Liang, P. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.503 · Cited by 153 · see citing works
Unsupervised Question Decomposition for Question Answering
Perez, E., Lewis, P., Yih, W.-T., Cho, K., & Kiela, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.713 · Cited by 123 · see citing works
Levy, M., Jacoby, A., & Goldberg, Y. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.818 · Cited by 119 · see citing works
Concealed Data Poisoning Attacks on NLP Models
Wallace, E., Zhao, T., Feng, S., & Singh, S. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.13 · Cited by 115 · see citing works
Self-citationLarge Language Models are not Fair Evaluators
Wang, P., Li, L., Chen, L., Cai, Z., Zhu, D., Lin, B., Cao, Y., Kong, L., Liu, Q., Liu, T., et al. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.511 · Cited by 138 · see citing works
Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering
Asai, A. (2025) · TIB Data Manager · doi:10.57702/2jfyctjr · Cited by 109 · see citing works
KLUE: Korean Language Understanding EvaluationPreprint
Park, S., Moon, J., Kim, S., Cho, W. I., Han, J., Park, J., Song, C., Kim, J., Song, Y., Oh, T., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.09680 · Cited by 153 · see citing works
HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification
Jiang, Y., Bordia, S., Zhong, Z., Dognin, C., Singh, M., & Bansal, M. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.309 · Cited by 121 · see citing works
B reak It Down: A Question Understanding Benchmark
Wolfson, T., Geva, M., Gupta, A., Gardner, M., Goldberg, Y., Deutch, D., & Berant, J. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00309 · Cited by 136 · see citing works
Self-citationAnswering Complex Open-domain Questions Through Iterative Query Generation
Qi, P., Lin, X., Mehr, L., Wang, Z., & Manning, C. D. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1261 · Cited by 96 · see citing works
Transformer-XH: Multi-Evidence Reasoning with eXtra Hop Attention
Zhao, C., Xiong, C., Rosset, C., Xia, S., Bennett, P. N., & Tiwary, S. (2020) · International Conference on Learning Representations · Cited by 94 · see citing works
Knowledge Guided Text Retrieval and Reading for Open Domain Question AnsweringPreprint
Min, S., Chen, D., Zettlemoyer, L., & Hajishirzi, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.03868 · Cited by 82 · see citing works
Self-citationInvariant RationalizationPreprint
Chang, S., Zhang, Y., Yu, M., & Jaakkola, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.09772 · Cited by 65 · see citing works
Logic-Guided Data Augmentation and Regularization for Consistent Question Answering
Asai, A., & Hajishirzi, H. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.499 · Cited by 66 · see citing works
Self-citationMulti-hop Question Answering via Reasoning ChainsPreprint
Chen, J., Lin, S.-T., & Durrett, G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.02610 · Cited by 73 · see citing works
MultiModalQA: Complex Question Answering over Text, Tables and ImagesPreprint
Talmor, A., Yoran, O., Catav, A., Lahav, D., Wang, Y., Asai, A., Ilharco, G., Hajishirzi, H., & Berant, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.06039 · Cited by 91 · see citing works
Self-citationSelf-Assembling Modular Networks for Interpretable Multi-Hop Reasoning
Jiang, Y., & Bansal, M. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1455 · Cited by 57 · see citing works
Evaluating NLP Models via Contrast SetsPreprint
Gardner, M., Artzi, Y., Basmova, V., Berant, J., Bogin, B., Chen, S., Dasigi, P., Dua, D., Elazar, Y., Gottumukkala, A., et al. (2020) · arXiv (Cornell University) · Cited by 65 · see citing works
Self-citationLook at the First Sentence: Position Bias in Question Answering
Ko, M., Lee, J., Kim, H., Kim, G., & Kang, J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.84 · Cited by 60 · see citing works
QuALITY: Question Answering with Long Input Texts, Yes!
Pang, R. Y., Parrish, A., Joshi, N., Nangia, N., Phang, J., Chen, A., Padmakumar, V., Ma, J., Thompson, J., He, H., et al. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.391 · Cited by 51 · see citing works
A Simple Yet Strong Pipeline for HotpotQA
Groeneveld, D., Khot, T., Mausam, & Sabharwal, A. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.711 · Cited by 46 · see citing works
CommonsenseQA 2.0: Exposing the Limits of AI through GamificationPreprint
Talmor, A., Yoran, O., Bras, R. J. L., Bhagavatula, C., Goldberg, Y., Choi, Y., & Berant, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.05320 · Cited by 59 · see citing works
Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models
Khot, T., Khashabi, D., Richardson, K., Clark, P. E., & Sabharwal, A. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.99 · Cited by 48 · see citing works
Learning to Model and Ignore Dataset Bias with Mixed Capacity Ensembles
Clark, C., Yatskar, M., & Zettlemoyer, L. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.272 · Cited by 48 · see citing works
Self-citationIIRC: A Dataset of Incomplete Information Reading Comprehension Questions
Ferguson, J., Gardner, M., Hajishirzi, H., Khot, T., & Dasigi, P. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.86 · Cited by 47 · see citing works
Self-citationR4C: A Benchmark for Evaluating RC Systems to Get the Right Answer for the Right Reason
Inoue, N., Stenetorp, P., & Inui, K. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.602 · Cited by 43 · see citing works
Evaluating Models' Local Decision Boundaries via Contrast SetsPreprint
Gardner, M., Artzi, Y., Basmova, V., Berant, J., Bogin, B., Chen, S., Dasigi, P., Dua, D., Elazar, Y., Gottumukkala, A., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.02709 · Cited by 83 · see citing works
Self-citationBreak It Down: A Question Understanding BenchmarkPreprint
Wolfson, T., Geva, M., Gupta, A., Gardner, M., Goldberg, Y., Deutch, D., & Berant, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.11770 · Cited by 40 · see citing works
Self-citationComplex Factoid Question Answering with a Free-Text Knowledge Graph
Zhao, C., Xiong, C., Qian, X., & Boyd-Graber, J. (2020) · International World Wide Web Conference (WWW) · doi:10.1145/3366423.3380197 · Cited by 37 · see citing works
Obtaining Faithful Interpretations from Compositional Neural Networks
Subramanian, S., Bogin, B., Gupta, N., Wolfson, T., Singh, S., Berant, J., & Gardner, M. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.495 · Cited by 36 · see citing works
Self-citationYang, L., Zhang, S., Qin, L., Li, Y., Wang, Y., Liu, H., Wang, J., Xie, X., & Zhang, Y. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.806 · Cited by 43 · see citing works
Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset BiasesPreprint
Clark, C., Yatskar, M., & Zettlemoyer, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.03683 · Cited by 61 · see citing works
Self-citationQuestion Answering is a Format; When is it Useful?Preprint
Gardner, M., Berant, J., Hajishirzi, H., Talmor, A., & Min, S. (2019) · arXiv.org · doi:10.48550/arxiv.1909.11291 · Cited by 60 · see citing works
Self-citationSemantic matching in machine reading comprehension: An empirical study
Liu, Q., Mao, R., Geng, X., & Cambria, E. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103145 · Cited by 34 · see citing works
Mavi, V., Jangra, A., & Jatowt, A. (2024) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000102 · Cited by 79 · see citing works
TextGraphs 2019 Shared Task on Multi-Hop Inference for Explanation Regeneration
Jansen, P. A., & Ustalov, D. (2019) · Workshop on Graph Based Methods for Natural Language Processing · doi:10.18653/v1/d19-5309 · Cited by 32 · see citing works
Why Machine Reading Comprehension Models Learn Shortcuts?
Lai, Y., Zhang, C., Feng, Y., Huang, Q., & Zhao, D. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.85 · Cited by 32 · see citing works
MASKER: Masked Keyword Regularization for Reliable Text Classification
Moon, S. J., Mo, S., Lee, K., Lee, J., & Shin, J. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17601 · Cited by 32 · see citing works
A Survey on Explainability in Machine Reading ComprehensionPreprint
Thayaparan, M., Valentino, M., & Freitas, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.00389 · Cited by 37 · see citing works
Is Multihop QA in DiRe Condition? Measuring and Reducing Disconnected Reasoning
Trivedi, H., Balasubramanian, N., Khot, T., & Sabharwal, A. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.712 · Cited by 29 · see citing works
General Greedy De-Bias Learning
Han, X., Wang, S., Su, C., Huang, Q., & Tian, Q. C. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3240337 · Cited by 27 · see citing works
On Making Reading Comprehension More Comprehensive
Gardner, M., Berant, J., Hajishirzi, H., Talmor, A., & Min, S. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5815 · Cited by 29 · see citing works
Self-citationBenefits of Intermediate Annotations in Reading Comprehension
Dua, D., Singh, S., & Gardner, M. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.497 · Cited by 24 · see citing works
Self-citationInterpreting Predictions of NLP Models
Wallace, E., Gardner, M., & Singh, S. (2020) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2020.emnlp-tutorials.3 · Cited by 25 · see citing works
Self-citationA Framework for Evaluation of Machine Reading Comprehension Gold StandardsPreprint
Schlegel, V., Valentino, M., Freitas, A., Nenadić, G., & Batista-Navarro, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.04642 · Cited by 21 · see citing works
Do Multi-hop Readers Dream of Reasoning Chains?
Wang, H., Yu, M., Guo, X., Das, R., Xiong, W., & Gao, T. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5813 · Cited by 22 · see citing works
Quizbowl: The Case for Incremental Question AnsweringPreprint
Rodríguez, P., Feng, S., Iyyer, M., He, H., & Boyd-Graber, J. L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.04792 · Cited by 30 · see citing works
Flexible Generation of Natural Language Deductions
Bostrom, K., Zhao, X., Chaudhuri, S., & Durrett, G. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.506 · Cited by 21 · see citing works
Why do you think that? Exploring Faithful Sentence-Level Rationales Without Supervision
Glockner, M., Habernal, I., & Gurevych, I. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.97 · Cited by 24 · see citing works
Joint Passage Ranking for Diverse Multi-Answer Retrieval
Min, S., Lee, K., Chang, M.-W., Toutanova, K., & Hajishirzi, H. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.560 · Cited by 26 · see citing works
Self-citationGMAT: Global Memory Augmentation for TransformersPreprint
Gupta, A., & Berant, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03274 · Cited by 26 · see citing works
Do Multi-Hop Question Answering Systems Know How to Answer the Single-Hop Sub-Questions?
Tang, Y., Ng, H. T., & Tung, A. K. H. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.283 · Cited by 20 · see citing works
From Easy to Hard: Two-Stage Selector and Reader for Multi-Hop Question Answering
Li, X., Lei, W.-J., & Yang, Y.-B. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096119 · Cited by 21 · see citing works
Benchmarking Machine Reading Comprehension: A Psychological Perspective
Sugawara, S., Stenetorp, P., & Aizawa, A. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.137 · Cited by 21 · see citing works
NOAHQA: Numerical Reasoning with Interpretable Graph Question Answering Dataset
Zhang, Q., Wang, L., Yu, S., Wang, S., Wang, Y., Jiang, J., & Lim, E. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.350 · Cited by 18 · see citing works
TextGraphs 2020 Shared Task on Multi-Hop Inference for Explanation Regeneration
Jansen, P. A., & Ustalov, D. (2020) · doi:10.18653/v1/2020.textgraphs-1.10 · Cited by 19 · see citing works
Schlegel, V., Nenadić, G., & Batista-Navarro, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.14709 · Cited by 15 · see citing works
Out-of-Distribution Generalization in Natural Language Processing: Past, Present, and Future
Yang, L., Song, Y., Ren, X., Lyu, C., Wang, Y., Zhuo, J., Liu, L., Wang, J., Foster, J., & Zhang, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.276 · Cited by 16 · see citing works
Rissanen Data Analysis: Examining Dataset Characteristics via Description LengthPreprint
Perez, E., Kiela, D., & Cho, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.03872 · Cited by 13 · see citing works
Connecting Attributions and QA Model Behavior on Realistic Counterfactuals
Ye, X., Nair, R., & Durrett, G. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.447 · Cited by 13 · see citing works
A Discrete Hard EM Approach for Weakly Supervised Question AnsweringPreprint
Min, S., Chen, D., Hajishirzi, H., & Zettlemoyer, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.04849 · Cited by 22 · see citing works
Self-citationAnswering Complex Open-domain Questions Through Iterative Query GenerationPreprint
Qi, P., Lin, X., Mehr, L., Wang, Z., & Manning, C. D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.07000 · Cited by 20 · see citing works
Analyzing Dynamic Adversarial Training Data in the Limit
Wallace, E., Williams, A., Jia, R., & Kiela, D. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.18 · Cited by 15 · see citing works
Self-citationMitigating the Position Bias of Transformer Models in Passage Re-ranking
Hofstätter, S., Lipani, A., Althammer, S., Zlabinger, M., & Hanbury, A. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-72113-8_16 · Cited by 14 · see citing works
Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledge.Preprint
Talmor, A., Tafjord, O., Clark, P. E., Goldberg, Y., & Berant, J. (2020) · arXiv (Cornell University) · Cited by 12 · see citing works
Customizing Triggers with Concealed Data Poisoning.Preprint
Wallace, E., Zhao, T. Z., Feng, S., & Singh, S. (2020) · arXiv (Cornell University) · Cited by 12 · see citing works
Self-citationWhat-if I ask you to explain: Explaining the effects of perturbations in procedural text
Rajagopal, D., Tandon, N., Clark, P., Dalvi, B. B., & Hovy, E. H. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.300 · Cited by 11 · see citing works
A Human–Machine Interaction Scheme Based on Background Knowledge in 6G-Enabled IoT Environment
Yuan, Y., & Cai, X. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3050880 · Cited by 10 · see citing works
Robustifying Multi-hop QA through Pseudo-Evidentiality Training
Lee, K., Hwang, S.-W., Han, S.-E., & Lee, D. (2021) · doi:10.18653/v1/2021.acl-long.476 · Cited by 9 · see citing works
Wang, P., Xun, R., Liu, T., Dai, D., Chang, B., & Sui, Z. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482236 · Cited by 14 · see citing works
Explicit Planning Helps Language Models in Logical Reasoning
Zhao, H., Wang, K., Yu, M., & Mei, H. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.688 · Cited by 11 · see citing works
Infusing Multi-Hop Medical Knowledge Into Smaller Language Models for Biomedical Question Answering
Chen, J., Wei, Z., Shen, W., & Shang, R. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3547444 · Cited by 7 · see citing works
Teaching Broad Reasoning Skills for Multi-Step QA by Generating Hard Contexts
Trivedi, H. K., Balasubramanian, N., Khot, T., & Sabharwal, A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.439 · Cited by 7 · see citing works
Nishida, K., Nishida, K., Saito, I., & Yoshida, S. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9534370 · Cited by 7 · see citing works
Wallace, E., Tomlin, N., Xu, A., Yang, K., Pathak, E., Ginsberg, M. L., & Klein, D. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.219 · Cited by 7 · see citing works
Self-citationEvaluating Explanations for Reading Comprehension with Realistic Counterfactuals.Preprint
Ye, X., Nair, R., & Durrett, G. (2021) · arXiv (Cornell University) · Cited by 6 · see citing works
Zhao, C., Xiong, C., Boyd-Graber, J. L., & Daumé, H. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.756 · Cited by 7 · see citing works
Generating Hypothetical Events for Abductive Inference
Paul, D., & Frank, A. (2021) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2021.starsem-1.6 · Cited by 6 · see citing works
Grounded Graph Decoding improves Compositional Generalization in Question Answering
Gai, Y. L., Jain, P., Zhang, W., Gonzalez, J. E., Song, D. X., & Stoica, I. G. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.157 · Cited by 6 · see citing works
Analyzing the Effectiveness of the Underlying Reasoning Tasks in Multi-hop Question Answering
Ho, X., Nguyen, A.-K. D., Sugawara, S., & Aizawa, A. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.87 · Cited by 5 · see citing works
What Else Do I Need to Know? The Effect of Background Information on Users’ Reliance on QA Systems
Goyal, N., Briakou, E., Liu, A., Baumler, C., Bonial, C., Micher, J. C., Voss, C. R., Carpuat, M. J., & Daumé, H. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.201 · Cited by 10 · see citing works
TextGraphs 2021 Shared Task on Multi-Hop Inference for Explanation Regeneration
Jansen, P. A., Thayaparan, M., Valentino, M., & Ustalov, D. (2021) · Workshop on Graph Based Methods for Natural Language Processing · doi:10.18653/v1/2021.textgraphs-1.17 · Cited by 19 · see citing works
Do Question Answering Modeling Improvements Hold Across Benchmarks?
Liu, N. F., Lee, T., Jia, R., & Liang, P. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.736 · Cited by 9 · see citing works
Counterfactual Multihop QA: A Cause-Effect Approach for Reducing Disconnected Reasoning
Guo, W., Gong, Q., Rao, Y., & Lai, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.231 · Cited by 4 · see citing works
Hop, Union, Generate: Explainable Multi-hop Reasoning without Rationale Supervision
Zhao, W., Chiu, J., Cardie, C., & Rush, A. M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.1001 · Cited by 5 · see citing works
Paired Examples as Indirect Supervision in Latent Decision Models
Gupta, N., Singh, S., Gardner, M., & Roth, D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.466 · Cited by 4 · see citing works
Self-citationSchlegel, V., Nenadić, G., & Batista-Navarro, R. (2022) · Natural Language Engineering · doi:10.1017/s1351324922000171 · Cited by 22 · see citing works
Weakly-Supervised Question Answering with Effective Rank and Weighted Loss over Candidates
Qin, H., Zhu, J., & Shen, B. (2021) · International World Wide Web Conference (WWW) · doi:10.1145/3442381.3449919 · Cited by 4 · see citing works
Self-Assembling Modular Networks for Interpretable Multi-Hop ReasoningPreprint
Jiang, Y., & Bansal, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.05803 · Cited by 6 · see citing works
Generating Followup Questions for Interpretable Multi-hop Question AnsweringPreprint
Malon, C. D., & Bai, B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.12344 · Cited by 8 · see citing works
- Authors
- Min, S., Wallace, E., Singh, S., Gardner, M., Hajishirzi, H., & Zettlemoyer, L.
- Affiliations
- University of Washington (US)
- Allen Institute for Artificial Intelligence (US)
- University of California, Irvine (US)
- Corresponding author
- Wallace, Eric
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Pages
- 4249-4257
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2953163841
- Cited by
- 138
- References
- 21
- Citations by year
- 2026: 6 · 2025: 3 · 2024: 3 · 2023: 20 · 2022: 11 · 2021: 38 · 2020: 39 · 2019: 18
- FWCI
- 10.14
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1616112, 1252835, 1562364)
- Allen Institute for Artificial Intelligence
- Samsung
- Office of Naval Research (N00014-17-S-B001, N00014-18, N00014-18-1-2826, N00014, N00014-17)
- Army Research Office (W911NF-16-1-, W911NF)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- HotpotQA, dataset construction, information retrieval evaluation, error analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
