The DOI matches its Crossref record; 19 of the 21 references it cites were found in a registry or an index.
Compositional Questions Do Not Necessitate Multi-hop Reasoning
Open accessAbstract
Multi-hop reading comprehension (RC) questions are challenging because they require reading and reasoning over multiple paragraphs.We argue that it can be difficult to construct large multi-hop RC datasets.For example, even highly compositional questions can be answered with a single hop if they target specific entity types, or the facts needed to answer them are redundant.Our analysis is centered on HOTPOTQA, where we show that single-hop reasoning can solve much more of the dataset than previously thought.We introduce a single-hop BERT-based RC model that achieves 67 F1-comparable to state-of-theart multi-hop models.We also design an evaluation setting where humans are not shown all of the necessary paragraphs for the intended multi-hop reasoning but can still answer over 80% of questions.Together with detailed error analysis, these results suggest there should be an increasing focus on the role of evidence in multi-hop reasoning and possibly even a shift towards information retrieval style evaluations with large and diverse evidence collections.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While dense retrievers like Dense Passage Retrieval (DPR) [14] and late-interaction models like ColBERT [15] improve recall on single-hop queries, they often struggle with multi-hop reasoning, where the answer depends on composing information from multiple disjoint documents [20, 6].”
From Replace, Don't Expand: Mitigating Context Dilution in Multi-Hop RAG via Fixed-Budget Evidence Assembly (2025) · cites this paper as [20]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
(2017) · Cited by 11,625
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
Teaching Machines to Read and ComprehendPreprint
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,855
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., & Manning, C. D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1259 · Cited by 1,804
Bidirectional Attention Flow for Machine ComprehensionPreprint
Seo, M. J., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01603 · Cited by 1,698
Self-citationTriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690
Self-citationSemantic Parsing on Freebase from Question-Answer Pairs
Berant, J., Chou, A., Frostig, R., & Liang, P. (2013) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d13-1160 · Cited by 1,593
Research works printed but not linked: 1
- Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2017. Constructing Datasets for Multi-hop Reading Comprehension Across Documents. In TACL.
Cited by
Sorted by citations
Items 101–142 of 142
Graph Sequential Network for Reasoning over SequencesPreprint
Tu, M., Huang, J., He, X., & Zhou, B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.02001 · Cited by 5
Why Machine Reading Comprehension Models Learn Shortcuts?Preprint
Lai, Y., Zhang, C., Feng, Y., Huang, Q., & Zhao, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.01024 · Cited by 4
Learning to Solve Complex Tasks by Talking to Agents.Preprint
Khot, T., Richardson, K., Khashabi, D., & Sabharwal, A. (2021) · arXiv (Cornell University) · Cited by 4
Meta-prompting Optimized Retrieval-Augmented Generation
Rodrigues, J., & Branco, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73503-5_17 · Cited by 3
Beyond decomposition: Hierarchical dependency management in multi‐document question answering
Zheng, X., Li, Z., Chen, Q., & Zhang, Y. (2024) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24971 · Cited by 3
Task-guided Disentangled Tuning for Pretrained Language Models
Zeng, J., Jiang, Y., Wu, S., Yin, Y., & Li, M. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.247 · Cited by 3
Hey AI, Can You Solve Complex Tasks by Talking to Agents?
Khot, T., Richardson, K., Khashabi, D., & Sabharwal, A. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.142 · Cited by 13
Valentino, M., Pratt-Hartmann, I. E., & Freitas, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.01974 · Cited by 3
Re3MHQA : Retrieve, Remove, and Return facts in multi-hop QA
Cao, Z., Xie, Y., Liu, B., & Bu, K. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.127566 · Cited by 3
Generative Context Pair Selection for Multi-hop Question Answering
Dua, D., dos Santos, C. N., Ng, P., Athiwaratkun, B., Xiang, B., Gardner, M., & Singh, S. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.561 · Cited by 2
Self-citationHey, T. (2023) · OECD eBooks · doi:10.1787/7b7b1bce-en · Cited by 4
Complex Reasoning in Natural Languag
Zhao, W., Geva, M., Lin, B. Y., Yasunaga, M., Madaan, A., & Yu, T. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-tutorials.2 · Cited by 5
Towards Non-task-specific Distillation of BERT via Sentence Representation Approximation
Wu, B., Zhang, H., Li, M., Wang, Z., Feng, Q., Huang, J., & Wang, B. (2020) · Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (AACL-IJCNLP) · doi:10.18653/v1/2020.aacl-main.9 · Cited by 3
Complementary Evidence Identification in Open-Domain Question Answering
Mou, X., Yu, M., Chang, S., Feng, Y., Zhang, L., & Su, H. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.234 · Cited by 1
Clean and Learn: Improving Robustness to Spurious Solutions in API Question Answering
Yuan, S., Qin, H., Gu, X., & Shen, B. (2022) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194022500449 · Cited by 2
R3: A Reading Comprehension Benchmark Requiring Reasoning ProcessesPreprint
Wang, R., Tao, K., Song, D., Zhang, Z., Ma, X., Su, X. H., & Dai, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.01251 · Cited by 4
Taking RAG Systems to the Next Level: Reasoning That Doesn't Blow Up Your Context
Lahmy, M., & Yozevitch, R. (2026) · IEEE Swiss Conference on Data Science (SDS) · doi:10.1109/sds70563.2026.00014 · Cited by 1
Lessons from shortcomings in machine learning for medical imaging
Varoquaux, G. P., & Cheplygina, V. (2023) · OECD eBooks · doi:10.1787/b885eecd-en · Cited by 2
Measuring and Reducing Non-Multifact Reasoning in Multi-hop Question Answering.
Trivedi, H. K., Balasubramanian, N., Khot, T., & Sabharwal, A. (2020) · Cited by 1
Similarity-based equational inference in physics
Meadows, J., & Freitas, A. (2021) · Physical Review Research · doi:10.1103/physrevresearch.3.l042010 · Cited by 1
Is the Understanding of Explicit Discourse Relations Required in Machine Reading Comprehension?
Wu, Y., Schlegel, V., & Batista-Navarro, R. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.311 · Cited by 2
Towards Non-task-specific Distillation of BERT via Sentence Representation ApproximationPreprint
Wu, B., Zhang, H., Li, M., Wang, Z., Feng, Q., Huang, J., & Wang, B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.03097 · Cited by 1
Can Edge Probing Tasks Reveal Linguistic Knowledge in QA Models?Preprint
Choudhury, S. R., Bhutani, N., & Augenstein, I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.07102 · Cited by 1
Lahmy, M., & Yozevitch, R. (2025) · arXiv · doi:10.48550/arxiv.2512.10787 · Cited by 1
R4C: A Benchmark for Evaluating RC Systems to Get the Right Answer for the Right ReasonPreprint
Inoue, N., Stenetorp, P., & Inui, K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.04601
Robustifying Multi-hop QA through Pseudo-Evidentiality TrainingPreprint
Lee, K., Hwang, S.-W., Han, S.-E., & Lee, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.03242
Grounded Graph Decoding Improves Compositional Generalization in Question AnsweringPreprint
Gai, Y. L., Jain, P., Zhang, W., Gonzalez, J. E., Song, D. X., & Stoica, I. G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.03642 · Cited by 1
Generative Context Pair Selection for Multi-hop Question AnsweringPreprint
Dua, D., dos Santos, C. N., Ng, P., Athiwaratkun, B., Xiang, B., Gardner, M., & Singh, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.08744 · Cited by 1
Self-citationConnecting Attributions and QA Model Behavior on Realistic CounterfactualsPreprint
Ye, X., Nair, R., & Durrett, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.04515 · Cited by 9
Tourassi, G. D., Shankar, M. A., & Wang, F. (2023) · OECD eBooks · doi:10.1787/0e1b4c2f-en
Eroom’s Law and the decline in the productivity of biopharmaceutical R&D
Scannell, J. W. (2023) · OECD eBooks · doi:10.1787/6f256d17-en
The end of Moore’s Law? Innovation in computer systems continues at a high pace
Kressel, H. (2023) · OECD eBooks · doi:10.1787/63e48242-en · Cited by 1
Quantifying the “cognitive extent” of science and how it has changed over time and across countries
Milojević, S. (2023) · OECD eBooks · doi:10.1787/1f717652-en
Interpretability: Should – and can – we understand the reasoning of machine-learning systems?
Cartwright, H. (2023) · OECD eBooks · doi:10.1787/2d0478b9-en
OECD (2023) · OECD eBooks · doi:10.1787/f1262928-en
Choi, S., Lee, Y., Kim, J., Lee, M., & Sim, J. (2025) · International Conference on Communications, Computing, Cybersecurity, and Informatics (CCCI) · doi:10.1109/ccci65983.2025.11215121
Multi-hop clustering for reasoning chain extraction in multi-hop question answering
JAMEHSHOURANI, M., Fatemi, A., & Nematbakhsh, M. A. (2026) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-026-01203-0
Sharma, V., Singh, A., Choudhary, P., & Revathi, A. R. (2026) · International Conference on Communications and Signal Processing (ICCSP) · doi:10.1109/iccsp68173.2026.11539462
StAR: Adaptive Structure-Aware Reranking for Semantic–Structural Alignment in GraphRAG
Oh, J., & Lim, S. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809897
Cortex: A Hybrid KG-LLM Approach for Multidocument Question Answering
Zawad, N., & Khan, L. R. (2026) · Communications in computer and information science · doi:10.1007/978-981-92-1546-1_40
HyperPatch: Sequential Knowledge Editing Under n -ary Structural Drift
Chan, Y.-K., Lien, W.-S., Yao, D.-T., Ruan, B.-K., Lin, K.-Y., Shuai, H.-H., & Chiang, M.-F. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817820
Metadata
- Authors
- Min, S., Wallace, E., Singh, S., Gardner, M., Hajishirzi, H., & Zettlemoyer, L.
- Affiliations
- University of Washington (US)
- Allen Institute for Artificial Intelligence (US)
- University of California, Irvine (US)
- Corresponding author
- Wallace, Eric
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Pages
- 4249-4257
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2953163841
- Cited by
- 138
- References
- 21
- Citations by year
- 2026: 6 · 2025: 3 · 2024: 3 · 2023: 20 · 2022: 11 · 2021: 38 · 2020: 39 · 2019: 18
- FWCI
- 10.14
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1616112, 1252835, 1562364)
- Allen Institute for Artificial Intelligence
- Samsung
- Office of Naval Research (N00014-17-S-B001, N00014-18, N00014-18-1-2826, N00014, N00014-17)
- Army Research Office (W911NF-16-1-, W911NF)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- HotpotQA, dataset construction, information retrieval evaluation, error analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
