The DOI matches its Crossref record; 19 of the 21 references it cites were found in a registry or an index.
Compositional Questions Do Not Necessitate Multi-hop Reasoning
Open accessAbstract
Multi-hop reading comprehension (RC) questions are challenging because they require reading and reasoning over multiple paragraphs.We argue that it can be difficult to construct large multi-hop RC datasets.For example, even highly compositional questions can be answered with a single hop if they target specific entity types, or the facts needed to answer them are redundant.Our analysis is centered on HOTPOTQA, where we show that single-hop reasoning can solve much more of the dataset than previously thought.We introduce a single-hop BERT-based RC model that achieves 67 F1-comparable to state-of-theart multi-hop models.We also design an evaluation setting where humans are not shown all of the necessary paragraphs for the intended multi-hop reasoning but can still answer over 80% of questions.Together with detailed error analysis, these results suggest there should be an increasing focus on the role of evidence in multi-hop reasoning and possibly even a shift towards information retrieval style evaluations with large and diverse evidence collections.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While dense retrievers like Dense Passage Retrieval (DPR) [14] and late-interaction models like ColBERT [15] improve recall on single-hop queries, they often struggle with multi-hop reasoning, where the answer depends on composing information from multiple disjoint documents [20, 6].”
From Replace, Don't Expand: Mitigating Context Dilution in Multi-Hop RAG via Fixed-Budget Evidence Assembly (2025) · cites this paper as [20]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
(2017) · Cited by 11,625
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
Teaching Machines to Read and ComprehendPreprint
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,855
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., & Manning, C. D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1259 · Cited by 1,804
Bidirectional Attention Flow for Machine ComprehensionPreprint
Seo, M. J., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01603 · Cited by 1,698
Self-citationTriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690
Self-citationSemantic Parsing on Freebase from Question-Answer Pairs
Berant, J., Chou, A., Frostig, R., & Liang, P. (2013) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d13-1160 · Cited by 1,593
Research works printed but not linked: 1
- Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2017. Constructing Datasets for Multi-hop Reading Comprehension Across Documents. In TACL.
Cited by
Sorted by citations
Don’t Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases
Clark, C., Yatskar, M., & Zettlemoyer, L. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1418 · Cited by 381
“Recent concern about dataset bias has led researchers to re-examine many popular datasets, resulting in the discovery of a wide variety of biases (Agrawal et al., 2018; Anand et al., 2018; Min et al., 2019; Schwartz et al., 2017).”
BackgroundSelf-citationPretrained Transformers Improve Out-of-Distribution Robustness
Hendrycks, D., Liu, X., Wallace, E., Dziedzic, A., Krishnan, R., & Song, D. X. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.244 · Cited by 406
Self-citation♫ MuSiQue: Multihop Questions via Single-hop Question Composition
Trivedi, H., Balasubramanian, N., Khot, T., & Sabharwal, A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00475 · Cited by 283
Evaluating Models’ Local Decision Boundaries via Contrast Sets
Gardner, M., Artzi, Y., Basmov, V., Berant, J., Bogin, B., Chen, S., Dasigi, P., Dua, D., Elazar, Y., Gottumukkala, A., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.117 · Cited by 323
Self-citationConstructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps
Ho, X., Duong Nguyen, A.-K., Sugawara, S., & Aizawa, A. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.580 · Cited by 287
“Min et al. (2019) clas-sified the comparison questions in HotpotQA in three types: multi-hop, context-dependent multi-hop, and single-hop.”
BackgroundMethodInfluentialHierarchical Graph Network for Multi-hop Question Answering
Fang, Y., Sun, S., Gan, Z., Pillai, R., Wang, S., & Liu, J. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.710 · Cited by 176
Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering
Asai, A., Hashimoto, K., Hajishirzi, H., Socher, R., & Xiong, C. (2019) · arXiv · doi:10.48550/arxiv.1911.10470 · Cited by 152
Self-citationA Discrete Hard EM Approach for Weakly Supervised Question Answering
Min, S., Chen, D., Hajishirzi, H., & Zettlemoyer, L. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1284 · Cited by 144
Self-citationSelective Question Answering under Domain Shift
Kamath, A., Jia, R., & Liang, P. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.503 · Cited by 229
Unsupervised Question Decomposition for Question Answering
Perez, E., Lewis, P., Yih, W.-T., Cho, K., & Kiela, D. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.713 · Cited by 123
Levy, M., Jacoby, A., & Goldberg, Y. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.818 · Cited by 119
Concealed Data Poisoning Attacks on NLP Models
Wallace, E., Zhao, T., Feng, S., & Singh, S. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.13 · Cited by 115
Self-citationLarge Language Models are not Fair Evaluators
Wang, P., Li, L., Chen, L., Cai, Z., Zhu, D., Lin, B., Cao, Y., Kong, L., Liu, Q., Liu, T., et al. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.511 · Cited by 138
Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering
Asai, A. (2025) · TIB Data Manager · doi:10.57702/2jfyctjr · Cited by 109
KLUE: Korean Language Understanding EvaluationPreprint
Park, S., Moon, J., Kim, S., Cho, W. I., Han, J., Park, J., Song, C., Kim, J., Song, Y., Oh, T., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.09680 · Cited by 193
HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification
Jiang, Y., Bordia, S., Zhong, Z., Dognin, C., Singh, M., & Bansal, M. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.309 · Cited by 173
B reak It Down: A Question Understanding Benchmark
Wolfson, T., Geva, M., Gupta, A., Gardner, M., Goldberg, Y., Deutch, D., & Berant, J. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00309 · Cited by 150
Self-citationAnswering Complex Open-domain Questions Through Iterative Query Generation
Qi, P., Lin, X., Mehr, L., Wang, Z., & Manning, C. D. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1261 · Cited by 106
Transformer-XH: Multi-Evidence Reasoning with eXtra Hop Attention
Zhao, C., Xiong, C., Rosset, C., Xia, S., Bennett, P. N., & Tiwary, S. (2020) · International Conference on Learning Representations · Cited by 94
Knowledge Guided Text Retrieval and Reading for Open Domain Question AnsweringPreprint
Min, S., Chen, D., Zettlemoyer, L., & Hajishirzi, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.03868 · Cited by 111
Self-citationInvariant RationalizationPreprint
Chang, S., Zhang, Y., Yu, M., & Jaakkola, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.09772 · Cited by 81
Logic-Guided Data Augmentation and Regularization for Consistent Question Answering
Asai, A., & Hajishirzi, H. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.499 · Cited by 81
Self-citationMulti-hop Question Answering via Reasoning ChainsPreprint
Chen, J., Lin, S.-T., & Durrett, G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.02610 · Cited by 97
MultiModalQA: Complex Question Answering over Text, Tables and ImagesPreprint
Talmor, A., Yoran, O., Catav, A., Lahav, D., Wang, Y., Asai, A., Ilharco, G., Hajishirzi, H., & Berant, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.06039 · Cited by 149
Self-citationSelf-Assembling Modular Networks for Interpretable Multi-Hop Reasoning
Jiang, Y., & Bansal, M. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1455 · Cited by 59
Evaluating NLP Models via Contrast SetsPreprint
Gardner, M., Artzi, Y., Basmova, V., Berant, J., Bogin, B., Chen, S., Dasigi, P., Dua, D., Elazar, Y., Gottumukkala, A., et al. (2020) · arXiv (Cornell University) · Cited by 65
Self-citationLook at the First Sentence: Position Bias in Question Answering
Ko, M., Lee, J., Kim, H., Kim, G., & Kang, J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.84 · Cited by 80
QuALITY: Question Answering with Long Input Texts, Yes!
Pang, R. Y., Parrish, A., Joshi, N., Nangia, N., Phang, J., Chen, A., Padmakumar, V., Ma, J., Thompson, J., He, H., et al. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.391 · Cited by 51
A Simple Yet Strong Pipeline for HotpotQA
Groeneveld, D., Khot, T., Mausam, & Sabharwal, A. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.711 · Cited by 48
CommonsenseQA 2.0: Exposing the Limits of AI through GamificationPreprint
Talmor, A., Yoran, O., Bras, R. J. L., Bhagavatula, C., Goldberg, Y., Choi, Y., & Berant, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.05320 · Cited by 93
Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models
Khot, T., Khashabi, D., Richardson, K., Clark, P. E., & Sabharwal, A. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.99 · Cited by 56
Learning to Model and Ignore Dataset Bias with Mixed Capacity Ensembles
Clark, C., Yatskar, M., & Zettlemoyer, L. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.272 · Cited by 55
Self-citationIIRC: A Dataset of Incomplete Information Reading Comprehension Questions
Ferguson, J., Gardner, M., Hajishirzi, H., Khot, T., & Dasigi, P. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.86 · Cited by 61
Self-citationR4C: A Benchmark for Evaluating RC Systems to Get the Right Answer for the Right Reason
Inoue, N., Stenetorp, P., & Inui, K. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.602 · Cited by 47
Evaluating Models' Local Decision Boundaries via Contrast SetsPreprint
Gardner, M., Artzi, Y., Basmova, V., Berant, J., Bogin, B., Chen, S., Dasigi, P., Dua, D., Elazar, Y., Gottumukkala, A., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.02709 · Cited by 132
Self-citationBreak It Down: A Question Understanding BenchmarkPreprint
Wolfson, T., Geva, M., Gupta, A., Gardner, M., Goldberg, Y., Deutch, D., & Berant, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.11770 · Cited by 40
Self-citationComplex Factoid Question Answering with a Free-Text Knowledge Graph
Zhao, C., Xiong, C., Qian, X., & Boyd-Graber, J. (2020) · International World Wide Web Conference (WWW) · doi:10.1145/3366423.3380197 · Cited by 39
Obtaining Faithful Interpretations from Compositional Neural Networks
Subramanian, S., Bogin, B., Gupta, N., Wolfson, T., Singh, S., Berant, J., & Gardner, M. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.495 · Cited by 44
Self-citationYang, L., Zhang, S., Qin, L., Li, Y., Wang, Y., Liu, H., Wang, J., Xie, X., & Zhang, Y. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.806 · Cited by 81
Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset BiasesPreprint
Clark, C., Yatskar, M., & Zettlemoyer, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.03683 · Cited by 95
Self-citationQuestion Answering is a Format; When is it Useful?Preprint
Gardner, M., Berant, J., Hajishirzi, H., Talmor, A., & Min, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.11291 · Cited by 47
Self-citationSemantic matching in machine reading comprehension: An empirical study
Liu, Q., Mao, R., Geng, X., & Cambria, E. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103145 · Cited by 34
Mavi, V., Jangra, A., & Jatowt, A. (2024) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000102 · Cited by 119
TextGraphs 2019 Shared Task on Multi-Hop Inference for Explanation Regeneration
Jansen, P. A., & Ustalov, D. (2019) · Workshop on Graph Based Methods for Natural Language Processing · doi:10.18653/v1/d19-5309 · Cited by 32
Why Machine Reading Comprehension Models Learn Shortcuts?
Lai, Y., Zhang, C., Feng, Y., Huang, Q., & Zhao, D. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.85 · Cited by 32
MASKER: Masked Keyword Regularization for Reliable Text Classification
Moon, S. J., Mo, S., Lee, K., Lee, J., & Shin, J. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17601 · Cited by 33
A Survey on Explainability in Machine Reading ComprehensionPreprint
Thayaparan, M., Valentino, M., & Freitas, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.00389 · Cited by 54
Is Multihop QA in DiRe Condition? Measuring and Reducing Disconnected Reasoning
Trivedi, H., Balasubramanian, N., Khot, T., & Sabharwal, A. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.712 · Cited by 29
General Greedy De-Bias Learning
Han, X., Wang, S., Su, C., Huang, Q., & Tian, Q. C. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3240337 · Cited by 29
On Making Reading Comprehension More Comprehensive
Gardner, M., Berant, J., Hajishirzi, H., Talmor, A., & Min, S. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5815 · Cited by 32
Self-citationBenefits of Intermediate Annotations in Reading Comprehension
Dua, D., Singh, S., & Gardner, M. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.497 · Cited by 26
Self-citationInterpreting Predictions of NLP Models
Wallace, E., Gardner, M., & Singh, S. (2020) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2020.emnlp-tutorials.3 · Cited by 28
Self-citationA Framework for Evaluation of Machine Reading Comprehension Gold StandardsPreprint
Schlegel, V., Valentino, M., Freitas, A., Nenadić, G., & Batista-Navarro, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.04642 · Cited by 23
Do Multi-hop Readers Dream of Reasoning Chains?
Wang, H., Yu, M., Guo, X., Das, R., Xiong, W., & Gao, T. (2019) · Workshop on Machine Reading for Question Answering (MRQA) · doi:10.18653/v1/d19-5813 · Cited by 26
Quizbowl: The Case for Incremental Question AnsweringPreprint
Rodríguez, P., Feng, S., Iyyer, M., He, H., & Boyd-Graber, J. L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.04792 · Cited by 47
Flexible Generation of Natural Language Deductions
Bostrom, K., Zhao, X., Chaudhuri, S., & Durrett, G. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.506 · Cited by 25
Why do you think that? Exploring Faithful Sentence-Level Rationales Without Supervision
Glockner, M., Habernal, I., & Gurevych, I. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.97 · Cited by 26
Joint Passage Ranking for Diverse Multi-Answer Retrieval
Min, S., Lee, K., Chang, M.-W., Toutanova, K., & Hajishirzi, H. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.560 · Cited by 39
Self-citationGMAT: Global Memory Augmentation for TransformersPreprint
Gupta, A., & Berant, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03274 · Cited by 47
Do Multi-Hop Question Answering Systems Know How to Answer the Single-Hop Sub-Questions?
Tang, Y., Ng, H. T., & Tung, A. K. H. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.283 · Cited by 32
From Easy to Hard: Two-Stage Selector and Reader for Multi-Hop Question Answering
Li, X., Lei, W.-J., & Yang, Y.-B. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096119 · Cited by 26
Benchmarking Machine Reading Comprehension: A Psychological Perspective
Sugawara, S., Stenetorp, P., & Aizawa, A. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.137 · Cited by 22
NOAHQA: Numerical Reasoning with Interpretable Graph Question Answering Dataset
Zhang, Q., Wang, L., Yu, S., Wang, S., Wang, Y., Jiang, J., & Lim, E. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.350 · Cited by 21
TextGraphs 2020 Shared Task on Multi-Hop Inference for Explanation Regeneration
Jansen, P. A., & Ustalov, D. (2020) · doi:10.18653/v1/2020.textgraphs-1.10 · Cited by 19
Schlegel, V., Nenadić, G., & Batista-Navarro, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.14709 · Cited by 15
Out-of-Distribution Generalization in Natural Language Processing: Past, Present, and Future
Yang, L., Song, Y., Ren, X., Lyu, C., Wang, Y., Zhuo, J., Liu, L., Wang, J., Foster, J., & Zhang, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.276 · Cited by 21
Rissanen Data Analysis: Examining Dataset Characteristics via Description LengthPreprint
Perez, E., Kiela, D., & Cho, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.03872 · Cited by 13
Connecting Attributions and QA Model Behavior on Realistic Counterfactuals
Ye, X., Nair, R., & Durrett, G. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.447 · Cited by 15
A Discrete Hard EM Approach for Weakly Supervised Question AnsweringPreprint
Min, S., Chen, D., Hajishirzi, H., & Zettlemoyer, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.04849 · Cited by 29
Self-citationAnswering Complex Open-domain Questions Through Iterative Query GenerationPreprint
Qi, P., Lin, X., Mehr, L., Wang, Z., & Manning, C. D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.07000 · Cited by 28
Analyzing Dynamic Adversarial Training Data in the Limit
Wallace, E., Williams, A., Jia, R., & Kiela, D. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.18 · Cited by 25
Self-citationMitigating the Position Bias of Transformer Models in Passage Re-ranking
Hofstätter, S., Lipani, A., Althammer, S., Zlabinger, M., & Hanbury, A. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-72113-8_16 · Cited by 19
Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledge.Preprint
Talmor, A., Tafjord, O., Clark, P. E., Goldberg, Y., & Berant, J. (2020) · arXiv (Cornell University) · Cited by 13
Customizing Triggers with Concealed Data Poisoning.Preprint
Wallace, E., Zhao, T. Z., Feng, S., & Singh, S. (2020) · arXiv (Cornell University) · Cited by 12
Self-citationWhat-if I ask you to explain: Explaining the effects of perturbations in procedural text
Rajagopal, D., Tandon, N., Clark, P., Dalvi, B. B., & Hovy, E. H. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.300 · Cited by 12
A Human–Machine Interaction Scheme Based on Background Knowledge in 6G-Enabled IoT Environment
Yuan, Y., & Cai, X. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2021.3050880 · Cited by 10
Robustifying Multi-hop QA through Pseudo-Evidentiality Training
Lee, K., Hwang, S.-W., Han, S.-E., & Lee, D. (2021) · doi:10.18653/v1/2021.acl-long.476 · Cited by 9
Wang, P., Xun, R., Liu, T., Dai, D., Chang, B., & Sui, Z. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482236 · Cited by 15
Explicit Planning Helps Language Models in Logical Reasoning
Zhao, H., Wang, K., Yu, M., & Mei, H. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.688 · Cited by 15
Infusing Multi-Hop Medical Knowledge Into Smaller Language Models for Biomedical Question Answering
Chen, J., Wei, Z., Shen, W., & Shang, R. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3547444 · Cited by 7
Teaching Broad Reasoning Skills for Multi-Step QA by Generating Hard Contexts
Trivedi, H. K., Balasubramanian, N., Khot, T., & Sabharwal, A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.439 · Cited by 9
Nishida, K., Nishida, K., Saito, I., & Yoshida, S. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9534370 · Cited by 8
Wallace, E., Tomlin, N., Xu, A., Yang, K., Pathak, E., Ginsberg, M. L., & Klein, D. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.219 · Cited by 10
Self-citationEvaluating Explanations for Reading Comprehension with Realistic Counterfactuals.Preprint
Ye, X., Nair, R., & Durrett, G. (2021) · arXiv (Cornell University) · Cited by 6
Zhao, C., Xiong, C., Boyd-Graber, J. L., & Daumé, H. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.756 · Cited by 8
Generating Hypothetical Events for Abductive Inference
Paul, D., & Frank, A. (2021) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2021.starsem-1.6 · Cited by 6
Grounded Graph Decoding improves Compositional Generalization in Question Answering
Gai, Y. L., Jain, P., Zhang, W., Gonzalez, J. E., Song, D. X., & Stoica, I. G. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.157 · Cited by 8
Analyzing the Effectiveness of the Underlying Reasoning Tasks in Multi-hop Question Answering
Ho, X., Nguyen, A.-K. D., Sugawara, S., & Aizawa, A. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.87 · Cited by 5
What Else Do I Need to Know? The Effect of Background Information on Users’ Reliance on QA Systems
Goyal, N., Briakou, E., Liu, A., Baumler, C., Bonial, C., Micher, J. C., Voss, C. R., Carpuat, M. J., & Daumé, H. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.201 · Cited by 12
TextGraphs 2021 Shared Task on Multi-Hop Inference for Explanation Regeneration
Jansen, P. A., Thayaparan, M., Valentino, M., & Ustalov, D. (2021) · Workshop on Graph Based Methods for Natural Language Processing · doi:10.18653/v1/2021.textgraphs-1.17 · Cited by 19
Do Question Answering Modeling Improvements Hold Across Benchmarks?
Liu, N. F., Lee, T., Jia, R., & Liang, P. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.736 · Cited by 23
Counterfactual Multihop QA: A Cause-Effect Approach for Reducing Disconnected Reasoning
Guo, W., Gong, Q., Rao, Y., & Lai, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.231 · Cited by 5
Hop, Union, Generate: Explainable Multi-hop Reasoning without Rationale Supervision
Zhao, W., Chiu, J., Cardie, C., & Rush, A. M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.1001 · Cited by 8
Paired Examples as Indirect Supervision in Latent Decision Models
Gupta, N., Singh, S., Gardner, M., & Roth, D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.466 · Cited by 6
Self-citationSchlegel, V., Nenadić, G., & Batista-Navarro, R. (2022) · Natural Language Engineering · doi:10.1017/s1351324922000171 · Cited by 27
Weakly-Supervised Question Answering with Effective Rank and Weighted Loss over Candidates
Qin, H., Zhu, J., & Shen, B. (2021) · International World Wide Web Conference (WWW) · doi:10.1145/3442381.3449919 · Cited by 4
Self-Assembling Modular Networks for Interpretable Multi-Hop ReasoningPreprint
Jiang, Y., & Bansal, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.05803 · Cited by 12
Generating Followup Questions for Interpretable Multi-hop Question AnsweringPreprint
Malon, C. D., & Bai, B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.12344 · Cited by 11
Metadata
- Authors
- Min, S., Wallace, E., Singh, S., Gardner, M., Hajishirzi, H., & Zettlemoyer, L.
- Affiliations
- University of Washington (US)
- Allen Institute for Artificial Intelligence (US)
- University of California, Irvine (US)
- Corresponding author
- Wallace, Eric
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Pages
- 4249-4257
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2953163841
- Cited by
- 138
- References
- 21
- Citations by year
- 2026: 6 · 2025: 3 · 2024: 3 · 2023: 20 · 2022: 11 · 2021: 38 · 2020: 39 · 2019: 18
- FWCI
- 10.14
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1616112, 1252835, 1562364)
- Allen Institute for Artificial Intelligence
- Samsung
- Office of Naval Research (N00014-17-S-B001, N00014-18, N00014-18-1-2826, N00014, N00014-17)
- Army Research Office (W911NF-16-1-, W911NF)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- HotpotQA, dataset construction, information retrieval evaluation, error analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
