HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering

HotpotQA is a multi-hop reasoning benchmark in which each question requires synthesising evidence from exactly two disjoint supporting paragraphs.

“HotpotQA is a multi-hop reasoning benchmark in which each question requires synthesising evidence from exactly two disjoint supporting paragraphs.[3]”

SQuAD and HotpotQA are two widely used datasets in QA and QG.

“We use SQuAD (Rajpurkar et al., 2016) and HotpotQA (Yang et al., 2018) as the base datasets, which are two widely used datasets in the field of QA and QG.”

The majority of existing QA datasets are designed to answer questions based on short passages.

“The majority of existing QA datasets (Rajpurkar et al., 2018; Yang et al., 2018; Trivedi et al., 2022) are designed to answer questions based on short passages.”

Existing benchmarks such as HotpotQA, Bamboogle, and 2wiki typically involve only two information-seeking steps and are not explicitly structured to support long-horizon interactions that necessitate memory management.

“Existing benchmarks, such as HotpotQA [59], Bamboogle [38], and 2wiki [21], are often cited as multi-hop benchmarks, yet they typically involve only two information-seeking steps. Moreover, these datasets are not explicitly structured to support long-horizon interactions that necessitate the agent to manage the memory state.”

Many benchmark sets do not allow for arbitrary generation of novel questions, or draw data from publicly available sources that LLM trainers have access to.

“Many benchmark sets do not allow for arbitrary generation of novel questions, or worse, draw data from publicly available sources-the same sources LLM trainers have access to [51, 39].”

The diversity and complexity of tasks make it difficult to represent them in a unified yet flexible manner.

“Specifically, each task can have different requirements, operations, and dependencies, which makes it difficult to represent them in a unified yet flexible manner (Chen et al., 2021;Cobbe et al., 2021;Yang et al., 2018;Luo et al., 2018).”

RAG remains systematically fragile on tasks requiring multi-hop inference, a limitation well-documented on HotpotQA.

“Despite these advances, RAG remains systematically fragile on tasks requiring multi-hop inference, where the relevant evidence spans paragraphs that are individually unremarkable but jointly decisive , a limitation well-documented on HotpotQA.[3]”

HotpotQA is particularly suitable for evaluating long-term memory storage and retrieval.

“This structure makes HotpotQA particularly suitable for evaluating long-term memory storage and retrieval.”

Past k = 3, retrieval recall continues to grow but F1 degrades because low-relevance chunks contaminate the averaged state, and the linear composition operates beyond the regime where Theorem 1's exactness argument applies.

“Past k =3 , retrieval recall continues to grow but F1 degrades: low-relevance chunks at lower retrieval ranks contaminate the averaged state, and the linear composition operates beyond the regime where Theorem 1's exactness argument applies.”

At k = 1, F1 is bounded above by retrieval recall (approximately 65% of gold-supporting paragraphs appear at rank 1), and even when recall is achieved, single-chunk PRECOG cannot integrate the second supporting fact.

“At k =1 , F1 is bounded above by retrieval recall ( ∼ 65% of gold-supporting paragraphs appear at rank 1), and even when recall is achieved, single-chunk PRECOG cannot integrate the second supporting fact.”

The empirically optimal k is corpus-dependent: k = 1 for single-paragraph evidence (SQuAD), and k = 2-3 for multi-hop (HotpotQA).

“The empirically optimal k is therefore corpus-dependent: k =1 for single-paragraph evidence (SQuAD), k =2 -3 for multi-hop (HotpotQA).”

Past benchmarks for information retrieval mostly focus on easily found information and have become saturated by recent language models.

“While past benchmarks have measured the ability to retrieve information (Joshi et al., 2017;Yang et al., 2018;Thorne et al., 2018;Dinan et al., 2019;Fan et al., 2019;Mialon et al., 2023), most of these benchmarks focus on retrieving information that can be found easily, and hence have become saturated by recent language models.”

HotpotQA offers multi-hop questions, but they are shallow in depth (2 hops), and the corpus is limited to Wikipedia.

“Although datasets like HotpotQA (Yang et al., 2018) offer multi-hop questions, they are shallow in depth (2 hops), and the corpus is limited to Wikipedia.”

HotpotQA is a multi-hop question answering dataset that requires reasoning over multiple Wikipedia paragraphs.

“HotpotQA (Yang et al., 2018) is a multi-hop question answering dataset that requires reasoning over multiple Wikipedia paragraphs.”

Tasks like HotPotQA often benefit more from concise, high-level instructions than from long contexts.

“Tasks like HotPotQA (Yang et al., 2018) often benefit more from concise, high-level instructions ( e.g., how to retrieve and synthesize evidence) than from long contexts.”

Multi-hop QA benchmarks (HotpotQA, 2WikiMultiHopQA, MuSiQue) and long-document QA such as QuALITY test compositional retrieval and reasoning over a fixed corpus.

“Multi-hop QA benchmarks (HotpotQA [52], 2WikiMultiHopQA [16], MuSiQue [42]) and long-document QA such as QuALITY [33] test compositional retrieval and reasoning over a fixed corpus.”

HotpotQA primarily tests bridge reasoning and comparison reasoning.

“This dataset primarily tests bridge reasoning (connecting entity A to entity B) and comparison reasoning (e.g., 'Who is older, X or Y?').”

SQuAD and HotpotQA are representative of single-hop and multi-hop question answering tasks respectively.

“For QA, we augment SQuAD (Rajpurkar et al., 2018) and HotpotQA (Yang et al., 2018) to simulate long-context scenario. They are representative of single-hop and multi-hop question answering tasks respectively.”

HotpotQA's annotated supporting facts naturally provide structured contextual information for Stage 1 supervision.

“In our experiments, we use HotpotQA for reinforcement learning training, as its annotated supporting facts naturally provide structured contextual information for Stage 1 supervision.”

BrowseComp-Plus retains the challenging nature of the original BrowseComp while providing a more controlled and transparent experimental setup.

“BrowseComp-Plus retains the challenging nature of the original BrowseComp while providing a more controlled and transparent experimental setup similar to early pivotal evaluation benchmarks like Natural Questions (NQ) (Kwiatkowski et al., 2019) and HotpotQA (Yang et al., 2018).”

In HotpotQA, only debate brings 3% gain while other topologies fail to improve or even degrade systematic performance.

“It is noticeable that not all topologies are beneficial to MAS design, whereas positively influenced topologies only represent a small fraction of the overall set, such that, in HotpotQA (Yang et al., 2018), only debate brings 3% gain while others fail to improve or even degrade systematic performance.”

On HotpotQA-distractor, F1 is non-monotonic in k: it peaks at k = 3 and then degrades as k increases further.

“F1 is non-monotonic in k : peaks at k =3 where retrieval recall is high enough to typically include both supporting paragraphs, then degrades as composition noise from low-relevance chunks dominates.”

Topk composition with k = 2-3 recovers most of the multi-hop gap, improving F1 by +7.5 over top-1.

“Topk composition with k =2 -3 recovers most of this multi-hop gap ( +7 . 5 F1 over top-1).”

3.7% of questions in SimpleQA Verified are classified as requiring reasoning and 7.3% are multi-step.

“3 . 7% of questions are classified as requiring reasoning and 7 . 3% are multi-step.”