The DOI matches its Crossref record; 30 of the 32 references it cites were found in a registry or an index.
Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies
Open accessAbstract
A key limitation in current datasets for multi-hop reasoning is that the required steps for answering the question are mentioned in it explicitly. In this work, we introduce StrategyQA, a question answering (QA) benchmark where the required reasoning steps are implicit in the question, and should be inferred using a strategy. A fundamental challenge in this setup is how to elicit such creative questions from crowdsourcing workers, while covering a broad range of potential strategies. We propose a data collection procedure that combines term-based priming to inspire annotators, careful control over the annotator population, and adversarial filtering for eliminating reasoning shortcuts. Moreover, we annotate each question with (1) a decomposition into reasoning steps for answering it, and (2) Wikipedia paragraphs that contain the answers to each step. Overall, StrategyQA includes 2,780 examples, each consisting of a strategy question, its decomposition, and evidence paragraphs. Analysis shows that questions in StrategyQA are short, topic-diverse, and cover a wide range of strategies. Empirically, we show that humans perform well (87%) on this task, while our best baseline reaches an accuracy of ∼ 66%.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“3 . 7% of questions are classified as requiring reasoning and 7 . 3% are multi-step.”
From SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge (2025) · cites this paper as Geva et al., 2021
In the citing paper
“Compared to fine-tuned models, LLMs keep the superiority on most datasets, such as StrategyQA [40] and ARC-C [28].”
From Harnessing the power of llms in practice: a survey on chatGPT and beyond (2024) · cites this paper as [40]
Citations
Cites
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342
Natural Questions: A benchmark for question answering research
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00276 · Cited by 2,069
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., & Manning, C. D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1259 · Cited by 1,804
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
Mihaylov, T., Clark, P. E., Khot, T., & Sabharwal, A. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1260 · Cited by 1,602
Self-citationAnnotation Artifacts in Natural Language Inference Data
Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S., & Smith, N. A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2017 · Cited by 907
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., & Toutanova, K. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1300 · Cited by 655
A Corpus for Reasoning about Natural Language Grounded in Photographs
Suhr, A., Zhou, S., Zhang, A. Q., Zhang, I., Bai, H., & Artzi, Y. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1644 · Cited by 634
Flagged first, then by citations
2 of the 32 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- Robertson, 1995, Okapi at TREC-3, 109
Cited by
Sorted by citations
Items 301–348 of 348
Exploiting Latent and Implicit Chain of Thought for Efficient Multi-Hop Question Answering
Shakil, H., Srinivasan, V., Jeelani, H., Gunaratna, K., & Chappidi, S. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11461764
CoV-ME: Memory Editing for Self-Consistent Reasoning in Large Language Models
Hossain, M. T., Nourani, M. S., & Sultana, N. (2025) · IEEE International Conference on Robotics, Automation, Artificial-Intelligence and Internet-of-Things (RAAICON) · doi:10.1109/raaicon69033.2025.11502550
HERTy-Wiki: A Benchmark for Hierarchical Entity Reasoning and Typing in Wikidata
Obretincheva, N., Yannakoudakis, H., & Simperl, E. P. B. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-25159-6_15
Automating structured reasoning with entity-driven semantic simplification in Llms
Liao, J., Yan, C., Wang, P., & Wan, Y. (2026) · Systems Science & Control Engineering · doi:10.1080/21642583.2026.2672171
Liu, D., Yu, Y., & Wu, Y. N. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.05.10.724161
Measuring Reasoning Quality in LLMs: A Multi-Dimensional Behavioral Framework
Şenol, A., Agrawal, G., & Liu, H. (2026) · Big Data and Cognitive Computing · doi:10.3390/bdcc10090300 · Cited by 3
PriRAG: Privacy-Preserving Inference for Retrieval-Augmented Large Language Models
Wang, H., Huang, Y., Zheng, W., Fan, P., Nan, G., Wang, S., Xu, M., & Qi, T. (2026) · IEEE Transactions on Dependable and Secure Computing · doi:10.1109/tdsc.2026.3698487
Sun, X., Wei, S., Pan, T., Wang, Y., Bosch, J. A., Echizen, I., Ali, A. E., & Sugawara, S. (2026) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3774935.3806190 · Cited by 1
Xu, Z., Gardiner, J., & Belguith, S. (2026) · ACM Asia Conference on Computer and Communications Security (ASIA CCS) · doi:10.1145/3779208.3785271
Adaptive Beam Search with Shannon Entropy for Data-Centric Reasoning in LLMs
Kim, Y., Jeong, Y., Kim, J., & Cho, S. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-1468-6_43
ASTRA: Adversarial Stealthy Trigger Reasoning Attacks for Black-Box LLMs
Park, S.-G., Park, S., & Choi, D. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-1468-6_41
Wang, Z., & Wang, L. (2026) · Machine Intelligence Research · doi:10.1007/s11633-025-1602-0
遠藤, 格., Shen, C., Wang, W., Fan, S., Xie, L., Tian, X., Chen, P., Liu, X. D., Chen, Y., & Ye, J. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.134297
Lee, S., & Lee, D.-G. (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2026.115378
DynaChain: On-Demand Retrieval-Augmented Generation via Dynamic Reasoning Chain Correction
Cui, H., Chen, Y., Kou, X., Ding, Y., & Wang, H. (2026) · International Conference on Artificial Intelligence and Industrial Technology Applications (AIITA) · doi:10.1109/aiita69518.2026.11567339
Agentic Self-RAG: Multi-Agent Reasoning for Self-Correcting Retrieval-Augmented Generation
Hassan, S. B., -, A., & Abbas, M. (2026) · International Conference on IT and Industrial Technologies (ICIT) · doi:10.1109/icit68548.2026.11577708 · Cited by 1
Self-Awareness before Action: Mitigating Logical Inertia via Proactive Cognitive Awareness
Fan, F., Liu, P., FengZhe, L., Yang, S., & Yan, G. (2026) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2026.findings-acl.793
Temporal Structure in LLM Reasoning: Analyzing Hidden States and Semantic Embeddings
Otomura, K., Nakamura, K., Kobayashi, K., Nakamura, M., Hase, R., & Hato, J. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29532-3_21
D$$^3$$: Dispute-Driven Debate for Stabilizing Multi-agent Reasoning in Large Language Models
Liu, Y., Sun, S. Y., Wang, Y., Zhang, S., & Li, J. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-2852-2_47
Toroghi, A., Kalarde, F. M., & Sanner, S. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3808628
Rajapakse, T. C., & de Rijke, M. (2026) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3805712.3809891
SARAG: Sufficiency-Aware Retrieval Augmented Generation
Zheng, Z., Zhang, Y., Wang, Y., Luo, W., & Zuo, L. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3417-2_37
A Survey of Commonsense Reasoning in LLMs
Teo, N., Huang, D., & Wang, Z. (2026) · ACM Computing Surveys · doi:10.1145/3832753
IMRDF: A Training-Free Framework for Memory-Reasoning Decoupling in Large Language Models
Li, Y., Weng, Z., Cai, Z., & Yang, B. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-3426-4_25
Lu, G., Zhan, W., Peng, L., & Li, L. (2026) · Information Processing & Management · doi:10.1016/j.ipm.2026.104980
Liu, Y., Huang, H., Yang, Y., Zeng, Z., Zhou, Y., Wu, Z., & Gao, Y. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.133921
Chu, Y., Shao, M., Liu, Y., Hao, B., Lin, Y., Wang, J., & Wang, R. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3818055 · Cited by 4
Minimal AIF-inspired response control for evidence-aware and dependable LLM behavior
Feng, C., Guo, Y., & Wang, G. (2026) · Computing · doi:10.1007/s00607-026-01723-z
Data and Knowledge Dual-Driven Text Embedding Approaches: A Survey
Chen, X., 詹佳青, Lin, J., Liu, H., Zhang, Q., Chen, J., & Chen, H. (2026) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2026.3721883
Gao, J., Chen, X., Miao, D., Zhang, H., Qin, X., & Lu, P. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.134106
ANCHOR: Enhancing complex reasoning in large language models via metacognitive task anchoring
Wang, J., Zheng, C., Tang, Z., & Xie, X. (2026) · Neurocomputing · doi:10.1016/j.neucom.2026.134855
GS-Chaff: Multi-Agent Prompt-Level Semantic Chaffing for Privacy-Preserving LLM Inference
Zhou, Q., Wang, Z., Yue, Z., Cai, L., Liu, K., & Qin, C. (2026) · Sensors · doi:10.3390/s26175385
A survey on synergizing large language models and knowledge graphs for reasoning
Wang, X., Bai, H., Huang, S., Wu, G., Yang, D., Wang, S., Liao, J., Zhang, Y., Liu, M., & Liu, L. (2026) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2026.116132
Large or Small? Cloud or Edge? Investigating the Use of Language Models for Smart Drone Services
He, P., Liu, X., Xu, J., Kua, J., & Jia, D. (2026) · IEEE International Conference on Web Services (ICWS) · doi:10.1109/icws72778.2026.00018
Personality Matters: Task-Specific Effects of Psychological Framings in Agentified LLMs
Bellan, P., Magnolini, S., Haez, S. G., Sanna, L., & Dragoni, M. (2025) · IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT) · doi:10.1109/wi-iat67162.2025.00060
Verifier-Guided Risk-Aware Search for LLM Reasoning Under Budget Constraints
Zhao, L., Qin, S., & Shi, Y. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-38410-2_35
LLM Prompt Optimization: From Zero-Shot to Automatic Role-Playing Generation Using Ontology
Hauna, A. D. A., Khomsah, S., Yunus, A. P., & Fukui, M. (2026) · Journal of Advanced Computational Intelligence and Intelligent Informatics · doi:10.20965/jaciii.2026.p1404
Load-Aware Adaptive Degradation Scheduling for Multi-User Concurrent Edge LLM Inference
Tan, Y.-H., Wu, W.-Y., & Zhang, C. (2026) · 2026 5th International Conference on Artificial Intelligence, Internet of Things and Cloud Computing Technology (AIoTC) · doi:10.1109/aiotc71250.2026.11688797
Towards Adaptive Reverse Thinking in Large Language Models: Quantifying and Optimizing
Fang, A., & 朝, 乐. (2026) · Data Intelligence · doi:10.3724/2096-7004.di.2026.2092
Feng, J., Guo, J., He, Y., Zeng, Q., Wang, K., Lin, S., & Lai, J. (2026) · International Journal of Pattern Recognition and Artificial Intelligence · doi:10.1142/s0218001426510183
COMPASS: Efficient Structured Reasoning Via Action Policy Compression*
Li, K., 杜友田, Xu, Z., Gou, Y., Chen, R., & Yan, C.-B. (2026) · IEEE International Conference on Automation Science and Engineering (CASE) · doi:10.1109/case69030.2026.11704303
ReVerTree: Recursive Verification Tree for Self-checking Mathematical Reasoning
You, C., Zhang, T., Chen, Q., Wang, C., & He, X. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-5696-9_14
Your Agent’s Memories Are Not Its Own: Forged Reasoning Attack on LLM Agent Memory and Its Defense
Karamchandani, N., Nagasubramaniam, P., Zhu, S., & Wu, D. (2026) · IEEE Conference on Communications and Network Security · doi:10.1109/cns69863.2026.11720906 · Cited by 8
Contrastive Chain-of-Thought Decoding for Large Language Models to Mitigate Inference Hallucination
Long, Q., Li, X., Liu, Y., Wu, H., & Lin, L. (2026) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn69854.2026.11729174
RL-Enhanced ProRAG: A Proactive Retrieval-Augmented Generation Framework for Multi-Hop Reasoning
Luo, X., Huo, Y., Qin, Y., Feng, F., Liu, G., Xia, D., & Wong Derek, F. (2026) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn69854.2026.11728741
When to Nudge? Budget-Constrained Test-Time Coordination for Reliable LLM Decision Making
Yu, C.-Y., & Zhang, W. (2026) · International Conference on Machine Learning and Applications · doi:10.1109/icmla69890.2026.00152
Budget-Aware Verifier-Guided Test-Time Reasoning for Efficient Large Language Models
Wang, S., Zhang, B., Gong, Y., & He, C. (2026) · doi:10.1109/iotaiee70863.2026.11749415
The Landscape of Emerging AI Agent Architectures for Reasoning, Planning, and Tool Calling: A Survey
Masterman, T., Besen, S., Sawtell, M., & Chao, A. (2024) · arXiv · doi:10.48550/arxiv.2404.11584 · Cited by 35
“StrategyQA improves upon this by assessing models’ reasoning abilities over multiple steps, but the answers are limited to Yes/No responses [7].”
Background
Metadata
- Authors
- Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., & Berant, J.
- Affiliations
- Tel Aviv University (IL)
- Allen Institute for Artificial Intelligence (US)
- University of Pennsylvania (US)
- Corresponding author
- Geva, Mor
- Khashabi, Daniel
- Segal, Elad
- Khot, Tushar
- Roth, Dan
- Berant, Jonathan
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2021
- Volume
- 9
- Pages
- 346-361
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W3159959439
- Cited by
- 242
- References
- 32
- Citations by year
- 2026: 77 · 2025: 71 · 2024: 38 · 2023: 42 · 2022: 6 · 2021: 8
- FWCI
- 8.27
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- question answering benchmark, multi-hop reasoning, StrategyQA, data collection, human performance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review