Cited by 110Topic ModelingNatural Language Processing TechniquesMultimodal Machine Learning Applications
CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text
Open accessSinha, Koustuv, Sodhani, Shagun, Dong, Jin, Pineau, Joelle, Hamilton, William L.·2019·Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
Abstract
Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, William L. Hamilton. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Crossref · 10.18653/v1/d19-1458
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
CLUTRR is a benchmark for multi-hop kinship reasoning.
“We evaluate on CLUTRR Sinha et al. [2019], a benchmark for multi-hop kinship reasoning.”
From Reason Popper-ly: Patching In-Context Reasoning with Inductive Logic Programming (2026) · cites this paper as Sinha et al. [2019]
Solving a CLUTRR instance requires composing a sequence of intermediate kinship relations.
“Solving an instance requires composing a sequence of intermediate kinship relations, for example composing sister and father to derive aunt .”
From Reason Popper-ly: Patching In-Context Reasoning with Inductive Logic Programming (2026) · cites this paper as Sinha et al. [2019]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 38 · Cited by 110
Research works printed but not linked: 31
- Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron Courville. 2019. Systematic generaliza- tion: What is required and can it be learned? In International Conference on Learning Representa- tions.
- Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large anno- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 632-642.
- Kyunghyun Cho, Bart van Merrienboer, Caglar Gul- cehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Nat- ural Language Processing (EMNLP), pages 1724- 1734.
- Rajarshi Das, Shehzaad Dhuliawala, Manzil Zaheer, Luke Vilnis, Ishan Durugkar, Akshay Krishna- murthy, Alex Smola, and Andrew McCallum. 2018. Go for a walk and arrive at the answer: Reasoning over paths in knowledge bases using reinforcement learning. In International Conference on Learning Representations.
- Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A Smith. 2018. Annotation artifacts in natural lan- guage inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107-112.
- Will Hamilton, Payal Bajaj, Marinka Zitnik, Dan Juraf- sky, and Jure Leskovec. 2018. Embedding logical queries on knowledge graphs. In S. Bengio, H. Wal- lach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Informa- tion Processing Systems 31, pages 2026-2037.
- Karl Moritz Hermann, Tomas Kocisky, Edward Grefen- stette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Informa- tion Processing Systems, pages 1693-1701.
- Geoffrey E Hinton et al. 1986. Learning distributed representations of concepts. In Proceedings of the eighth annual conference of the cognitive science so- ciety, volume 1, page 12. Amherst, MA.
- Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
- Drew Arad Hudson and Christopher D. Manning. 2018. Compositional attention networks for machine rea- soning. In International Conference on Learning Representations.
- Robin Jia and Percy Liang. 2017. Adversarial exam- ples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empiri- cal Methods in Natural Language Processing, pages 2021-2031.
- Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017. Clevr: A diagnostic dataset for com- positional language and elementary visual reason- ing. In Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on, pages 1988- 1997. IEEE.
- Dimitri Kartsaklis, Mohammad Taher Pilehvar, and Nigel Collier. 2018. Mapping text to knowl- edge graph entities using multi-sense lstms. arXiv preprint arXiv:1808.07724. · arXiv
- Divyansh Kaushik and Zachary C Lipton. 2018. How much reading does reading comprehension require? a critical investigation of popular benchmarks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5010-5015.
- Stanley Kok and Pedro Domingos. 2007. Statistical predicate invention. In Proceedings of the 24th Inter- national Conference on Machine Learning, ICML '07, pages 433-440, New York, NY, USA. ACM. · DOI
- Brenden Lake and Marco Baroni. 2018. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In In- ternational Conference on Machine Learning, pages 2879-2888.
- Nada Lavrac and Saso Dzeroski. 1994. Inductive logic programming. In WLP, pages 146-160. Springer.
- Alexander Miller, Will Feng, Dhruv Batra, Antoine Bordes, Adam Fisch, Jiasen Lu, Devi Parikh, and Jason Weston. 2017. Parlai: A dialog research soft- ware platform. In Proceedings of the 2017 Con- ference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 79-84.
- Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A cor- pus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of NAACL- HLT, pages 839-849.
- Stephen Muggleton. 1991. Inductive logic program- ming. New Generation Computing, 8(4):295-318. · DOI · Paper page
- Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268. · arXiv
- Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word repre- sentations. In NAACL.
- Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392.
- Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013. Mctest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 193-203.
- Tim Rocktäschel and Sebastian Riedel. 2017. End-to- end differentiable proving. In Advances in Neural Information Processing Systems, pages 3788-3800.
- Adam Santoro, David Raposo, David G Barrett, Ma- teusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. 2017. A simple neural net- work module for relational reasoning. In Advances in neural information processing systems, pages 4967-4976.
- Shagun Sodhani, Sarath Chandar, and Yoshua. Bengio. 2018. On Training Recurrent Neural Networks for Lifelong Learning. arXiv e-prints.
- Yu Su, Huan Sun, Brian Sadler, Mudhakar Srivatsa, Izzeddin Gur, Zenghui Yan, and Xifeng Yan. 2016. On generating characteristic-rich question sets for qa evaluation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Process- ing, pages 562-572.
- Z. Wang, L. Li, D. D. Zeng, and Y. Chen. 2018. Attention-based multi-hop reasoning for knowledge graph. In 2018 IEEE International Conference on Intelligence and Security Informatics (ISI), pages 211-213. · DOI
- Jason Weston, Antoine Bordes, Sumit Chopra, Alexan- der M Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. 2015. Towards AI-Complete question answering: A set of prerequisite toy tasks. · DOI
- Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume 1 (Long Papers), volume 1, pages 1112-1122.
- Authors
- Sinha, K., Sodhani, S., Dong, J., Pineau, J., & Hamilton, W. L.
- Affiliations
- McGill University (CA)
- Mila - Quebec Artificial Intelligence Institute (CA)
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 4505-4514
- OpenAlex
- W2968398601
- Cited by
- 110
- References
- 38
- Citations by year
- 2026: 8 · 2025: 12 · 2024: 9 · 2023: 24 · 2022: 23 · 2021: 16 · 2020: 16 · 2019: 2
- FWCI
- 5.45
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Canadian Institute for Advanced Research
- Samsung Advanced Institute of Technology
- Samsung
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- inductive reasoning, diagnostic benchmark, benchmark design, natural language reasoning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
