CoChat

Benchmarks for Automated Commonsense Reasoning: A Survey

Open access
Davis, Ernest · 2024 · ACM Computing Surveys
Cited by 76

The DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.

Abstract

More than one hundred benchmarks have been developed to test the commonsense knowledge and commonsense reasoning abilities of artificial intelligence (AI) systems. However, these benchmarks are often flawed, and many aspects of common sense remain untested. Consequently, there is currently no reliable way of measuring to what extent existing AI systems have achieved these abilities. This article surveys the development and uses of AI commonsense benchmarks. It enumerates 139 commonsense benchmarks that have been developed: 102 text-based, 18 image-based, 12 video-based, and 7 based in simulated physical environments. It gives more detailed descriptions of twelve of these, three from each category. It surveys the various methods used to construct commonsense benchmarks. It discusses the nature of common sense, the role of common sense in AI, the goals served by constructing commonsense benchmarks, desirable features of commonsense benchmarks, and flaws and gap in existing benchmarks. It concludes with a number of recommendations for future development of commonsense AI benchmarks; most importantly, that the creators of benchmarks invest the work needed to ensure that benchmark examples are consistently high quality.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3615355

Cites

35 of 35 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 126

  1. 10.1080/11663081.2022.2041352 · DOI
  2. Kelsey R. Allen, Anton Bakhtin, Kevin Smith, Joshua B. Tenenbaum, and Laurens van der Maaten. 2020. Ogre: An object-based generalization for reasoning environment. In Proceedings of the NeurIPS Workshop on Object Representations for Learning and Reasoning. Retrieved from http://lvdmaaten.github.io/publications/papers/ogre.pdf
  3. Saja Al-Tawalbeh and Mohammad Al-Smadi. 2020. A benchmark Arabic dataset for commonsense explanation. arXiv:2012.10251. Retrieved from https://arxiv.org/abs/2012.10251 · arXiv
  4. Stéphane Aroca-Ouellette, Cory Paik, Alessandro Roncone, and Katharina Kann. 2021. PROST: Physical reasoning about objects through space and time. In Findings of the Association for Computational Linguistics: (ACL-IJCNLP’21), 4597–4608. Retrieved from https://aclanthology.org/2021.findings-acl.404.pdf
  5. Mana Ashida and Saku Sugawara. 2022. Possible stories: Evaluating situated commonsense reasoning under multiple possible scenarios. In Proceedings of the 29th International Conference on Computational Linguistics, 3606–3630. Retrieved from https://aclanthology.org/2022.coling-1.319/
  6. Anton Bakhtin, Laurens van der Maaten, Justin Johnson, Laura Gustafson, and Ross Girshick. 2019. Phyre: A new benchmark for physical reasoning. Advances in Neural Information Processing Systems 32 (2019). Retrieved from https://arxiv.org/abs/1908.05656 · arXiv
  7. Fabien Baradel, Natalia Neverova, Julien Mille, Greg Mori, and Christian Wolf. 2019. CoPhy: Counterfactual learning of physical dynamics. In Proceedings of the International Conference on Learning Representations. Retrieved from https://arxiv.org/abs/1909.12000 · arXiv
  8. Daniel M. Bear, Elias Wang, Damian Mrowca, Felix Binder, Hsiao-Yu Fish Tung, R. T. Pramod, Cameron Holdaway, Sirui Tao, Kevin A. Smith, Fan-Yun Sun, Li Fei-Fei, Nancy Kanwisher, Joshua B. Tenenbaum, Daniel L. K. Yamins, and Judith E. Fan. 2021. Physion: Evaluating physical prediction from vision in humans and machines. Conference on Neural Information Processing Systems 35. Retrieved from https://openreview.net/forum?id=CXyZrKPz4CU
  9. 10.1609/aaai.v36i11.21496 · DOI
  10. 10.1609/aaai.v34i05.6239 · DOI · Paper page
  11. Yonatan Bitton, Nitzan Bitton Guetta, Ron Yosef, Yuval Elovici, Mohit Bansal, Gabriel Stanovsky, and Roy Schwartz. 2022. WinoGAViL: Gamified association benchmark to challenge vision-and-language models. Advances in Neural Information Processing Systems 35 (2022), 26549–26564. Retrieved from https://proceedings.neurips.cc/paper_files/paper/2022/hash/a96fe863f85c59789bba63588a9557b4-Abstract-Datasets_and_Benchmarks.html
  12. Rishi Bomassani et al. 2021. On the opportunities and risks of foundation models. arXiv:2105.07258. Retrieved from https://arxiv.org/abs/2108.07258 · arXiv
  13. 10.18653/v1/2020.emnlp-main.85 · DOI · Paper page
  14. T. L. Brown, H. E. LeMay, and B. Bursten. 2003. Chemistry: The Central Science (9th. ed.). Prentice Hall.
  15. Qingxing Cao Bailin Li Xiaodan Liang and Liang Lin. 2019. Explainable high-order visual question reasoning: A new benchmark and knowledge-routed network. arXiv:1909.10128. Retrieved from https://arxiv.org/abs/1909.10128 · arXiv
  16. 10.3758/BF03206384 · DOI · Paper page
  17. 10.18653/v1/W19-2008 · doi:10.18653/v1/w19-2008
  18. Matthew Crosby, Benjamin Beyret, Murray Shanahan, José Hernández-Orallo, Lucy Cheke, and Marta Halina. 2020. The Animal-AI testbed and competition. Neurips 2019 Competition and Demonstration Track 123 (2020), 164–176. Retrieved from http://proceedings.mlr.press/v123/crosby20a/crosby20a.pdf
  19. 10.1017/S1351324909990234 · DOI · Paper page
  20. Ernest Davis. 1990. Representations of Commonsense Knowledge. Morgan Kaufmann San Mateo Calif.
  21. 10.1080/13875868.2013.824976 · doi:10.1080/13875868.2013.824976
  22. 10.1609/aimag.v37i1.2637 · DOI
  23. Ernest Davis. 2021. Using human skills taxonomies and tests as measures of AI. In Proceedings of the AI and the Future of Skills, Volume 1: Capabilities and Assessments. Stuart Elliott (Ed.), OECD Publishing.
  24. Ernest Davis. To appear. Mathematics word problems common sense and artificial intelligence. Bulletin of the American Mathematical Society. To appear. Retrieved from https://arxiv.org/abs/2301.09723 · arXiv
  25. Ernest Davis and Gary Marcus. 2015. Communications of the ACM 58, 9 (2015), 92–105. Retrieved from http://cacm.acm.org/magazines/2015/9/191169-commonsense-reasoning-and-commonsense-knowledge-in-artificial-intelligence/
  26. Guy Deutscher. 2010. Through the Language Glass: Why the World Looks Different in Different Languages. Metropolitan Books, New York
  27. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the NAACL-2019.
  28. Yan Ding Xiaohan Zhang Saeid Amiri Nieqing Cao Hao Yang Chad Esselink and Shiqi Zhang. 2022. Robot task planning and situation handling in open worlds. arXiv:2210.01287. Retrieved from https://arxiv.org/abs/2210.01287 · arXiv
  29. 10.18653/v1/2021.emnlp-main.670 · DOI
  30. Jonathan St BT Evans. 2017. Dual process theory: Perspectives and problems. In Proceedings of the Dual Process Theory 2.0. Wim de Neys (Ed.), Taylor and Francis. 137–155.
  31. 10.18653/v1/2020.emnlp-main.61 · DOI
  32. 10.1109/IROS51168.2021.9636830 · DOI
  33. Kanishk Gandhi, Gala Stojnic, Brenden M. Lake, and Moira R. Dillon. 2021. Baby Intuitions Benchmark (BIB): Discerning the goals, preferences, and actions of others. Advances in Neural Information Processing Systems 34 (2021), 9963–9976.
  34. Difei Gao, Ruiping Wang, Shiguang Shan, and Xilin Chen. 2022. CRIC: A VQA dataset for compositional reasoning on vision and commonsense. IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 5 (2022), 5561–5578. Retrieved from https://ieeexplore.ieee.org/abstract/document/9905976
  35. 10.18653/v1/P18-1086 · DOI
  36. Xavier Garcia, Yamini Bansal, Colin Cherry, George Foster, Maxim Krikun, Melvin Johnson, and Orhan Firat. 2023. The unreasonable effectiveness of few-shot learning for machine translation. In International Conference on Machine Learning, 10867–10878. Retrieved from https://proceedings.mlr.press/v202/garcia23a.html
  37. Prajwal Gatti, Abhirama Subramanyam Penamakuri, Revant Teotia, Anand Mishra, Shubhashis Sengupta, and Roshni Ramnani. 2022. COFAR: Commonsense and factual reasoning in image search. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing, 1185–1199. Retrieved from https://aclanthology.org/2022.aacl-main.87/
  38. Deepanway Ghosal, Pengfei Hong, Siqi Shen, Navonil Majumder, Rada Mihalcea, and Soujanya Poria. 2021. CIDER: Commonsense inference for dialogue explanation and reasoning. In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue. 301–313.
  39. Rohit Girdhar and Deva Ramanan. 2019. CATER: A diagnostic dataset for compositional actions & TEmporal reasoning. In International Conference on Learning Representations. Retrieved from https://openreview.net/forum?id=HJgzt2VKPB
  40. 10.1609/aaai.v30i1.9881 · DOI
  41. Andrew S. Gordon and Jerry R. Hobbs. 2017. A Formal Theory oF Commonsense Psychology: How People Think People Think. Cambridge University Press, Cambridge.
  42. 10.1109/CVPR46437.2021.01113 · DOI
  43. 10.1162/tacl_a_00469 · DOI
  44. 10.18653/v1/2023.eacl-main.30 · DOI
  45. 10.18653/v1/2021.emnlp-main.307 · DOI
  46. Dan Hendrycks Collin Burns Steven Basart Andrew Critch Jerry Li Dawn Song and Jacob Steinhardt. 2020. Aligning AI with shared human values. arXiv:2008.02275. Retrieved from https://arxiv.org/abs/2008.02275 · arXiv
  47. 10.1007/978-3-031-20059-5_32 · DOI
  48. Yining Hong, Li Yi, Josh Tenenbaum, Antonio Torralba, and Chuang Gan. 2021. Ptr: A benchmark for part-based conceptual, relational, and physical reasoning. Advances in Neural Information Processing Systems 34 (2021), 17427–17440. Retrieved from https://proceedings.neurips.cc/paper/2021/hash/918f5cd5a5c0d48671d4d4fc54bab2e9-Abstract.html
  49. 10.18653/v1/2020.acl-main.132 · DOI
  50. 10.18653/v1/D19-1243 · DOI · Paper page
  51. Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. 2020. (COMET-)ATOMIC 2020: On symbolic and neural commonsense knowledge graphs. Proceedings of the AAAI Conference on Artificial Intelligence 35, 7 (2020), 6384–6392. Retrieved from https://arxiv.org/abs/2010.05953 · arXiv
  52. 10.1007/978-3-030-35288-2_24 · doi:10.1007/978-3-030-35288-2_24
  53. Nicos Isaak and Loizos Michael. 2020. Winventor: A machine-driven approach for the development of winograd schemas. ICAART (2), 26–35. Retrieved from https://www.scitepress.org/Papers/2020/89026/89026.pdf
  54. 10.1109/CVPR42600.2020.01025 · DOI · Paper page
  55. Minqi Jiang, Jelena Luketina, Nantas Nardelli, Pasquale Minervini, Philip H. S. Torr, Shimon Whiteson, and Tim Rocktäschel. 2020. WordCraft: An environment for benchmarking commonsense agents. In Language in Reinforcement Learning Workshop at International Conference on Machine Learning. Retrieved from https://openreview.net/forum?id=A0wHsDMUXAE
  56. 10.18653/v1/2020.findings-emnlp.445 · DOI · Paper page
  57. 10.1007/978-3-031-19842-7_21 · DOI
  58. 10.18653/v1/2022.naacl-main.66 · DOI
  59. Bernard Koch, Emily Denton, Alex Hanna, and Jacob G. Foster. 2021. Reduced, reused and recycled: The life of a dataset in machine learning research. In Proceedings of the 35th Conference on Neural Information Processing Systems. Retrieved from https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/file/3b8a614226a953a8cd9526fca6fe9ba5-Paper-round2.pdf
  60. Vid Kocijan Ernest Davis Thomas Lukasiewicz Gary Marcus and Leora Morgenstern. 2023. The defeat of the Winograd schema challenge. Artificial Intelligence in press. Retrieved from https://www.sciencedirect.com/science/article/pii/S0004370223001170 · DOI
  61. Michael Kosinski. 2023. Theory of mind may have spontaneously emerged in large language models. arXiv:2302.02083. Retrieved from https://arxiv.org/abs/2302.02083 · arXiv
  62. 10.18653/v1/2021.findings-acl.53 · DOI
  63. 10.18653/v1/2020.deelio-1.5 · DOI
  64. 10.1609/aimag.v6i4.510 · doi:10.1609/aimag.v6i4.510 · Paper page
  65. 10.1109/ICCV48922.2021.00205 · DOI
  66. Peng-Hsuan Li, Tsan-Yu Yang, and Wei-Yun Ma. 2020. CA-EHN: Commonsense analogy from E-HowNet. In Proceedings of the 12th Language Resources and Evaluation Conference. 2984–2990. Retrieved from https://aclanthology.org/2020.lrec-1.365/
  67. 10.18653/v1/2020.emnlp-main.557 · DOI · Paper page
  68. 10.18653/v1/2021.acl-long.102 · DOI
  69. Bill Yuchen Lin Ziyi Wu Yichi Yang Dong-Ho Lee and Xiang Ren. 2021. RiddleSense: Reasoning about riddle questions featuring linguistic creativity and commonsense knowledge. In Findings of the Association for Computational Linguistics: (ACL-IJCNLP’21) 1504–1515. Retrieved from https://aclanthology.org/2021.findings-acl.131.pdf
  70. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common objects in context. In Proceedings of the European Conference on Computer Vision. 740–755.
  71. Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. 2021. IconQA: A new benchmark for abstract diagram understanding and visual language reasoning. In 35th Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). Retrieved from https://openreview.net/forum?id=uXa9oBDZ9V1
  72. Gary Marcus and Ernest Davis. 2023. How not to test GPT. Communications of the ACM Blog. Retrieved from https://cacm.acm.org/blogs/blog-cacm/270142-how-not-to-test-gpt-3/fulltext
  73. John McWhorter. 2014. The Language Hoax: Why The World Looks the Same in Any Language. Oxford University Press, Oxford.
  74. 10.18653/v1/D18-1260 · DOI · Paper page
  75. Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, and Ashwin Kalyan. 2022. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP). 5807–5832. Retrieved from https://aclanthology.org/2022.emnlp-main.392/
  76. 10.1016/B978-012369388-4/50072-2 · DOI
  77. Kai Nakamura, Sharon Levy, and William Yang Wang. 2020. Fakeddit: A new multimodal benchmark dataset for fine-grained fake news detection. In Proceedings of the 12th Language Resources and Evaluation Conference. 6149–6157. Retrieved from https://aclanthology.org/2020.lrec-1.755/
  78. Arvind Narayanan and Sayash Kapoor. 2023. Is GPT-4 getting worse over time? Retrieved from https://www.aisnakeoil.com/p/is-gpt-4-getting-worse-over-time. Accessed October 5, 2023.
  79. Qiang Ning, Hao Wu, Rujun Han, Nanyun Peng, Matt Gardner, and Dan Roth. 2020. TORQUE: A reading comprehension dataset of temporal ordering questions. arXiv:2005.00242. Retrieved from https://arxiv.org/abs/2005.00242 · arXiv
  80. Yasumasa Onoe, Michael J. Q. Zhang, Eunsol Choi, and Greg Durrett. 2021. CREAK: A dataset for commonsense reasoning over entity knowledge. arXiv:2109.01653. Retrieved from https://arxiv.org/abs/2109.01653 · arXiv
  81. Simon Ostermann, Ashutosh Modi, Michael Roth, Stefan Thater, Manfred Pinkal. 2018. MCScript: A novel dataset for assessing machine comprehension using script knowledge. In Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC’18). 3567–3574. Retrieved from https://aclanthology.org/L18-1564.pdf
  82. 10.18653/v1/2021.conll-1.28 · DOI
  83. Amandalynne Paullada, Inioluwa Deborah Raji, Emily M. Bender, Emily Denton, and Alex Hanna. 2021. Data and its (dis)contents: A survey of dataset development and use in machine learning research. Patterns 2, 11 (2021). https://www.cell.com/patterns/pdf/S2666-3899(21)00184-7.pdf
  84. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning, 90–95. Retrieved from https://people.ict.usc.edu/gordon/public_html/publications/AAAI-SPRING11A.PDF
  85. 10.1007/s11263-016-0987-1 · DOI · Paper page
  86. Stuart Russell and Peter Norvig. 2020. Artificial Intelligence: A Modern Approach (4th. ed.). Prentice Hall.
  87. 10.18653/v1/2020.findings-emnlp.418 · DOI
  88. Henrique Santos Ke Shen Alice M. Mulvehill Yasaman Razeghi Deborah L. McGuinness and Mayank Kejriwal. 2022. A Theoretically Grounded Benchmark for Evaluating Machine Commonsense. arXiv:2203.12184. Retrieved from https://arxiv.org/abs/2203.12184 · arXiv
  89. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019. SocialIQA: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. 4463–4473. Retrieved from https://aclanthology.org/D19-1454/
  90. 10.18653/v1/2021.acl-short.85 · DOI · Paper page
  91. Jaehyung Seo, Chanjun Park, Hyeonseok Moon, Sugyeong Eo, Myunghoon Kang, Seounghoon Lee, and Heuiseok Lim. 2021. KommonGen: A dataset for Korean generative commonsense reasoning evaluation. In Proceedings of the Annual Conference on Human and Language Technology. 55–60. Retrieved from https://koreascience.kr/article/CFKO202130060697830.pdf
  92. Tianmin Shu, Abhishek Bhandwaldar, Chuang Gan, Kevin Smith, Shari Liu, Dan Gutfreund, Elizabeth Spelke, Joshua Tenenbaum, and Tomer Ullman. 2021. AGENT: A benchmark for core psychological reasoning. In Proceedings of the International Conference on Machine Learning. 9614–9625. Retrieved from https://research.ibm.com/publications/agent-a-benchmark-for-core-psychological-reasoning
  93. 10.1007/978-3-319-46448-0_31 · DOI · Paper page
  94. Ashish Singh, Hang Su, SouYoung Jin, Huaizu Jiang, Chetan Manjesh, Geng Luo, Ziwei He, Li Hong, Erik Learned-Miller, and Rosie Cowell. 2019. Half&Half: New tasks and benchmarks for studying visual common sense. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 1–4.
  95. Shikhar Singh Nuan Wen Yu Hou Pegah Alipoormolabashi Te-Lin Wu Xuezhe Ma and Nanyun Peng. 2021. COM2SENSE: A commonsense reasoning benchmark with complementary sentences. In Findings of the Association for Computational Linguistics: (ACL-IJCNLP’21). 883–898. Retrieved from https://aclanthology.org/2021.findings-acl.78/ · DOI
  96. 10.1609/aaai.v31i1.11164 · DOI · Paper page
  97. Elizabeth Spelke. 2022. What Babies Know: Core Knowledge and Composition. Volume 1. Oxford University Press, Oxford.
  98. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, et al. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research. Retrieved from https://openreview.net/forum?id=uyTL5Bvosj
  99. Shane Storks Qiaozi Gao and Joyce Chai. 2019. Commonsense reasoning for natural language understanding: A survey of benchmarks resources and approaches. arXiv:1904.01172. https://arxiv.org/abs/1904.01172 · arXiv
  100. 10.18653/v1/2021.findings-emnlp.422 · DOI
  101. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4149–4158. Retrieved from https://aclanthology.org/N19-1421/
  102. Alon Talmor, Ori Yoran, Ronan Le Bras, Chandra Bhagavatula, Yoav Goldberg, Yejin Choi, and Jonathan Berant. 2021. CommonsenseQA 2.0: Exposing the Limits of AI through Gamification. In 35th Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1). Retrieved from https://openreview.net/forum?id=qF7FlUT5dxa
  103. 10.1145/2812802 · DOI · Paper page
  104. Frank van Harmelen, Vladimir Lifschitz, and Bruce Porter (eds). 2008. Handbook of Knowledge Representation. Elsevier, Amsterdam.
  105. Veniamin Veselovsky, Manoel Horta Ribeiro, and Robert West. 2023. Artificial artificial artificial intelligence: Crowd workers widely use large language models for text production tasks. arXiv:2306.07899. Retrieved from https://arxiv.org/abs/2306.07899 · arXiv
  106. 10.1068/p5262 · DOI · Paper page
  107. 10.5555/3171642.3171825 · doi:10.5555/3171642.3171825
  108. 10.1109/TPAMI.2017.2754246 · DOI · Paper page
  109. 10.18653/v1/N18-2049 · DOI
  110. Luca Weihs, Amanda Rose Yuile, Renée Baillargeon, Cynthia L. Fisher, Gary Marcus, Roozbeh Mottaghi, and Aniruddha Kembhavi. 2022. Benchmarking progress to infant-Level physical reasoning in AI. Transactions on Machine Learning.
  111. Adina, Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through Inference. In Proceedings of NAACL-HLT, 1112–1122. Retrieved from https://aclanthology.org/N18-1101/
  112. 10.18653/v1/P18-2016 · DOI
  113. 10.18653/v1/2020.coling-main.419 · DOI · Paper page
  114. Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum. 2019. Clevrer: Collision events for video representation and reasoning. In International Conference on Learning Representations. Retrieved from https://openreview.net/forum?id=HkxYzANYDB
  115. 10.18653/v1/2021.emnlp-main.162 · DOI
  116. 10.18653/v1/2022.emnlp-main.132 · DOI
  117. Haoxuan You Rui Sun Zhecan Wang Kai-Wei Chang and Shih-Fu Chang. 2022. Find someone who: Visual commonsense understanding in human-centric grounding. In Findings of the Association for Computational Linguistics: EMNLP. 5444–5454. Retrieved from https://aclanthology.org/2022.findings-emnlp.399/ · DOI
  118. 10.1007/978-3-031-19836-6_17 · DOI
  119. Mo Yu Xiaoxiao Guo Yufei Feng Xiaodan Zhu Michael Greenspan and Murray Campbell. 2020. Deriving commonsense inference tasks from interactive fictions. arXiv:2010.09788. Retrieved from https://arxiv.org/abs/2010.09788 · arXiv
  120. Ales Zagar, Marko Robnik-Sikonja, Teja Goli, and Spela Arhar Holdt. 2020. Slovene translation of SuperGLUE, Slovenian language resource repository CLARIN.SI, ISSN 2820–4042. Retrieved from http://hdl.handle.net/11356/1380. Accessed October 5, 2023.
  121. 10.18653/v1/D18-1009 · DOI · Paper page
  122. 10.18653/v1/2020.acl-main.508 · DOI
  123. 10.1162/tacl_a_00068 · DOI · Paper page
  124. Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. Record: Bridging the gap between human and machine commonsense reading comprehension. arXiv:1810.12885. Retrieved from https://arxiv.org/abs/1810.12885 · arXiv
  125. 10.18653/v1/D19-1332 · DOI · Paper page
  126. 10.18653/v1/2021.naacl-main.107 · DOI

Cited by

See all 76

Metadata

Authors
Davis, E.
Venue
ACM Computing Surveys
Published
2023-10-23
Volume
56
Issue
4
Pages
1-41
ISSN
  • 0360-0300 (ISSN-L)
  • 1557-7341
OpenAlex
W4386607580
Cited by
76
References
35
FWCI
6.59
Type
journal article
Language
en
Open access
hybrid

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.