Benchmarks for Automated Commonsense Reasoning: A Survey
Open accessThe DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.
Abstract
More than one hundred benchmarks have been developed to test the commonsense knowledge and commonsense reasoning abilities of artificial intelligence (AI) systems. However, these benchmarks are often flawed, and many aspects of common sense remain untested. Consequently, there is currently no reliable way of measuring to what extent existing AI systems have achieved these abilities. This article surveys the development and uses of AI commonsense benchmarks. It enumerates 139 commonsense benchmarks that have been developed: 102 text-based, 18 image-based, 12 video-based, and 7 based in simulated physical environments. It gives more detailed descriptions of twelve of these, three from each category. It surveys the various methods used to construct commonsense benchmarks. It discusses the nature of common sense, the role of common sense in AI, the goals served by constructing commonsense benchmarks, desirable features of commonsense benchmarks, and flaws and gap in existing benchmarks. It concludes with a number of recommendations for future development of commonsense AI benchmarks; most importantly, that the creators of benchmarks invest the work needed to ensure that benchmark examples are consistently high quality.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Maintaining knowledge about temporal intervals
Allen, J. F. (1983) · Communications of the ACM · doi:10.1145/182.358434 · Cited by 7,593 · cited 1×
“Other issues in temporal reasoning including relations of temporal intervals [2] and hypothetical time lines are not well tested in any benchmark.”
BackgroundWimmer, H. (1983) · Cognition · doi:10.1016/0010-0277(83)90004-5 · Cited by 6,305
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107 · cited 11×
“ž [77] The images in Visual Genome consisted of all the creative-commons images in the intersection of the preexisting collections MS-COCO [89] and YFCC100M [134].”
BackgroundMethodInfluentialGLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222
LabelMe: A Database and Web-Based Tool for Image Annotation
Russell, B. C., Torralba, A., Murphy, K. P., & Freeman, W. T. (2008) · International Journal of Computer Vision · doi:10.1007/s11263-007-0090-8 · Cited by 4,210
A large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 3×
“In fact, these artifacts allowed a system to solve two-thirds of the problems in SNLI and more than half the problems in MultiNLI just by seeing Q, without seeing P.”
MethodActivityNet: A large-scale video benchmark for human activity understanding
Heilbron, F. C., Escorcia, V., Ghanem, B., & Niebles, J. C. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298698 · Cited by 2,578
ConceptNet 5.5: An Open Multilingual Graph of General Knowledge
Speer, R., Chin, J., & Havasi, C. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11164 · Cited by 2,312
Research works printed but not linked: 126
- 10.1080/11663081.2022.2041352 · DOI
- Kelsey R. Allen, Anton Bakhtin, Kevin Smith, Joshua B. Tenenbaum, and Laurens van der Maaten. 2020. Ogre: An object-based generalization for reasoning environment. In Proceedings of the NeurIPS Workshop on Object Representations for Learning and Reasoning. Retrieved from http://lvdmaaten.github.io/publications/papers/ogre.pdf
- Saja Al-Tawalbeh and Mohammad Al-Smadi. 2020. A benchmark Arabic dataset for commonsense explanation. arXiv:2012.10251. Retrieved from https://arxiv.org/abs/2012.10251 · arXiv
- Stéphane Aroca-Ouellette, Cory Paik, Alessandro Roncone, and Katharina Kann. 2021. PROST: Physical reasoning about objects through space and time. In Findings of the Association for Computational Linguistics: (ACL-IJCNLP’21), 4597–4608. Retrieved from https://aclanthology.org/2021.findings-acl.404.pdf
- Mana Ashida and Saku Sugawara. 2022. Possible stories: Evaluating situated commonsense reasoning under multiple possible scenarios. In Proceedings of the 29th International Conference on Computational Linguistics, 3606–3630. Retrieved from https://aclanthology.org/2022.coling-1.319/
- Anton Bakhtin, Laurens van der Maaten, Justin Johnson, Laura Gustafson, and Ross Girshick. 2019. Phyre: A new benchmark for physical reasoning. Advances in Neural Information Processing Systems 32 (2019). Retrieved from https://arxiv.org/abs/1908.05656 · arXiv
- Fabien Baradel, Natalia Neverova, Julien Mille, Greg Mori, and Christian Wolf. 2019. CoPhy: Counterfactual learning of physical dynamics. In Proceedings of the International Conference on Learning Representations. Retrieved from https://arxiv.org/abs/1909.12000 · arXiv
- Daniel M. Bear, Elias Wang, Damian Mrowca, Felix Binder, Hsiao-Yu Fish Tung, R. T. Pramod, Cameron Holdaway, Sirui Tao, Kevin A. Smith, Fan-Yun Sun, Li Fei-Fei, Nancy Kanwisher, Joshua B. Tenenbaum, Daniel L. K. Yamins, and Judith E. Fan. 2021. Physion: Evaluating physical prediction from vision in humans and machines. Conference on Neural Information Processing Systems 35. Retrieved from https://openreview.net/forum?id=CXyZrKPz4CU
- 10.1609/aaai.v36i11.21496 · DOI
- 10.1609/aaai.v34i05.6239 · DOI · Paper page
- Yonatan Bitton, Nitzan Bitton Guetta, Ron Yosef, Yuval Elovici, Mohit Bansal, Gabriel Stanovsky, and Roy Schwartz. 2022. WinoGAViL: Gamified association benchmark to challenge vision-and-language models. Advances in Neural Information Processing Systems 35 (2022), 26549–26564. Retrieved from https://proceedings.neurips.cc/paper_files/paper/2022/hash/a96fe863f85c59789bba63588a9557b4-Abstract-Datasets_and_Benchmarks.html
- Rishi Bomassani et al. 2021. On the opportunities and risks of foundation models. arXiv:2105.07258. Retrieved from https://arxiv.org/abs/2108.07258 · arXiv
- 10.18653/v1/2020.emnlp-main.85 · DOI · Paper page
- T. L. Brown, H. E. LeMay, and B. Bursten. 2003. Chemistry: The Central Science (9th. ed.). Prentice Hall.
- Qingxing Cao Bailin Li Xiaodan Liang and Liang Lin. 2019. Explainable high-order visual question reasoning: A new benchmark and knowledge-routed network. arXiv:1909.10128. Retrieved from https://arxiv.org/abs/1909.10128 · arXiv
- 10.3758/BF03206384 · DOI · Paper page
- 10.18653/v1/W19-2008 · doi:10.18653/v1/w19-2008
- Matthew Crosby, Benjamin Beyret, Murray Shanahan, José Hernández-Orallo, Lucy Cheke, and Marta Halina. 2020. The Animal-AI testbed and competition. Neurips 2019 Competition and Demonstration Track 123 (2020), 164–176. Retrieved from http://proceedings.mlr.press/v123/crosby20a/crosby20a.pdf
- 10.1017/S1351324909990234 · DOI · Paper page
- Ernest Davis. 1990. Representations of Commonsense Knowledge. Morgan Kaufmann San Mateo Calif.
- 10.1080/13875868.2013.824976 · doi:10.1080/13875868.2013.824976
- 10.1609/aimag.v37i1.2637 · DOI
- Ernest Davis. 2021. Using human skills taxonomies and tests as measures of AI. In Proceedings of the AI and the Future of Skills, Volume 1: Capabilities and Assessments. Stuart Elliott (Ed.), OECD Publishing.
- Ernest Davis. To appear. Mathematics word problems common sense and artificial intelligence. Bulletin of the American Mathematical Society. To appear. Retrieved from https://arxiv.org/abs/2301.09723 · arXiv
- Ernest Davis and Gary Marcus. 2015. Communications of the ACM 58, 9 (2015), 92–105. Retrieved from http://cacm.acm.org/magazines/2015/9/191169-commonsense-reasoning-and-commonsense-knowledge-in-artificial-intelligence/
- Guy Deutscher. 2010. Through the Language Glass: Why the World Looks Different in Different Languages. Metropolitan Books, New York
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the NAACL-2019.
- Yan Ding Xiaohan Zhang Saeid Amiri Nieqing Cao Hao Yang Chad Esselink and Shiqi Zhang. 2022. Robot task planning and situation handling in open worlds. arXiv:2210.01287. Retrieved from https://arxiv.org/abs/2210.01287 · arXiv
- 10.18653/v1/2021.emnlp-main.670 · DOI
- Jonathan St BT Evans. 2017. Dual process theory: Perspectives and problems. In Proceedings of the Dual Process Theory 2.0. Wim de Neys (Ed.), Taylor and Francis. 137–155.
- 10.18653/v1/2020.emnlp-main.61 · DOI
- 10.1109/IROS51168.2021.9636830 · DOI
- Kanishk Gandhi, Gala Stojnic, Brenden M. Lake, and Moira R. Dillon. 2021. Baby Intuitions Benchmark (BIB): Discerning the goals, preferences, and actions of others. Advances in Neural Information Processing Systems 34 (2021), 9963–9976.
- Difei Gao, Ruiping Wang, Shiguang Shan, and Xilin Chen. 2022. CRIC: A VQA dataset for compositional reasoning on vision and commonsense. IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 5 (2022), 5561–5578. Retrieved from https://ieeexplore.ieee.org/abstract/document/9905976
- 10.18653/v1/P18-1086 · DOI
- Xavier Garcia, Yamini Bansal, Colin Cherry, George Foster, Maxim Krikun, Melvin Johnson, and Orhan Firat. 2023. The unreasonable effectiveness of few-shot learning for machine translation. In International Conference on Machine Learning, 10867–10878. Retrieved from https://proceedings.mlr.press/v202/garcia23a.html
- Prajwal Gatti, Abhirama Subramanyam Penamakuri, Revant Teotia, Anand Mishra, Shubhashis Sengupta, and Roshni Ramnani. 2022. COFAR: Commonsense and factual reasoning in image search. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing, 1185–1199. Retrieved from https://aclanthology.org/2022.aacl-main.87/
- Deepanway Ghosal, Pengfei Hong, Siqi Shen, Navonil Majumder, Rada Mihalcea, and Soujanya Poria. 2021. CIDER: Commonsense inference for dialogue explanation and reasoning. In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue. 301–313.
- Rohit Girdhar and Deva Ramanan. 2019. CATER: A diagnostic dataset for compositional actions & TEmporal reasoning. In International Conference on Learning Representations. Retrieved from https://openreview.net/forum?id=HJgzt2VKPB
- 10.1609/aaai.v30i1.9881 · DOI
- Andrew S. Gordon and Jerry R. Hobbs. 2017. A Formal Theory oF Commonsense Psychology: How People Think People Think. Cambridge University Press, Cambridge.
- 10.1109/CVPR46437.2021.01113 · DOI
- 10.1162/tacl_a_00469 · DOI
- 10.18653/v1/2023.eacl-main.30 · DOI
- 10.18653/v1/2021.emnlp-main.307 · DOI
- Dan Hendrycks Collin Burns Steven Basart Andrew Critch Jerry Li Dawn Song and Jacob Steinhardt. 2020. Aligning AI with shared human values. arXiv:2008.02275. Retrieved from https://arxiv.org/abs/2008.02275 · arXiv
- 10.1007/978-3-031-20059-5_32 · DOI
- Yining Hong, Li Yi, Josh Tenenbaum, Antonio Torralba, and Chuang Gan. 2021. Ptr: A benchmark for part-based conceptual, relational, and physical reasoning. Advances in Neural Information Processing Systems 34 (2021), 17427–17440. Retrieved from https://proceedings.neurips.cc/paper/2021/hash/918f5cd5a5c0d48671d4d4fc54bab2e9-Abstract.html
- 10.18653/v1/2020.acl-main.132 · DOI
- 10.18653/v1/D19-1243 · DOI · Paper page
- Jena D. Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosselut, and Yejin Choi. 2020. (COMET-)ATOMIC 2020: On symbolic and neural commonsense knowledge graphs. Proceedings of the AAAI Conference on Artificial Intelligence 35, 7 (2020), 6384–6392. Retrieved from https://arxiv.org/abs/2010.05953 · arXiv
- 10.1007/978-3-030-35288-2_24 · doi:10.1007/978-3-030-35288-2_24
- Nicos Isaak and Loizos Michael. 2020. Winventor: A machine-driven approach for the development of winograd schemas. ICAART (2), 26–35. Retrieved from https://www.scitepress.org/Papers/2020/89026/89026.pdf
- 10.1109/CVPR42600.2020.01025 · DOI · Paper page
- Minqi Jiang, Jelena Luketina, Nantas Nardelli, Pasquale Minervini, Philip H. S. Torr, Shimon Whiteson, and Tim Rocktäschel. 2020. WordCraft: An environment for benchmarking commonsense agents. In Language in Reinforcement Learning Workshop at International Conference on Machine Learning. Retrieved from https://openreview.net/forum?id=A0wHsDMUXAE
- 10.18653/v1/2020.findings-emnlp.445 · DOI · Paper page
- 10.1007/978-3-031-19842-7_21 · DOI
- 10.18653/v1/2022.naacl-main.66 · DOI
- Bernard Koch, Emily Denton, Alex Hanna, and Jacob G. Foster. 2021. Reduced, reused and recycled: The life of a dataset in machine learning research. In Proceedings of the 35th Conference on Neural Information Processing Systems. Retrieved from https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/file/3b8a614226a953a8cd9526fca6fe9ba5-Paper-round2.pdf
- Vid Kocijan Ernest Davis Thomas Lukasiewicz Gary Marcus and Leora Morgenstern. 2023. The defeat of the Winograd schema challenge. Artificial Intelligence in press. Retrieved from https://www.sciencedirect.com/science/article/pii/S0004370223001170 · DOI
- Michael Kosinski. 2023. Theory of mind may have spontaneously emerged in large language models. arXiv:2302.02083. Retrieved from https://arxiv.org/abs/2302.02083 · arXiv
- 10.18653/v1/2021.findings-acl.53 · DOI
- 10.18653/v1/2020.deelio-1.5 · DOI
- 10.1609/aimag.v6i4.510 · doi:10.1609/aimag.v6i4.510 · Paper page
- 10.1109/ICCV48922.2021.00205 · DOI
- Peng-Hsuan Li, Tsan-Yu Yang, and Wei-Yun Ma. 2020. CA-EHN: Commonsense analogy from E-HowNet. In Proceedings of the 12th Language Resources and Evaluation Conference. 2984–2990. Retrieved from https://aclanthology.org/2020.lrec-1.365/
- 10.18653/v1/2020.emnlp-main.557 · DOI · Paper page
- 10.18653/v1/2021.acl-long.102 · DOI
- Bill Yuchen Lin Ziyi Wu Yichi Yang Dong-Ho Lee and Xiang Ren. 2021. RiddleSense: Reasoning about riddle questions featuring linguistic creativity and commonsense knowledge. In Findings of the Association for Computational Linguistics: (ACL-IJCNLP’21) 1504–1515. Retrieved from https://aclanthology.org/2021.findings-acl.131.pdf
- Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common objects in context. In Proceedings of the European Conference on Computer Vision. 740–755.
- Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. 2021. IconQA: A new benchmark for abstract diagram understanding and visual language reasoning. In 35th Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). Retrieved from https://openreview.net/forum?id=uXa9oBDZ9V1
- Gary Marcus and Ernest Davis. 2023. How not to test GPT. Communications of the ACM Blog. Retrieved from https://cacm.acm.org/blogs/blog-cacm/270142-how-not-to-test-gpt-3/fulltext
- John McWhorter. 2014. The Language Hoax: Why The World Looks the Same in Any Language. Oxford University Press, Oxford.
- 10.18653/v1/D18-1260 · DOI · Paper page
- Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, and Ashwin Kalyan. 2022. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP). 5807–5832. Retrieved from https://aclanthology.org/2022.emnlp-main.392/
- 10.1016/B978-012369388-4/50072-2 · DOI
- Kai Nakamura, Sharon Levy, and William Yang Wang. 2020. Fakeddit: A new multimodal benchmark dataset for fine-grained fake news detection. In Proceedings of the 12th Language Resources and Evaluation Conference. 6149–6157. Retrieved from https://aclanthology.org/2020.lrec-1.755/
- Arvind Narayanan and Sayash Kapoor. 2023. Is GPT-4 getting worse over time? Retrieved from https://www.aisnakeoil.com/p/is-gpt-4-getting-worse-over-time. Accessed October 5, 2023.
- Qiang Ning, Hao Wu, Rujun Han, Nanyun Peng, Matt Gardner, and Dan Roth. 2020. TORQUE: A reading comprehension dataset of temporal ordering questions. arXiv:2005.00242. Retrieved from https://arxiv.org/abs/2005.00242 · arXiv
- Yasumasa Onoe, Michael J. Q. Zhang, Eunsol Choi, and Greg Durrett. 2021. CREAK: A dataset for commonsense reasoning over entity knowledge. arXiv:2109.01653. Retrieved from https://arxiv.org/abs/2109.01653 · arXiv
- Simon Ostermann, Ashutosh Modi, Michael Roth, Stefan Thater, Manfred Pinkal. 2018. MCScript: A novel dataset for assessing machine comprehension using script knowledge. In Proceedings of the 11th International Conference on Language Resources and Evaluation (LREC’18). 3567–3574. Retrieved from https://aclanthology.org/L18-1564.pdf
- 10.18653/v1/2021.conll-1.28 · DOI
- Amandalynne Paullada, Inioluwa Deborah Raji, Emily M. Bender, Emily Denton, and Alex Hanna. 2021. Data and its (dis)contents: A survey of dataset development and use in machine learning research. Patterns 2, 11 (2021). https://www.cell.com/patterns/pdf/S2666-3899(21)00184-7.pdf
- Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning, 90–95. Retrieved from https://people.ict.usc.edu/gordon/public_html/publications/AAAI-SPRING11A.PDF
- 10.1007/s11263-016-0987-1 · DOI · Paper page
- Stuart Russell and Peter Norvig. 2020. Artificial Intelligence: A Modern Approach (4th. ed.). Prentice Hall.
- 10.18653/v1/2020.findings-emnlp.418 · DOI
- Henrique Santos Ke Shen Alice M. Mulvehill Yasaman Razeghi Deborah L. McGuinness and Mayank Kejriwal. 2022. A Theoretically Grounded Benchmark for Evaluating Machine Commonsense. arXiv:2203.12184. Retrieved from https://arxiv.org/abs/2203.12184 · arXiv
- Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019. SocialIQA: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. 4463–4473. Retrieved from https://aclanthology.org/D19-1454/
- 10.18653/v1/2021.acl-short.85 · DOI · Paper page
- Jaehyung Seo, Chanjun Park, Hyeonseok Moon, Sugyeong Eo, Myunghoon Kang, Seounghoon Lee, and Heuiseok Lim. 2021. KommonGen: A dataset for Korean generative commonsense reasoning evaluation. In Proceedings of the Annual Conference on Human and Language Technology. 55–60. Retrieved from https://koreascience.kr/article/CFKO202130060697830.pdf
- Tianmin Shu, Abhishek Bhandwaldar, Chuang Gan, Kevin Smith, Shari Liu, Dan Gutfreund, Elizabeth Spelke, Joshua Tenenbaum, and Tomer Ullman. 2021. AGENT: A benchmark for core psychological reasoning. In Proceedings of the International Conference on Machine Learning. 9614–9625. Retrieved from https://research.ibm.com/publications/agent-a-benchmark-for-core-psychological-reasoning
- 10.1007/978-3-319-46448-0_31 · DOI · Paper page
- Ashish Singh, Hang Su, SouYoung Jin, Huaizu Jiang, Chetan Manjesh, Geng Luo, Ziwei He, Li Hong, Erik Learned-Miller, and Rosie Cowell. 2019. Half&Half: New tasks and benchmarks for studying visual common sense. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 1–4.
- Shikhar Singh Nuan Wen Yu Hou Pegah Alipoormolabashi Te-Lin Wu Xuezhe Ma and Nanyun Peng. 2021. COM2SENSE: A commonsense reasoning benchmark with complementary sentences. In Findings of the Association for Computational Linguistics: (ACL-IJCNLP’21). 883–898. Retrieved from https://aclanthology.org/2021.findings-acl.78/ · DOI
- 10.1609/aaai.v31i1.11164 · DOI · Paper page
- Elizabeth Spelke. 2022. What Babies Know: Core Knowledge and Composition. Volume 1. Oxford University Press, Oxford.
- Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, et al. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research. Retrieved from https://openreview.net/forum?id=uyTL5Bvosj
- Shane Storks Qiaozi Gao and Joyce Chai. 2019. Commonsense reasoning for natural language understanding: A survey of benchmarks resources and approaches. arXiv:1904.01172. https://arxiv.org/abs/1904.01172 · arXiv
- 10.18653/v1/2021.findings-emnlp.422 · DOI
- Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4149–4158. Retrieved from https://aclanthology.org/N19-1421/
- Alon Talmor, Ori Yoran, Ronan Le Bras, Chandra Bhagavatula, Yoav Goldberg, Yejin Choi, and Jonathan Berant. 2021. CommonsenseQA 2.0: Exposing the Limits of AI through Gamification. In 35th Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1). Retrieved from https://openreview.net/forum?id=qF7FlUT5dxa
- 10.1145/2812802 · DOI · Paper page
- Frank van Harmelen, Vladimir Lifschitz, and Bruce Porter (eds). 2008. Handbook of Knowledge Representation. Elsevier, Amsterdam.
- Veniamin Veselovsky, Manoel Horta Ribeiro, and Robert West. 2023. Artificial artificial artificial intelligence: Crowd workers widely use large language models for text production tasks. arXiv:2306.07899. Retrieved from https://arxiv.org/abs/2306.07899 · arXiv
- 10.1068/p5262 · DOI · Paper page
- 10.5555/3171642.3171825 · doi:10.5555/3171642.3171825
- 10.1109/TPAMI.2017.2754246 · DOI · Paper page
- 10.18653/v1/N18-2049 · DOI
- Luca Weihs, Amanda Rose Yuile, Renée Baillargeon, Cynthia L. Fisher, Gary Marcus, Roozbeh Mottaghi, and Aniruddha Kembhavi. 2022. Benchmarking progress to infant-Level physical reasoning in AI. Transactions on Machine Learning.
- Adina, Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through Inference. In Proceedings of NAACL-HLT, 1112–1122. Retrieved from https://aclanthology.org/N18-1101/
- 10.18653/v1/P18-2016 · DOI
- 10.18653/v1/2020.coling-main.419 · DOI · Paper page
- Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum. 2019. Clevrer: Collision events for video representation and reasoning. In International Conference on Learning Representations. Retrieved from https://openreview.net/forum?id=HkxYzANYDB
- 10.18653/v1/2021.emnlp-main.162 · DOI
- 10.18653/v1/2022.emnlp-main.132 · DOI
- Haoxuan You Rui Sun Zhecan Wang Kai-Wei Chang and Shih-Fu Chang. 2022. Find someone who: Visual commonsense understanding in human-centric grounding. In Findings of the Association for Computational Linguistics: EMNLP. 5444–5454. Retrieved from https://aclanthology.org/2022.findings-emnlp.399/ · DOI
- 10.1007/978-3-031-19836-6_17 · DOI
- Mo Yu Xiaoxiao Guo Yufei Feng Xiaodan Zhu Michael Greenspan and Murray Campbell. 2020. Deriving commonsense inference tasks from interactive fictions. arXiv:2010.09788. Retrieved from https://arxiv.org/abs/2010.09788 · arXiv
- Ales Zagar, Marko Robnik-Sikonja, Teja Goli, and Spela Arhar Holdt. 2020. Slovene translation of SuperGLUE, Slovenian language resource repository CLARIN.SI, ISSN 2820–4042. Retrieved from http://hdl.handle.net/11356/1380. Accessed October 5, 2023.
- 10.18653/v1/D18-1009 · DOI · Paper page
- 10.18653/v1/2020.acl-main.508 · DOI
- 10.1162/tacl_a_00068 · DOI · Paper page
- Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. Record: Bridging the gap between human and machine commonsense reading comprehension. arXiv:1810.12885. Retrieved from https://arxiv.org/abs/1810.12885 · arXiv
- 10.18653/v1/D19-1332 · DOI · Paper page
- 10.18653/v1/2021.naacl-main.107 · DOI
Cited by
Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.ijcnlp-main.45 · Cited by 682
“…robust complex reasoning capabilities, which should also be facilitated by the creation of more comprehensive benchmarks for assessing these abilities, such as works by Laskar et al. (2023b); Qin et al. (2023); Davis (2023a), particularly when multiple abilities are required to complete the tasks.”
BackgroundA Categorical Archive of ChatGPT Failures
Borji, A. (2023) · arXiv · doi:10.48550/arxiv.2302.03494 · Cited by 514
A Categorical Archive of ChatGPT Failures
Borji, A. (2023) · doi:10.21203/rs.3.rs-2895792/v1 · Cited by 514
Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.04023 · Cited by 337
“…robust complex reasoning capabilities, which should also be facilitated by the creation of more comprehensive benchmarks for assessing these abilities, such as works by Laskar et al. (2023b); Qin et al. (2023); Davis (2023a), particularly when multiple abilities are required to complete the tasks.”
BackgroundInadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence
McIntosh, T. R., Susnjak, T., Arachchilage, N., Liu, T., Watters, P., & Halgamuge, M. N. (2024) · arXiv · doi:10.48550/arxiv.2402.09880 · Cited by 104
Metadata
- Authors
- Davis, E.
- Venue
- ACM Computing Surveys
- Published
- 2023-10-23
- Volume
- 56
- Issue
- 4
- Pages
- 1-41
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W4386607580
- Cited by
- 76
- References
- 35
- FWCI
- 6.59
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Benchmarks for Automated Commonsense Reasoning: A Survey. https://app.cochat.ai/papers/10.1145/3615355
- The paper
- Davis, E. (2024). Benchmarks for Automated Commonsense Reasoning: A Survey. ACM Computing Surveys. https://doi.org/10.1145/3615355
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.