CoChat
Sign Up
PreprintCited by 4

The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domain

Moskvichev, Arseny, Odouard, Victor Vikram, Mitchell, Melanie·2023·arXiv
DOI 10.48550/arxiv.2305.07141arXiv 2305.07141
Open this paper in CoChat — free account

Abstract

The abilities to form and abstract concepts is key to human intelligence, but such abilities remain lacking in state-of-the-art AI systems. There has been substantial research on conceptual abstraction in AI, particularly using idealized domains such as Raven's Progressive Matrices and Bongard problems, but even when AI systems succeed on such problems, the systems are rarely evaluated in depth to see if they have actually grasped the concepts they are meant to capture. In this paper we describe an in-depth evaluation benchmark for the Abstraction and Reasoning Corpus (ARC), a collection of few-shot abstraction and analogy problems developed by Chollet [2019]. In particular, we describe ConceptARC, a new, publicly available benchmark in the ARC domain that systematically assesses abstraction and generalization abilities on a number of basic spatial and semantic concepts. ConceptARC differs from the original ARC dataset in that it is specifically organized around "concept groups" -- sets of problems that focus on specific concepts and that are vary in complexity and level of abstraction. We report results on testing humans on this benchmark as well as three machine solvers: the top two programs from a 2021 ARC competition and OpenAI's GPT-4. Our results show that humans substantially outperform the machine solvers on this benchmark, showing abilities to abstract and generalize concepts that are not yet captured by AI systems. We believe that this benchmark will spur improvements in the development of AI systems for conceptual abstraction and in the effective evaluation of such systems.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2305.07141

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Recent studies highlight a number of shortcomings of frontier multimodal models.

    “Recent studies highlight a number of shortcomings of frontier multimodal models (Moskvichev et al., 2023; Tong et al., 2024; Chen et al., 2024a; Fu et al., 2024b).”

    From Does Spatial Cognition Emerge in Frontier Models? (2024) · cites this paper as Moskvichev et al., 2023

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 29

  1. S. Acquaviva, Y. Pu, M. Kryven, T. Sechopoulos, C. Wong, G. Ecanow, M. Nye, M. Tessler, and J. Tenenbaum. Communicating natural programs to humans and machines. Advances in Neural Information Processing Systems, 35:3731-3743, 2022.
  2. J. Ainooson, D. Sanyal, J. P. Michelson, Y. Yang, and M. Kunda. An approach for solving tasks on the Abstract Reasoning Corpus, 2023. arXiv:2302.09425. · arXiv
  3. S. Alford, A. Gandhi, A. Rangamani, A. Banburski, T. Wang, S. Dandekar, J. Chin, T. Poggio, and P. Chin. Neural-guided, bidirectional program search for abstraction and reasoning. In Proceedings of the Tenth International Conference on Complex Networks and Their Applica- tions, pages 657-668. Springer, 2022.
  4. R. Assouel, P. Rodriguez, P. Taslakian, D. Vazquez, and Y. Bengio. Object-centric compo- sitional imagination for visual abstract reasoning. In ICLR Workshop on the Elements of Reasoning: Objects, Structure and Causality, 2022.
  5. S. Carey. The Origin of Concepts. MIT Press, Cambridge, MA, 2011.
  6. P. A. Carpenter, M. A. Just, and P. Shell. What one intelligence test measures: A theoretical account of the processing in the Raven progressive matrices test. Psychological Review, 97(3): 404-431, 1990.
  7. F. Chollet. On the measure of intelligence, 2019. arXiv:1911.01547. · arXiv
  8. K. Ellis, C. Wong, M. Nye, M. Sablé-Meyer, L. Cary, L. Morales, L. Hewitt, A. Solar-Lezama, and J. B. Tenenbaum. DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning, 2020. arXiv:2006.08381. · arXiv
  9. S. Ferré. First steps of an approach to the ARC challenge based on descriptive grid models and the minimum description length principle, 2021. arXiv:2112.00848. · arXiv
  10. R. Fischer, M. Jakobs, S. Mücke, and K. Morik. Solving abstract reasoning tasks with gram- matical evolution. In Proceedings of the Conference Lernen, Wissen, Daten, Analysen, LWDA, pages 6-10, 2020.
  11. R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wich- mann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665-673, 2020.
  12. D. R. Hofstadter and M. Mitchell. The Copycat project: A model of mental fluidity and analogy-making. In K. J. Holyoak and J. A. Barnden, editors, Advances in Connectionist and Neural Computation Theory, volume 2, pages 31-112. Ablex Publishing Corporation, 1994.
  13. A. Johnson, W. K. Vong, B. M. Lake, and T. M. Gureckis. Fast and flexible: Human program induction in abstract reasoning tasks, 2021. arXiv:2103.05823. · arXiv
  14. S. Kim, P. Phunyaphibarn, D. Ahn, and S. Kim. Playgrounds for abstraction and reasoning. In NeurIPS Workshop on Neuro, Causal, and Symbolic AI (nCSI), 2022.
  15. S. Lapuschkin, S. Wäldchen, A. Binder, G. Montavon, W. Samek, and K.-R. Müller. Unmask- ing Clever Hans predictors and assessing what machines really learn. Nature Communications, 10(1):1096, 2019.
  16. A. Lovett and K. D. Forbus. Modeling visual problem solving as analogical reasoning. Psy- chological Review, 124(1), 2017.
  17. M. Ma lkiński and J. Mańdziuk. Deep learning methods for abstract visual reasoning: A survey on Raven's Progressive Matrices, 2022. arXiv:2201.12382. · arXiv
  18. M. Ma lkiński and J. Mańdziuk. A review of emerging research directions in abstract visual reasoning. Information Fusion, 2022.
  19. M. Mitchell. Abstraction and analogy-making in artificial intelligence. Annals of the New York Academy of Sciences, 1505:79-101, 2021.
  20. M. Mitchell and D. C. Krakauer. The debate over understanding in AI's large language models. Proceedings of the National Academy of Sciences, 120(13):e2215907120, 2023.
  21. W. Nie, Z. Yu, L. Mao, A. B. Patel, Y. Zhu, and A. Anandkumar. Bongard-LOGO: A new benchmark for human-level concept learning and reasoning. Advances in Neural Information Processing Systems, 33:16468-16480, 2020.
  22. V. V. Odouard and M. Mitchell. Evaluating understanding on conceptual abstraction bench- marks, 2022. arXiv:2206.14187. · arXiv
  23. OpenAI. GPT-4 Technical Report, 2023. arXiv:2303.08774. · arXiv
  24. A. Sonwane, S. Chitlangia, T. Dash, L. Vig, G. Shroff, and A. Srinivasan. Using program synthesis and inductive logic programming to solve Bongard problems, 2021. arXiv:2110.09947. · arXiv
  25. E. S. Spelke and K. D. Kinzler. Core knowledge. Developmental Science, 10(1):89-96, 2007.
  26. S. Ullman. Visual routines. In Readings in Computer Vision, pages 298-328. Elsevier, 1987.
  27. T. Webb, K. J. Holyoak, and H. Lu. Emergent analogical reasoning in large language models, 2022. arXiv:2212.09196. · arXiv
  28. Y. Xu, E. B. Khalil, and S. Sanner. Graphs, constraints, and search for the Abstraction and Reasoning Corpus, 2022. arXiv:2210.09880. · arXiv
  29. C. Zhang, F. Gao, B. Jia, Y. Zhu, and S.-C. Zhu. RAVEN: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pages 5312-5322, 2019.

Cited by

Sorted by citations

Authors
Moskvichev, A., Odouard, V. V., & Mitchell, M.
Venue
arXiv
Publisher
arXiv
Published
2023
DOI
10.48550/arxiv.2305.07141
arXiv
2305.07141
References
0
Type
Preprint
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review