Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
PreprintCited by 1

The DOI matches its DataCite record.

Systematic human learning and generalization from a brief tutorial with explanatory feedback

Nam, Andrew J., McClelland, James L.·2021·arXiv
DOI 10.48550/arxiv.2107.06994arXiv 2107.06994
Open this paper in CoChat — free account

Abstract

Neural networks have long been used to model human intelligence, capturing elements of behavior and cognition, and their neural basis. Recent advancements in deep learning have enabled neural network models to reach and even surpass human levels of intelligence in many respects, yet unlike humans, their ability to learn new tasks quickly remains a challenge. People can reason not only in familiar domains, but can also rapidly learn to reason through novel problems and situations, raising the question of how well modern neural network models capture human intelligence and in which ways they diverge. In this work, we explore this gap by investigating human adults' ability to learn an abstract reasoning task based on Sudoku from a brief instructional tutorial with explanatory feedback for incorrect responses using a narrow range of training examples. We find that participants who master the task do so within a small number of trials and generalize well to puzzles outside of the training range. We also find that most of those who master the task can describe a valid solution strategy, and such participants perform better on transfer puzzles than those whose strategy descriptions are vague or incomplete. Interestingly, fewer than half of our human participants were successful in acquiring a valid solution strategy, and this ability is associated with high school mathematics education. We consider the challenges these findings pose for building computational models that capture all aspects of our findings and point toward a possible role for learning to engage in explanation-based reasoning to support rapid learning and generalization.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2107.06994

Cites

Statistics

Figures

Figure 1 Examples of hidden single puzzles. Puzzles were generated using the participant ' s individualized, random assignment of the training house type, house index, cell index, and digit set. (A) The full hidden single puzzle configuration as the participant might see it part way through the tutorial, conforming to the house type, house index, cell index, and containing the target (1) and the distractor (7) digits, both drawn from the training digit set. Examples used in the practice phase and control examples from the test phase would all use the same house type, house index, cell index, and training digit set, in this case the digits (1, 3, 5, 7). (B -F) Examples of non-control puzzles used in the test phase containing changes in features from the puzzles used in the tutorial and practice phase. (B -E) Puzzles with exactly one changed feature. (F) Puzzle with all four features changed.
Figure 1 Examples of hidden single puzzles. Puzzles were generated using the participant ' s individualized, random assignment of the training house type, house index, cell index, and digit set. (A) The full hidden single puzzle configuration as the participant might see it part way through the tutorial, conforming to the house type, house index, cell index, and containing the target (1) and the distractor (7) digits, both drawn from the training digit set. Examples used in the practice phase and control examples from the test phase would all use the same house type, house index, cell index, and training digit set, in this case the digits (1, 3, 5, 7). (B -F) Examples of non-control puzzles used in the test phase containing changes in features from the puzzles used in the tutorial and practice phase. (B -E) Puzzles with exactly one changed feature. (F) Puzzle with all four features changed.CC-BY · From the PDF
Figure 2 (A) The tutorial puzzle randomly generated for one participant. (B) The contradiction exercise. The goal is to select the two cells containing 7s. (C) The Full House exercise. The goal is to fill in the green cell with 1. (D) The tutorial puzzle with distractors removed. The goal is to enter 1 in the green cell. (E -G) Positive example tasks. The goal is to select the 1s that intersect with the purple cells. (H -J) Negative example tasks. The goal is to select the blue cells that intersect with the purple 7s. (K -L) Summary grids for both positive and negative examples.
Figure 2 (A) The tutorial puzzle randomly generated for one participant. (B) The contradiction exercise. The goal is to select the two cells containing 7s. (C) The Full House exercise. The goal is to fill in the green cell with 1. (D) The tutorial puzzle with distractors removed. The goal is to enter 1 in the green cell. (E -G) Positive example tasks. The goal is to select the 1s that intersect with the purple cells. (H -J) Negative example tasks. The goal is to select the blue cells that intersect with the purple 7s. (K -L) Summary grids for both positive and negative examples.CC-BY · From the PDF
Figure 3 Sample feedback sequences during practice phase. (A) Feedback for submitting 7, an in-house response. (B) Feedback for submitting 4, an absent response. (C) Feedback for submitting 8, a distractor response. (D) Feedback for a correcting a previously incorrect response.
Figure 3 Sample feedback sequences during practice phase. (A) Feedback for submitting 7, an in-house response. (B) Feedback for submitting 4, an absent response. (C) Feedback for submitting 8, a distractor response. (D) Feedback for a correcting a previously incorrect response.CC-BY · From the PDF
Figure 4 Overall accuracy during the practice and test phases. Each point represents n = 1 to 5 solvers or non-solvers with identical scores. Marginal histograms show the proportion of individuals in each group achieving each level of accuracy in each phase.
Figure 4 Overall accuracy during the practice and test phases. Each point represents n = 1 to 5 solvers or non-solvers with identical scores. Marginal histograms show the proportion of individuals in each group achieving each level of accuracy in each phase.CC-BY · From the PDF
Figure 5 Response types and inferred strategies during the practice phase for participants identified as solvers. (A) Actual frequency of incorrect response types. (B) Inferred distribution of strategies by the aggregate model. (C) Predicted distribution of incorrect response types by the aggregate model. (D) Proportion of solvers inferred by individually fitted models to be using a strategy at least as effective as the strategy indicated.
Figure 5 Response types and inferred strategies during the practice phase for participants identified as solvers. (A) Actual frequency of incorrect response types. (B) Inferred distribution of strategies by the aggregate model. (C) Predicted distribution of incorrect response types by the aggregate model. (D) Proportion of solvers inferred by individually fitted models to be using a strategy at least as effective as the strategy indicated.CC-BY · From the PDF
Image 6
CC-BY · From the PDF
Figure 7 Response types and inferred strategies during the practice phase for participants identified as non-solver. (A) Actual frequency of incorrect response types. (B) Inferred distribution of strategies by the aggregate model. (C) Predicted distribution of incorrect response types by the aggregate model. (D) Proportion of non-solvers inferred by individually fitted models to be using a strategy at least as effective as the strategy indicated.
Figure 7 Response types and inferred strategies during the practice phase for participants identified as non-solver. (A) Actual frequency of incorrect response types. (B) Inferred distribution of strategies by the aggregate model. (C) Predicted distribution of incorrect response types by the aggregate model. (D) Proportion of non-solvers inferred by individually fitted models to be using a strategy at least as effective as the strategy indicated.CC-BY · From the PDF
Image 8
CC-BY · From the PDF
Image 9
CC-BY · From the PDF
Image 10
CC-BY · From the PDF
Image 11
CC-BY · From the PDF
Image 12
CC-BY · From the PDF
Image 13
CC-BY · From the PDF
Figure 9 Ratings of self-reported strategies between persistent solvers and PD guessers.
Figure 9 Ratings of self-reported strategies between persistent solvers and PD guessers.CC-BY · From the PDF
Figure 10 Overall accuracy across both the practice and test phases of the experiment by highest education. Color represents whether the participant has had formal education in both, one, or neither of algebra and geometry. Darker, bordered points represent group means and lighter points represent individual participants. All error bars show 95% bootstrapped confidence intervals (groups with N < 2 excluded). Means and intervals computed using log-odds of accuracy scores.
Figure 10 Overall accuracy across both the practice and test phases of the experiment by highest education. Color represents whether the participant has had formal education in both, one, or neither of algebra and geometry. Darker, bordered points represent group means and lighter points represent individual participants. All error bars show 95% bootstrapped confidence intervals (groups with N < 2 excluded). Means and intervals computed using log-odds of accuracy scores.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 88

  1. Abdool, M., Nam, A. J., & McClelland, J. L. (2023). Continual learning and out of distribution generalization in a systematic reasoning task. In MATH-AI: The 3rd Workshop on Mathemat- ical Reasoning and AI at NeurIPS'23. https://nips.cc/virtual /2023/74990
  2. Ahn, W.-K., Brewer, W. F., & Mooney, R. J. (1992). Schema acqui- sition from a single example. Journal of Experimental Psychology: Learning, Memory, and Cognition, 18(2), 391-412. https://doi .org/10.1037/0278-7393.18.2.391 · DOI
  3. Ashby, F. G., Alfonso-Reese, L. A., Turken, A. U., & Waldron, E. M. (1998). A neuropsychological theory of multiple systems in category learning. Psychological Review, 105(3), 442-481. https://doi.org/10.1037/0033-295X.105.3.442, PubMed: 9697427 · DOI
  4. Baker, D. P., Eslinger, P. J., Benavides, M., Peters, E., Dieckmann, N. F., & Leon, J. (2015). The cognitive impact of the education revolution: A possible cause of the Flynn effect on population IQ. Intelligence, 49, 144-158. https://doi.org/10.1016/j.intell.2015 .01.003 · DOI
  5. Berglund, L., Tong, M., Kaufmann, M., Balesni, M., Stickland, A. C., Korbak, T., & Evans, O. (2023). The reversal curse: LLMs trained on "A is B" fail to learn "B is A." arXiv:2309.12288. https://doi .org/10.48550/arXiv.2309.12288 · DOI · Paper page
  6. Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., Brynjolfsson, E., Buch, S., Card, D., Castellon, R., Chatterji, N., Chen, A., Creel, K., Davis, J. Q., Demszky, D., …, Liang, P. (2021). On the opportunities and risks of foundation models. arXiv:2108.07258. https://doi.org/10.48550/arXiv.2108.07258 · DOI · Paper page
  7. Brophy, C., & Hahn, L. (2014). Engaging students in a large lecture: An experiment using sudoku puzzles. Journal of Statistics Education, 22(1). https://doi.org/10.1080/10691898.2014.11889690 · DOI
  8. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., … Amodei, D. (2020). Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems (NIPS'20) (pp. 1877-1901).
  9. Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., Nori, H., Palangi, H., Tulio Ribeiro, M., & Zhang, Y. (2023). Sparks of arti- ficial general intelligence: Early experiments with GPT-4. arXiv:2303.12712. https://doi.org/10.48550/arXiv.2303.12712 · DOI · Paper page
  10. Burger, W. F., & Shaughnessy, J. M. (1986). Characterizing the van Hiele levels of development in geometry. Journal for Research in Mathematics Education, 17(1), 31-48. https://doi.org/10.5951 /jresematheduc.17.1.0031 · DOI
  11. Bürkner, P.-C. (2017). brms: An R package for Bayesian multi- level models using Stan. Journal of Statistical Software, 80(1), 1-28. https://doi.org/10.18637/jss.v080.i01 · DOI · Paper page
  12. Camburu, O.-M., Rocktäschel, T., Lukasiewicz, T., & Blunsom, P. (2018). e-SNLI: Natural language inference with natural language explanations. In Proceedings of the 32nd International Confer- ence on Neural Information Processing Systems (NIPS'18) (pp. 9560-9572).
  13. Chan, S. C. Y., Santoro, A., Lampinen, A. K., Wang, J. X., Singh, A., Richemond, P. H., McClelland, J., & Hill, F. (2022). Data distribu- tional properties drive emergent in-context learning in transformers. arXiv:2205.05055. https://doi.org/10.48550/arXiv.2205.05055 · DOI · Paper page
  14. Chi, M. T. H., Bassok, M., Lewis, M. W., Reimann, P., & Glaser, R. (1989). Self-explanations: How students study and use examples in learning to solve problems. Cognitive Science, 13(2), 145-182. https://doi.org/10.1207/s15516709cog1302_1 · DOI · Paper page
  15. Chi, M. T. H., De Leeuw, N., Chiu, M.-H., & Lavancher, C. (1994). Eli- citing self-explanations improves understanding. Cognitive Science, 18(3), 439-477. https://doi.org/10.1207/s15516709cog1803_3 · DOI · Paper page
  16. Chomsky, N. (1957). Syntactic structures. De Gruyter Mouton. https://doi.org/10.1515/9783112316009 · DOI · Paper page
  17. Ciregan, D., Meier, U., & Schmidhuber, J. (2012). Multi-column deep neural networks for image classification. In Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR'12) (pp. 3642-3649). IEEE. https://doi.org/10 .1109/CVPR.2012.6248110 · DOI
  18. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., & Schulman, J. (2021). Training verifiers to solve math word problems. arXiv:2110.14168. https://doi.org/10.48550/arXiv.2110.14168 · DOI · Paper page
  19. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46. https://doi.org/10.1177/001316446002000104 · DOI · Paper page
  20. DeJong, G., & Mooney, R. (1986). Explanation-based learning: An alternative view. Machine Learning, 1(2), 145-176. https://doi .org/10.1007/BF00114116 · DOI
  21. Duncan, J., Assem, M., & Shashidhara, S. (2020). Integrated intelligence from distributed brain activity. Trends in Cognitive Sciences, 24(10), 838-852. https://doi.org/10.1016/j.tics.2020 .06.012, PubMed: 32771330 · DOI
  22. Ericsson, K. A. (2017). Protocol analysis. In W. Bechtel & G. Graham (Eds.), A companion to cognitive science (pp. 425-432). Basil Blackwell. https://doi.org/10.1002/9781405164535. · DOI
  23. ch33 · DOI
  24. Ericsson, K. A., & Simon, H. A. (1980). Verbal reports as data. Psy- chological Review, 87(3), 215-251. https://doi.org/10.1037/0033 -295X.87.3.215 · DOI
  25. Falkenhainer, B., Forbus, K. D., & Gentner, D. (1989). The structure-mapping engine: Algorithm and examples. Artificial Intelligence, 41(1), 1-63. https://doi.org/10.1016/0004 -3702(89)90077-5 · DOI · Paper page
  26. Felgenhauer, B., & Jarvis, F. (2006). Mathematics of sudoku I. Math- ematical Spectrum, 39(1), 15-22.
  27. Flynn, J. R. (1984). The mean IQ of Americans: Massive gains 1932 to 1978. Psychological Bulletin, 95(1), 29-51. https://doi.org/10 .1037/0033-2909.95.1.29 · DOI · Paper page
  28. Fodor, J. A. (1975). The language of thought. Harvard University Press.
  29. Fodor, J. A., & Pylyshyn, Z. W. (1988). Connectionism and cogni- tive architecture: A critical analysis. Cognition, 28(1-2), 3-71. https://doi.org/10.1016/0010-0277(88)90031-5, PubMed: 2450716 · DOI · Paper page
  30. Frank, M. C., Slemmer, J. A., Marcus, G. F., & Johnson, S. P. (2009). Information from multiple modalities helps 5-month-olds learn abstract rules. Developmental Science, 12(4), 504-509. https:// doi.org/10.1111/j.1467-7687.2008.00794.x, PubMed: 19635078 · DOI
  31. Geiger, A., Carstensen, A., Frank, M. C., & Potts, C. (2023). Rela- tional reasoning and generalization using nonsymbolic neural networks. Psychological Review, 130(2), 308-333. https://doi .org/10.1037/rev0000371, PubMed: 35834185 · DOI
  32. Gick, M. L., & Holyoak, K. J. (1980). Analogical problem solving. Cognitive Psychology, 12(3), 306-355. https://doi.org/10.1016 /0010-0285(80)90013-4 · DOI · Paper page
  33. Gray, J. R., & Thompson, P. M. (2004). Neurobiology of intelli- gence: Science and ethics. Nature Reviews Neuroscience, 5(6), 471-482. https://doi.org/10.1038/nrn1405, PubMed: 15152197 · DOI · Paper page
  34. Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., & Steinhardt, J. (2021). Measuring mathematical problem solving with the MATH dataset. arXiv:2103.03874. https://doi.org/10.48550/arXiv.2103.03874 · DOI · Paper page
  35. Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimi- zation. arXiv:1412.6980. https://doi.org/10.48550/arXiv.1412.6980 · DOI · Paper page
  36. Lake, B. M., & Baroni, M. (2023). Human-like systematic general- ization through a meta-learning neural network. Nature, 623(7985), 115-121. https://doi.org/10.1038/s41586-023-06668-3, PubMed: 37880371 · DOI · Paper page
  37. Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. (2017). Building machines that learn and think like people. Behavioral and Brain Sciences, 40, Article e253. https://doi.org /10.1017/S0140525X16001837, PubMed: 27881212 · DOI · Paper page
  38. Lampinen, A. K., Dasgupta, I., Chan, S. C. Y, Matthewson, K., Tessler, M. H., Creswell, A., McClelland, J. L., Wang, J. X., & Hill, F. (2022). Can language models learn from explanations in context? In Y. Goldberg, Z. Kozareva, & Y. Zhang (Eds.), Findings of the Association for Computational Linguistics: EMNLP 2022 (pp. 537-563). Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.findings-emnlp.38 · DOI · Paper page
  39. Lampinen, A. K., Roy, N. A., Dasgupta, I., Chan, S. C. Y., Tam, A. C., McClelland, J. L., Yan, C., Santoro, A., Rabinowitz, N. C., Wang, J. X., & Hill, F. (2022). Tell me why! Explanations support learning relational and causal structure. In Proceedings of the 39th International Conference on Machine Learning (pp. 11868-11890).
  40. PMLR.
  41. Lee, N. Y. L., Goodwin, G. P., & Johnson-Laird, P. N. (2008). The psychological puzzle of sudoku. Thinking & Reasoning, 14(4), 342-364. https://doi.org/10.1080/13546780802236308 · DOI
  42. Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., Wu, Y., Neyshabur, B., Gur-Ari, G., & Misra, V. (2022). Solving quantitative reasoning problems with language models. arXiv:2206.14858. https://doi.org/10.48550/arXiv.2206.14858 · DOI · Paper page
  43. Lombrozo, T. (2006). The structure and function of explanations. Trends in Cognitive Sciences, 10(10), 464-470. https://doi.org /10.1016/j.tics.2006.08.004, PubMed: 16942895 · DOI
  44. Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., & Kalyan, A. (2022). Learn to explain: Multimodal reasoning via thought chains for science question answering. arXiv:2209.09513. https://doi.org/10.48550/arXiv.2209.09513 · DOI
  45. Marcus, G. F. (2003). The algebraic mind: Integrating connection- ism and cognitive science. MIT Press. https://doi.org/10.7551 /mitpress/1187.001.0001 · DOI · Paper page
  46. Matuschek, H., Kliegl, R., Vasishth, S., Baayen, H., & Bates, D. (2017). Balancing type I error and power in linear mixed models. Journal of Memory and Language, 94, 305-315. https://doi.org /10.1016/j.jml.2017.01.001 · DOI
  47. McClelland, J. L. (2013). Incorporating rapid neocortical learning of new schema-consistent information into complementary learn- ing systems theory. Journal of Experimental Psychology: General, 142(4), 1190-1210. https://doi.org/10.1037/a0033812, PubMed: 23978185 · DOI
  48. McClelland, J. L. (2022). Capturing advanced human cognitive abilities with deep neural networks. Trends in Cognitive Sciences, 26(12), 1047-1050. https://doi.org/10.1016/j.tics.2022.09.018, PubMed: 36335015 · DOI
  49. McHugh, M. L. (2012). Interrater reliability: The kappa statistic. Biochemia Medica, 22(3), 276-282. https://doi.org/10.11613 /BM.2012.031, PubMed: 23092060 · DOI · Paper page
  50. Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022). Rethinking the role of demon- strations: What makes in-context learning work? In Y. Goldberg, Z. Kozareva, & Y. Zhang (Eds.), Proceedings of the 2022 Confer- ence on Empirical Methods in Natural Language Processing (pp. 11048-11064). Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.emnlp-main.759 · DOI · Paper page
  51. Mishra, B. D., Tafjord, O., & Clark, P. (2022). Towards teachable reasoning systems: Using a dynamic memory of user feedback for continual system improvement. In Y. Goldberg, Z. Kozareva, & Y. Zhang (Eds.), Proceedings of the 2022 Conference on Empir- ical Methods in Natural Language Processing (pp. 9465-9480). Association for Computational Linguistics. https://doi.org/10 .18653/v1/2022.emnlp-main.644 · DOI
  52. Momennejad, I., Hasanbeig, H., Vieira, F., Sharma, H., Ness, R. O., Jojic, N., Palangi, H., & Larson, J. (2023). Evaluating cognitive maps and planning in large language models with CogEval. arXiv:2309.15129. https://doi.org/10.48550/arXiv.2309.15129 · DOI · Paper page
  53. Nam, A. J., Ren, M., Finn, C., & McClelland, J. L. (2022). Learning to reason with relational abstractions. arXiv:2210.02615. https:// doi.org/10.48550/arXiv.2210.02615 · DOI
  54. Nathan, M. J., & Petrosino, A. (2003). Expert blind spot among pre- service teachers. American Educational Research Journal, 40(4), 905-928. https://doi.org/10.3102/00028312040004905 · DOI
  55. Newell, A. & Simon, H. A. (1961). GPS, a program that simulates human thought. In H. Billing (Ed.), Lernende automaten (pp. 109-124).
  56. Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84(3), 231-259. https://doi.org/10.1037/0033-295X.84 .3.231 · DOI · Paper page
  57. Oberauer, K. (2022). The importance of random slopes in mixed models for Bayesian hypothesis testing. Psychological Science, 33(4), 648-665. https://doi.org/10.1177/09567976211046884, PubMed: 35357978 · DOI
  58. Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Johnston, S., Jones, A., Kernion, J., Lovitt, L., … Olah, C. (2022). In-context learning and induction heads. arXiv:2209.11895. https://doi.org/10.48550/arXiv.2209.11895 · DOI · Paper page
  59. Palm, R. B., Paquet, U., & Winther, O. (2018). Recurrent relational networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems (NIPS'18) (pp. 3368-3378).
  60. Pearl, J. (2009). Causality: Models, reasoning, and inference (2nd ed.). Cambridge University Press. https://doi.org/10.1017 /CBO9780511803161 · DOI · Paper page
  61. Pearl, J. (2019). The seven tools of causal inference, with reflections on machine learning. Communications of the ACM, 62(3), 54-60. https://doi.org/10.1145/3241036 · DOI · Paper page
  62. Reber, A. S. (1969). Transfer of syntactic structure in synthetic lan- guages. Journal of Experimental Psychology, 81(1), 115-119. https://doi.org/10.1037/h0027454 · DOI
  63. Reber, A. S. (1976). Implicit learning of synthetic languages: The role of instructional set. Journal of Experimental Psychology: Human Learning and Memory, 2(1), 88-94. https://doi.org/10 .1037/0278-7393.2.1.88 · DOI · Paper page
  64. Reber, A. S. (1989). Implicit learning and tacit knowledge. Journal of Experimental Psychology: General, 118(3), 219-235. https:// doi.org/10.1037/0096-3445.118.3.219 · DOI · Paper page
  65. Ripp, I., Emch, M., Wu, Q., Lizarraga, A., Udale, R., von Bastian, C. C., Koch, K., & Yakushev, I. (2022). Adaptive working memory training does not produce transfer effects in cognition and neu- roimaging. Translational Psychiatry, 12(1), Article 512. https://doi .org/10.1038/s41398-022-02272-7, PubMed: 36513642 · DOI · Paper page
  66. Rumelhart, D. E. (1980). Schemata: The building blocks of cog- nition. In R. J. Spiro, B. C. Bruce, & W. F. Brewer (Eds.), The- oretical issues in reading comprehension: Perspectives from cognitive psychology, linguistics, artificial intelligence and education (pp. 33-58). Routledge. https://doi.org/10.4324 /9781315107493-4 · DOI
  67. Russell, E., & Jarvis, F. (2006). Mathematics of sudoku II. Mathemat- ical Spectrum, 39(2), 54-58.
  68. Saffran, J. R., Aslin, R. N., & Newport, E. L. (1996). Statistical learn- ing by 8-month-old infants. Science, 274(5294), 1926-1928. https://doi.org/10.1126/science.274.5294.1926, PubMed: 8943209 · DOI · Paper page
  69. Scribner, S., & Cole, M. (1973). Cognitive consequences of formal and informal education: New accommodations are needed between school-based learning and learning experiences of everyday life. Science, 182(4112), 553-559. https://doi.org/10 .1126/science.182.4112.553, PubMed: 17739714 · DOI
  70. Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D., Nham, J., Kalchbrenner, N., Sutskever, I., Lillicrap, T., Leach, M., Kavukcuoglu, K., Graepel, T., & Hassabis, D. (2016). Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587), 484-489. https://doi.org/10.1038/nature16961, PubMed: 26819042 · DOI · Paper page
  71. Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., Kluska, A., Lewkowycz, A., Agarwal, A., Power, A., Ray, A., Warstadt, A., Kocurek, A. W., Safaya, A., Tazarv, A., … Wu, Z. (2022). Beyond the Imitation Game: Quantifying and extrapo- lating the capabilities of language models. arXiv:2206.04615. https://doi.org/10.48550/arXiv.2206.04615 · DOI · Paper page
  72. Sternberg, D. A., & McClelland, J. L. (2012). Two mechanisms of human contingency learning. Psychological Science, 23(1): 59-68. https://doi.org/10.1177/0956797611429577, PubMed: 22198929 · DOI
  73. Stuhlmüller, A., Tenenbaum, J. B., & Goodman, N. D. (2010). Learning structured generative concepts. In Proceedings of the 32nd Annual Meeting of the Cognitive Science Society (pp. 2296-2301). Cognitive Science Society.
  74. Tenenbaum, J. B., Kemp, C., Griffiths, T. L., & Goodman, N. D. (2011). How to grow a mind: Statistics, structure, and abstraction. Science, 331(6022), 1279-1285. https://doi.org/10.1126/science .1192788, PubMed: 21393536 · DOI · Paper page
  75. Tsividis, P. A., Pouncy, T., Xu, J. L., Tenenbaum, J. B., & Gershman, S. J. (2017). Human learning in Atari. In AAAI 2017 Spring Sym- posium on Science of Intelligence: Computational Principles of Natural and Artificial Intelligence (pp. 643-646). AAAI.
  76. Turing, A. M. (1950). Computing machinery and intelligence. Mind, 59(236), 433-460. https://doi.org/10.1093/mind/LIX.236.433 · DOI · Paper page
  77. Uesato, J., Kushman, N., Kumar, R., Song, F., Siegel, N., Wang, L., Creswell, A., Irving, G., & Higgins, I. (2022). Solving math word problems with process-and outcome-based feedback. arXiv:2211.14275. https://doi.org/10.48550/arXiv.2211.14275 · DOI · Paper page
  78. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS' 17 ) (pp. 5998-6008).
  79. Vermaercke, B., Cop, E., Willems, S., D'Hooge, R., & Op de Beeck, H. P. (2014). More complex brains are not always better: Rats outperform humans in implicit category-based generalization by implementing a similarity-based strategy. Psychonomic Bulle- tin & Review, 21(4), 1080-1086. https://doi.org/10.3758/s13423 -013-0579-9, PubMed: 24408657 · DOI
  80. Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., Oh, J., Horgan, D., Kroiss, M., Danihelka, I., Huang, A., Sifre, L., Cai, T., Agapiou, J. P., Jaderberg, M., … Silver, D. (2019). Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7782), 350-354. https://doi.org/10.1038/s41586-019-1724-z, PubMed: 31666705 · DOI · Paper page
  81. Vygotsky, L. S. (1934). Thought and language. MIT Press.
  82. Waltz, J. A., Knowlton, B. J., Holyoak, K. J., Boone, K. B., Mishkin, F. S., de Menezes Santos, M., Thomas, C. R., & Miller, B. L. (1999). A system for relational reasoning in human prefrontal cortex. Psychological Science, 10(2), 119-125. https://doi.org /10.1111/1467-9280.00118 · DOI · Paper page
  83. Watson, J. B. (1913). Psychology as the behaviorist views it. Psy- chological Review, 20(2), 158-177. https://doi.org/10.1037 /h0074428 · DOI · Paper page
  84. Webb, T., Holyoak, K. J., & Lu, H. (2023). Emergent analogical rea- soning in large language models. Nature Human Behaviour, 7(9), 1526-1541. https://doi.org/10.1038/s41562-023-01659-w, PubMed: 37524930 · DOI · Paper page
  85. Webson, A., & Pavlick, E. (2022). Do prompt-based models really understand the meaning of their prompts? In M. Carpuat, M.-C. de Marneffe, & I. V. M. Ruiz (Eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 2300-2344). Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.naacl-main.167 · DOI · Paper page
  86. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. arXiv:2201.11903. https://doi.org/10.48550/arXiv.2201.11903 · DOI · Paper page
  87. Xie, S. M., Raghunathan, A., Liang, P., & Ma, T. (2021). An expla- nation of in-context learning as implicit Bayesian inference. arXiv:2111.02080. https://doi.org/10.48550/arXiv.2111.02080 · DOI · Paper page
  88. Yadlowsky, S., Doshi, L., & Tripuraneni, N. (2023). Pretraining data mixtures enable narrow model selection capabilities in trans- former models. arXiv:2311.00871. https://doi.org/10.48550 /arXiv.2311.00871 · DOI · Paper page

Cited by

Metadata

Authors
Nam, A. J., & McClelland, J. L.
Affiliations
Stanford University
Venue
arXiv
Publisher
arXiv
Published
2021
DOI
10.48550/arxiv.2107.06994
arXiv
2107.06994
References
0
Type
Preprint
Licence
creativecommons.org/licenses/by-nc-nd/4.0/legalcode

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.