Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense
Open accessThe DOI matches its Crossref record; 74 of the 74 references it cites were found in a registry or an index.
Abstract
Recent progress in deep learning is essentially based on a “big data for small tasks” paradigm, under which massive amounts of data are used to train a classifier for a single narrow task. In this paper, we call for a shift that flips this paradigm upside down. Specifically, we propose a “small data for big tasks” paradigm, wherein a single artificial intelligence (AI) system is challenged to develop “common sense,” enabling it to solve a wide range of tasks with little training data. We illustrate the potential power of this new paradigm by reviewing models of common sense that synthesize recent breakthroughs in both machine and human vision. We identify functionality, physics, intent, causality, and utility (FPICU) as the five core domains of cognitive AI with humanlike common sense. When taken as a unified concept, FPICU is concerned with the questions of “why” and “how,” beyond the dominant “what” and “where” framework for understanding vision. They are invisible in terms of pixels but nevertheless drive the creation, maintenance, and development of visual scenes. We therefore coin them the “dark matter” of vision. Just as our universe cannot be understood by merely studying observable matter, we argue that vision cannot be understood without studying FPICU. We demonstrate the power of this perspective to develop cognitive AI systems with humanlike common sense by showing how to observe and apply FPICU with little training data to solve a wide range of challenging tasks, including tool use, planning, utility inference, and social learning. In summary, we argue that the next generation of AI must embrace “dark” humanlike common sense for solving novel tasks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589
Histograms of Oriented Gradients for Human Detection
Dalal, N., & Triggs, B. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.177 · Cited by 33,584
Multiple View Geometry in Computer Vision
Hartley, R., & Zisserman, A. (2004) · doi:10.1017/cbo9780511811685 · Cited by 21,070
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
The origins of intelligence in children.
Piaget, J. (1952) · doi:10.1037/11494-000 · Cited by 10,687
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625
Playing Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471
Estimating causal effects of treatments in randomized and nonrandomized studies
Rubin, D. B. (1974) · Journal of Educational Psychology · doi:10.1037/h0037350 · Cited by 9,436 · cited 1×
“Rubin [146] laid the foundation for causal analysis in sta-”
BackgroundDoes the chimpanzee have a theory of mind?
Premack, D., & Woodruff, G. (1978) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00076512 · Cited by 8,743
Rizzolatti, G., & Craighero, L. (2004) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.27.070203.144230 · Cited by 7,433 · cited 1×
“Accumulating neurophysiological evidence supports such simulations in the human brain; one example is the mirror neuron [331], which has been linked to intent understanding in many studies [102,332].”
BackgroundCognitive maps in rats and men
Tolman, E. C. (1948) · Psychological Review · doi:10.1037/h0061626 · Cited by 6,641
Indoor segmentation and support inference from rgbd images
Silberman, N., Hoiem, D., Kohli, P., & Fergus, R. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33715-4_54 · Cited by 5,760
Clark, H. H. (1996) · doi:10.1017/cbo9780511620539 · Cited by 5,123
Fisher, R. A. (1936) · BMJ · doi:10.1136/bmj.1.3923.554-a · Cited by 4,980
Understanding and sharing intentions: The origins of cultural cognition
Tomasello, M., Carpenter, M., Call, J., Behne, T., & Moll, H. (2005) · Behavioral and Brain Sciences · doi:10.1017/s0140525x05000129 · Cited by 4,882
The construction of reality in the child.
Piaget, J. (1954) · doi:10.1037/11168-000 · Cited by 4,558
Causation, Prediction, and Search
Spirtes, P., Glymour, C., & Scheines, R. (2001) · doi:10.7551/mitpress/1754.001.0001 · Cited by 4,521
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction
Imbens, G. W., & Rubin, D. B. (2015) · doi:10.1017/cbo9781139025751 · Cited by 4,210
Microstructure of a spatial map in the entorhinal cortex
Hafting, T., Fyhn, M., Molden, S., Moser, M.-B., & Moser, E. I. (2005) · Nature · doi:10.1038/nature03721 · Cited by 4,068
Speed of processing in the human visual system
Thorpe, S., Fize, D., & Marlot, C. (1996) · Nature · doi:10.1038/381520a0 · Cited by 3,538
Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., & Mordatch, I. (2017) · arXiv · doi:10.48550/arxiv.1706.02275 · Cited by 3,377
Self-citationAn Experimental Study of Apparent Behavior
Heider, F., & Simmel, M. (1944) · The American Journal of Psychology · doi:10.2307/1416950 · Cited by 3,340
A survey of robot learning from demonstration
Argall, B. D., Chernova, S., Veloso, M., & Browning, B. (2009) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2008.10.024 · Cited by 3,227
Object vision and spatial vision: Two cortical pathways
Mishkin, M., Ungerleider, L. G., & Macko, K. A. (1983) · Trends in Neurosciences · doi:10.1016/0166-2236(83)90190-x · Cited by 2,947
ShapeNet: An Information-Rich 3D Model Repository
Chang, A. X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., et al. (2015) · arXiv · doi:10.48550/arxiv.1512.03012 · Cited by 2,330
Carey, S. (2009) · doi:10.1093/acprof:oso/9780195367638.001.0001 · Cited by 2,322
AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles
Shah, S., Dey, D., Lovett, C., & Kapoor, A. (2018) · Springer Proceedings in Advanced Robotics · doi:10.1007/978-3-319-67361-5_40 · Cited by 2,254
Improving fluid intelligence with training on working memory
Jaeggi, S. M., Buschkuehl, M., Jonides, J., & Perrig, W. J. (2008) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0801268105 · Cited by 2,187
Grasping the intentions of others with one's own mirror neuron system
Iacoboni, M., Molnar-Szakacs, I., Gallese, V., Buccino, G., Mazziotta, J. C., & Rizzolatti, G. (2005) · PLoS Biology · doi:10.1371/journal.pbio.0030079 · Cited by 2,044 · cited 2×
“Accumulating neurophysiological evidence supports such simulations in the human brain; one example is the mirror neuron [331], which has been linked to intent understanding in many studies [102,332].”
BackgroundThe eyes have it: the neuroethology, function and evolution of social gaze
Emery, N. J. (2000) · Neuroscience & Biobehavioral Reviews · doi:10.1016/s0149-7634(00)00025-7 · Cited by 2,037
Describing objects by their attributes
Farhadi, A., Endres, I., Hoiem, D., & Forsyth, D. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206772 · Cited by 1,917
Counterfactual Multi-Agent Policy Gradients
Foerster, J., Farquhar, G., Afouras, T., Nardelli, N., & Whiteson, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11794 · Cited by 1,873
Nonlinear Continuum Mechanics for Finite Element Analysis
Bonet, J., & Wood, R. D. (2008) · doi:10.1017/cbo9780511755446 · Cited by 1,824 · cited 3×
“Lagrangian mesh-based methods, represented by FEM [391, 392, 393], where the material is described with and embedded in a deforming mesh.”
BackgroundMethodTextons, the elements of texture perception, and their interactions
Julesz, B. (1981) · Nature · doi:10.1038/290091a0 · Cited by 1,703 · cited 1×
“” Marr conjectured that the perception of a 2D image is an explicit multiphase information process [1], involving (i) an early vision system for perceiving [187, 188] and textons [189, 190] to form a primal sketch [191, 192]; (ii) a mid-level vision system to form 2.”
BackgroundCarpenter, P. A., Just, M. A., & Shell, P. (1990) · Psychological Review · doi:10.1037/0033-295x.97.3.404 · Cited by 1,648
Gaze and eye contact: a research review
Kleinke, C. L. (1986) · Psychological Bulletin · doi:10.1037/0033-2909.100.1.78 · Cited by 1,455
Rational imitation in preverbal infants
Gergely, G., Bekkering, H., & Király, I. (2002) · Nature · doi:10.1038/415755a · Cited by 1,391
Learning to Communicate with Deep Multi-Agent Reinforcement Learning
Foerster, J. N., Assael, Y. M., de Freitas, N., & Whiteson, S. (2016) · arXiv · doi:10.48550/arxiv.1605.06676 · Cited by 1,217 · cited 3×
“Multiple-Agent Systems (MAS) setting [367, 368, 369].”
BackgroundThe Complexity of Decentralized Control of Markov Decision Processes
Bernstein, D. S., Givan, R., Immerman, N., & Zilberstein, S. (2002) · Mathematics of Operations Research · doi:10.1287/moor.27.4.819.297 · Cited by 1,192
Terzopoulos, D., Platt, J., Barr, A., & Fleischer, K. (1987) · ACM SIGGRAPH Computer Graphics · doi:10.1145/37402.37427 · Cited by 1,110
Perceptual learning: Differentiation or enrichment?
Gibson, J. J., & Gibson, E. J. (1955) · Psychological Review · doi:10.1037/h0048826 · Cited by 993 · cited 1×
“Originating from perceptual learning [478,479], it is well established in the field of psychology and education [480–484] that teaching new concepts by comparing noisy examples is quite effective.”
Data-Driven Grasp Synthesis—A Survey
Bohg, J., Morales, A., Asfour, T., & Kragic, D. (2014) · IEEE Transactions on Robotics · doi:10.1109/tro.2013.2289018 · Cited by 929
Action understanding as inverse planning
Baker, C. L., Saxe, R., & Tenenbaum, J. B. (2009) · Cognition · doi:10.1016/j.cognition.2009.07.005 · Cited by 920
Self-citationPredicting pragmatic reasoning in language games
Frank, M. C., & Goodman, N. D. (2012) · Science · doi:10.1126/science.1218633 · Cited by 868 · cited 2×
“[360] Michael C Frank, Noah D Goodman, Predicting pragmatic reasoning in language games, Science 336 (6084) (2012) 998–998.”
BackgroundEvidence for grid cells in a human memory network
Doeller, C. F., Barry, C., & Burgess, N. (2010) · Nature · doi:10.1038/nature08704 · Cited by 828 · cited 2×
“[58] Christian F Doeller, Caswell Barry, Neil Burgess, Evidence for grid cells in a human memory network, Nature 463 (7281) (2010) 657.”
BackgroundStructure-mapping: A theoretical framework for analogy
GENTNER, D. (1983) · Cognitive Science · doi:10.1016/s0364-0213(83)80009-3 · Cited by 816 · cited 2×
“[484] Dedre Gentner, Structure-mapping: A theoretical framework for analogy, Cognitive science 7 (2) (1983) 155–170.”
BackgroundUniversal intelligence: a definition of machine intelligence
Legg, S., & Hutter, M. (2007) · Minds and Machines · doi:10.1007/s11023-007-9079-x · Cited by 803
Gibson Env: Real-World Perception for Embodied Agents
Xia, F., Zamir, A. R., He, Z., Sax, A., Malik, J., & Savarese, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00945 · Cited by 763
From Blobs to Boundary Edges: Evidence for Time- and Spatial-Scale-Dependent Scene Recognition
Schyns, P. G., & Oliva, A. (1994) · Psychological Science · doi:10.1111/j.1467-9280.1994.tb00500.x · Cited by 715 · cited 3×
“The human brain can grasp the “gist” of a scene in an image within 200 milliseconds, as observed by Potter in the 1970s [8, 9], and Schyns [10] and Thorpe [11] in the 1990s.”
BackgroundSimulation as an engine of physical scene understanding
Battaglia, P. W., Hamrick, J. B., & Tenenbaum, J. B. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1306572110 · Cited by 681
Self-citationPragmatic Language Interpretation as Probabilistic Inference
Goodman, N. D., & Frank, M. C. (2016) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2016.08.005 · Cited by 594
Curvilinear Motion in the Absence of External Forces: Naïve Beliefs About the Motion of Objects
McCloskey, M., Caramazza, A., & Green, B. (1980) · Science · doi:10.1126/science.210.4474.1139 · Cited by 586
AI2-THOR: An Interactive 3D Environment for Visual AI
Kolve, E., Mottaghi, R., Han, W., VanderBilt, E., Weihs, L., Herrasti, A., Deitke, M., Ehsani, K., Gordon, D., Zhu, Y., et al. (2017) · arXiv · doi:10.48550/arxiv.1712.05474 · Cited by 578 · cited 1×
“To reach this goal, we have seen the increasing use of synthetic data and simulation platforms for indoor scenes in recent years by leveraging state-of-the-art game engines and free, publicly available 3D content [288,375–377], including MINOS [378], HoME [379], Gibson Environment [380], House3D [381], AI2THOR [382],…”
Self-citationVQA: Visual Question Answering: www.visualqa.org
Agrawal, A., Lu, J., Antol, S., Mitchell, M., Zitnick, C. L., Parikh, D., & Batra, D. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0966-6 · Cited by 570
The ontogeny of fairness in seven societies
Blake, P. R., McAuliffe, K., Corbit, J., Callaghan, T. C., Barry, O., Bowie, A., Kleutsch, L., Kramer, K. L., Ross, E., Vongsachang, H., et al. (2015) · Nature · doi:10.1038/nature15703 · Cited by 534
Emergence of grounded compositional language in multi-agent populations
Mordatch, I., & Abbeel, P. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11492 · Cited by 440
Foo, P., Warren, W. H., Duchon, A., & Tarr, M. J. (2005) · Journal of Experimental Psychology: Learning, Memory, and Cognition · doi:10.1037/0278-7393.31.2.195 · Cited by 406
An algorithmic perspective on imitation learning
Osa, T., Pajarinen, J., Neumann, G., Bagnell, J. A., Abbeel, P., & Peters, J. (2018) · Foundations and Trends in Robotics · doi:10.1561/2300000053 · Cited by 405 · cited 1×
“from Demonstration (LfD); see more details in two previous surveys [281, 282].”
BackgroundAnalogy and Relational Reasoning
Holyoak, K. J. (2012) · The Oxford Handbook of Thinking and Reasoning · doi:10.1093/oxfordhb/9780199734689.013.0013 · Cited by 394
VirtualHome: Simulating Household Activities Via Programs
Puig, X., Ra, K., Boben, M., Li, J., Wang, T., Fidler, S., & Torralba, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00886 · Cited by 378
The class of continuous timelike curves determines the topology of spacetime
Malament, D. B. (1977) · Journal of Mathematical Physics · doi:10.1063/1.523436 · Cited by 357
Tversky, B. (1992) · Geoforum · doi:10.1016/0016-7185(92)90011-r · Cited by 272 · cited 1×
“e 3D layout of a scene in a way that fundamentally differs from any current computer vision algorithms [40,41]. In fact, multiple experimental studies do not countenance global metric representations [42,43,44,45,46,47]; human vision is errorprone and distorted in terms of localization [48,49,50,51,52]. In a case stud…”
BackgroundMind games: Game engines as an architecture for intuitive physics
Ullman, T. D., Spelke, E., Battaglia, P., & Tenenbaum, J. B. (2017) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2017.05.012 · Cited by 234 · cited 2×
“[179] Tomer D Ullman, Elizabeth Spelke, Peter Battaglia, Joshua B Tenenbaum, Mind games: Game engines as an architecture for intuitive physics, Trends in Cognitive Sciences 21 (9) (2017) 649–665.”
BackgroundSelf-citationMulti-Agent Cooperation and the Emergence of (Natural) Language
Lazaridou, A., Peysakhovich, A., & Baroni, M. (2016) · arXiv · doi:10.48550/arxiv.1612.07182 · Cited by 231 · cited 2×
“The emergence of more realistic communication protocols using discrete messages has been explored in various types of communication games [443, 444, 445, 446], in which agents need to process visual signals and attach discrete tokens to attributes or seman-”
BackgroundCultural differences in moral judgment and behavior, across and within societies
Graham, J., Meindl, P., Beall, E., Johnson, K. M., & Zhang, L. (2016) · Current Opinion in Psychology · doi:10.1016/j.copsyc.2015.09.007 · Cited by 220
Dynamic anticrack propagation in snow
Gaume, J., Gast, T., Teran, J., van Herwijnen, A., & Jiang, C. (2018) · Nature Communications · doi:10.1038/s41467-018-05181-w · Cited by 173 · cited 1×
“In the field of visual computing, existing work includes snow [407,408], foam [409–411], sand [412,413], rigid bodies [414], fractures [415,416], cloth [417], hair [418], water [419], and solid–fluid”
BackgroundWormholes in virtual space: From cognitive maps to cognitive graphs
Warren, W. H., Rothman, D. B., Schnapp, B. H., & Ericson, J. D. (2017) · Cognition · doi:10.1016/j.cognition.2017.05.020 · Cited by 167 · cited 2×
“In fact, multiple experimental studies countenance against global metric representations [41, 42, 43, 44, 45, 46]; human vision is errorprone and distorted in terms of localization [47, 48, 49, 50, 51].”
BackgroundFrom Cognitive Maps to Cognitive Graphs
Chrastil, E. R., & Warren, W. H. (2014) · PLoS ONE · doi:10.1371/journal.pone.0112544 · Cited by 146
Formalizing Neurath’s ship: approximate algorithms for online causal learning.
Bramley, N. R., Dayan, P., Griffiths, T. L., & Lagnado, D. A. (2017) · Psychological Review · doi:10.1037/rev0000061 · Cited by 136 · cited 1×
“vention, which enables agents to explore possible directions of undirected causal edges [153, 154].”
BackgroundEmergence of Linguistic Communication from Referential Games with Symbolic and Pixel Input
Lazaridou, A., Hermann, K. M., Tuyls, K., & Clark, S. (2018) · arXiv · doi:10.48550/arxiv.1804.03984 · Cited by 86
A glove-based system for studying hand-object manipulation via joint pose and force sensing
Liu, H., Xie, X., Millar, M., Edmonds, M., Gao, F., Zhu, Y., Santos, V. J., Rothrock, B., & Zhu, S.-C. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8206575 · Cited by 60 · cited 5×
“By using the Velostat [294] glove in demonstration, the fine forces used to unlock the child-safety mecha- nism become observable.”
BackgroundMethodInfluentialSelf-citation
Cited by
A Survey of Embodied AI: From Simulators to Research Tasks
Duan, J., Yu, S., Tan, H. L., Zhu, H., & Tan, C. (2022) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2022.3141105 · Cited by 420
Quilty-Dunn, J., Porot, N., & Mandelbaum, E. (2023) · Behavioral and Brain Sciences · doi:10.1017/s0140525x22002849 · Cited by 205
Butz, M. V. (2021) · KI - Künstliche Intelligenz · doi:10.1007/s13218-021-00705-x · Cited by 42
Metadata
- Authors
- Zhu, Y., Gao, T., Fan, L., Huang, S., Edmonds, M., Liu, H., Gao, F., Zhang, C., Qi, S., Wu, Y. N., Tenenbaum, J. B., & Zhu, S.-C.
- Venue
- Engineering
- Publisher
- Elsevier BV
- Published
- 2020-03
- Volume
- 6
- Issue
- 3
- Pages
- 310-345
- ISSN
- 2095-8099 (ISSN-L)
- OpenAlex
- W3007580291
- Cited by
- 132
- References
- 74
- FWCI
- 7.20
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.