Promises and challenges of human computational ethology
Open accessThe DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Abstract
SUMMARY The movements an organism makes provide insights into its internal states and motives. This principle is the foundation of the new field of computational ethology, which links rich automatic measurements of natural behaviors to motivational states and neural activity. Computational ethology has proven transformative for animal behavioral neuroscience. This success raises the question whether rich automatic measurements of behavior can similarly drive progress in human neuroscience and psychology? New technologies for capturing and analyzing complex behaviors in real and virtual environments enable us to probe the human brain during naturalistic dynamic interactions with the environment that were beyond experimental investigation so far. Inspired by nonhuman computational ethology, we explore how these new tools can be used to test important questions in human neuroscience. We argue that the application of this methodology will help human neuroscience and psychology extend on limited behavioral measurements such as reaction time and accuracy, permit novel insights into how the human brain produces behavior, and ultimately reduce the growing measurement gap between human and animal neuroscience.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Carreira, J., & Zisserman, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.502 · Cited by 9,570
Watkins, C. J. C. H., & Dayan, P. (1992) · Machine Learning · doi:10.1007/bf00992698 · Cited by 9,083
DeepLabCut: markerless pose estimation of user-defined body parts with deep learning
Mathis, A., Mamidanna, P., Cury, K. M., Abe, T., Murthy, V. N., Mathis, M. W., & Bethge, M. (2018) · Nature Neuroscience · doi:10.1038/s41593-018-0209-y · Cited by 5,187
Generative Adversarial Networks
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.2661 · Cited by 4,477
DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., & Darrell, T. (2013) · arXiv · doi:10.48550/arxiv.1310.1531 · Cited by 4,019 · cited 1×
“…from algorithms designed for the detection of human poses (Insafutdinov et al., 2016), and have been made possible through transfer learning (Donahue et al., 2013), which takes advantage of pre-trained neural networks to facilitate performance without the need for extensive training data…”
Distributed and overlapping representations of faces and objects in ventral temporal cortex
Haxby, J. V., Gobbini, M. I., Furey, M. L., Ishai, A., Schouten, J. L., & Pietrini, P. (2001) · Science · doi:10.1126/science.1063736 · Cited by 3,940
Hamilton, W. D. (1971) · Journal of Theoretical Biology · doi:10.1016/0022-5193(71)90189-5 · Cited by 3,506
Grandmaster level in StarCraft II using multi-agent reinforcement learning
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al. (2019) · Nature · doi:10.1038/s41586-019-1724-z · Cited by 3,455
Representational similarity analysis-connecting the branches of systems neuroscience
Kriegeskorte, N. (2008) · Frontiers in Systems Neuroscience · doi:10.3389/neuro.06.004.2008 · Cited by 3,355
Self-citationTracking whole-brain connectivity dynamics in the resting state
Allen, E. A., Damaraju, E., Plis, S. M., Erhardt, E. B., Eichele, T., & Calhoun, V. D. (2014) · Cerebral Cortex · doi:10.1093/cercor/bhs352 · Cited by 2,992
Kaiser, R. H., Andrews-Hanna, J. R., Wager, T. D., & Pizzagalli, D. A. (2015) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2015.0071 · Cited by 2,281
Uncertainty and anticipation in anxiety: an integrated neurobiological and psychological perspective
Grupe, D. W., & Nitschke, J. B. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3524 · Cited by 2,007
Rethinking the Emotional Brain
LeDoux, J. (2012) · Neuron · doi:10.1016/j.neuron.2012.02.004 · Cited by 1,594 · cited 1×
“In the natural world, animals modify their behavior in response to changes in their environment, such as predation and competition, as well as changes in their internal metabolic drives (e.g., hunger and thirst; LeDoux, 2012; Mobbs et al., 2018).”
BackgroundSpontaneous behaviors drive multidimensional, brainwide activity
Stringer, C., Pachitariu, M., Steinmetz, N., Reddy, C. B., Carandini, M., & Harris, K. D. (2019) · Science · doi:10.1126/science.aav7893 · Cited by 1,510 · cited 2×
“Finally, Stringer et al. (2019) combined data-driven parsing of high-dimensional natural behaviors and neural activity in mice to demonstrate that a large proportion of neural activity across the cortex is linked to behavioral patterns.”
BackgroundCellular networks underlying human spatial navigation
Ekstrom, A. D., Kahana, M. J., Caplan, J. B., Fields, T. A., Isham, E. A., Newman, E. L., & Fried, I. (2003) · Nature · doi:10.1038/nature01964 · Cited by 1,425 · cited 1×
“…studies have shown VR using desktop computers, head mounted displays and other technologies can be used to study several aspects of human cognition and related neural mechanisms (Ekstrom et al., 2003; Jacobs et al., 2013; Chrastil et al., 2015; Diersch and Wolbers, 2019; Hartley et al., 2003).”
BackgroundMoving magnetoencephalography towards real-world applications with a wearable system
Boto, E., Holmes, N., Leggett, J., Roberts, G., Shah, V., Meyer, S. S., Muñoz, L. D., Mullinger, K. J., Tierney, T. M., Bestmann, S., et al. (2018) · Nature · doi:10.1038/nature26147 · Cited by 1,377 · cited 1×
“Further, with the creation of moveable optically pumped magnetometer magnetoencephalography (OPM-MEG; (Boto et al., 2018)), providing electrophysiological measurements at the millisecond resolution, there is promise in combining VR with spatially and temporally high-resolution brain imaging on a…”
BackgroundNeuroscience Needs Behavior: Correcting a Reductionist Bias
Krakauer, J. W., Ghazanfar, A. A., Gomez-Marin, A., MacIver, M. A., & Poeppel, D. (2017) · Neuron · doi:10.1016/j.neuron.2016.12.041 · Cited by 1,358 · cited 1×
“In turn, this will provide better models of how the brain produces behaviors (Babayan and Konen, 2019; Niv, 2020; Krakauer et al., 2017).”
BackgroundCluster-extent based thresholding in fMRI analyses: Pitfalls and recommendations
Woo, C.-W., Krishnan, A., & Wager, T. D. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2013.12.058 · Cited by 1,253
Hippocampal place-cell sequences depict future paths to remembered goals
Pfeiffer, B. E., & Foster, D. J. (2013) · Nature · doi:10.1038/nature12112 · Cited by 1,216
Mastering Atari, Go, chess and shogi by planning with a learned model
Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., et al. (2020) · Nature · doi:10.1038/s41586-020-03051-4 · Cited by 1,214
Virtual reality in neuroscience research and therapy
Bohil, C. J., Alicea, B., & Biocca, F. A. (2011) · Nature Reviews Neuroscience · doi:10.1038/nrn3122 · Cited by 1,108 · cited 1×
“Its use in human neuroimaging and psychological experiments has been revolutionary as it provides a more enriched and naturalistic approach to computerized environments (Bohil et al., 2011; Reggente et al., 2018; see Fig.”
BackgroundWhen fear is near: threat imminence elicits prefrontal periaqueductal gray shifts in humans.
Mobbs, D., Petrovic, P., Marchant, J. L., Hassabis, D., Weiskopf, N., Seymour, B., Dolan, R. J., & Frith, C. D. (2007) · Science · doi:10.1126/science.1144298 · Cited by 951
Self-citationCarl, E., Stein, A. T., Levihn-Coon, A., Pogue, J. R., Rothbaum, B., Emmelkamp, P., Asmundson, G. J. G., Carlbring, P., & Powers, M. B. (2019) · Journal of Anxiety Disorders · doi:10.1016/j.janxdis.2018.08.003 · Cited by 889
The Well-Worn Route and the Path Less Traveled
Hartley, T., Maguire, E. A., Spiers, H. J., & Burgess, N. (2003) · Neuron · doi:10.1016/s0896-6273(03)00095-3 · Cited by 817
Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks
Dosovitskiy, A., Fischer, P., Springenberg, J. T., Riedmiller, M., & Brox, T. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2496141 · Cited by 762
Mapping Sub-Second Structure in Mouse Behavior
Wiltschko, A. B., Johnson, M. J., Iurilli, G., Peterson, R. E., Katon, J. M., Pashkovski, S. L., Abraira, V. E., Adams, R. P., & Datta, S. R. (2015) · Neuron · doi:10.1016/j.neuron.2015.11.031 · Cited by 760 · cited 3×
“…(Graving et al., 2019; Mathis et al., 2018; Pereira et al., 2019), where an experimenter manually identifies the features of the animal that should be tracked, or an unsupervised way (Berman et al., 2014; Wiltschko et al., 2015), where features of interest are identified without human intervention.”
BackgroundMethodHuman-level performance in 3D multiplayer games with population-based reinforcement learning
Jaderberg, M., Czarnecki, W. M., Dunning, I., Marris, L., Lever, G., Castañeda, A. G., Beattie, C., Rabinowitz, N. C., Morcos, A. S., Ruderman, A., et al. (2019) · Science · doi:10.1126/science.aau6249 · Cited by 662
Self-citationCharacterizing a psychiatric symptom dimension related to deficits in goal-directed control
Gillan, C. M., Kosinski, M., Whelan, R., Phelps, E. A., & Daw, N. D. (2016) · eLife · doi:10.7554/elife.11305 · Cited by 586 · cited 1×
“As an example, prior work has considered the importance of MB planning in compulsive symptoms, showing that individuals high in these traits have difficulty in learning a model of the world and in using this learned model to guide behavior (Gillan et al., 2016; Sharp et al., 2020).”
BackgroundMapping the stereotyped behaviour of freely moving fruit flies
Berman, G. J., Choi, D. M., Bialek, W., & Shaevitz, J. W. (2014) · Journal of The Royal Society Interface · doi:10.1098/rsif.2014.0672 · Cited by 565
DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning
Graving, J. M., Chae, D., Naik, H., Li, L., Koger, B., Costelloe, B. R., & Couzin, I. D. (2019) · eLife · doi:10.7554/elife.47994 · Cited by 547
Cognitive computational neuroscience
Kriegeskorte, N., & Douglas, P. K. (2018) · Nature Neuroscience · doi:10.1038/s41593-018-0210-5 · Cited by 535
Self-citationFrom threat to fear: the neural organization of defensive fear systems in humans.
Mobbs, D., Marchant, J. L., Hassabis, D., Seymour, B., Tan, G., Gray, M., Petrovic, P., Dolan, R. J., & Frith, C. D. (2009) · The Journal of Neuroscience · doi:10.1523/jneurosci.2378-09.2009 · Cited by 476
Self-citationComputational Neuroethology: A Call to Action
Datta, S. R., Anderson, D. J., Branson, K., Perona, P., & Leifer, A. (2019) · Neuron · doi:10.1016/j.neuron.2019.09.038 · Cited by 453
Prioritized memory access explains planning and hippocampal replay
Mattar, M. G., & Daw, N. D. (2018) · Nature Neuroscience · doi:10.1038/s41593-018-0232-z · Cited by 393 · cited 1×
“Mattar and Daw, 2018; Pfeiffer and Foster, 2013; Wise et al., 2020)).”
BackgroundDirect Fit to Nature: An Evolutionary Perspective on Biological and Artificial Neural Networks
Hasson, U., Nastase, S. A., & Goldstein, A. (2020) · Neuron · doi:10.1016/j.neuron.2019.12.002 · Cited by 387
Automated monitoring and analysis of social behavior in Drosophila
Dankert, H., Wang, L., Hoopfer, E. D., Anderson, D. J., & Perona, P. (2009) · Nature Methods · doi:10.1038/nmeth.1310 · Cited by 382
Prediction during natural language comprehension
Willems, R. M., Frank, S. L., Nijhof, A. D., Hagoort, P., & van den Bosch, A. (2016) · Cerebral Cortex · doi:10.1093/cercor/bhv075 · Cited by 288 · cited 1×
“…distributed patterns of activity associated with specific behavioral patterns emerging from naturalistic behavior, in a similar way to previous work relating activation measured using fMRI to naturalistic movie stimuli (Spiers and Maguire, 2007) or continuous speech (Willems et al., 2016).”
ResultLearning with Opponent-Learning Awareness
Foerster, J., Chen, R. Y., Al-Shedivat, M., Whiteson, S., Abbeel, P., & Mordatch, I. (2018) · International Joint Conference on Autonomous Agents and Multiagent Systems · doi:10.65109/hgwa8807 · Cited by 215
Gold, A. L., Morey, R. A., & McCarthy, G. (2015) · Biological Psychiatry · doi:10.1016/j.biopsych.2014.03.030 · Cited by 192
Rizzo, A. “., Difede, J., Rothbaum, B. O., Reger, G., Spitalnick, J., Cukor, J., & Mclay, R. (2010) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.2010.05755.x · Cited by 185
Stupar-Rutenfrans, S., Ketelaars, L. E. H., & van Gisbergen, M. S. (2017) · Cyberpsychology Behavior and Social Networking · doi:10.1089/cyber.2017.0174 · Cited by 144 · cited 1×
“In contrast, VR has permitted assessment of common fears, such as height and public speaking, in the lab (Gromer et al., 2019; Stupar-Rutenfrans et al., 2017).”
BackgroundChrastil, E. R., Sherrill, K. R., Hasselmo, M. E., & Stern, C. E. (2015) · The Journal of Neuroscience · doi:10.1523/jneurosci.1209-15.2015 · Cited by 128 · cited 1×
“…studies have shown VR using desktop computers, head mounted displays and other technologies can be used to study several aspects of human cognition and related neural mechanisms (Ekstrom et al., 2003; Jacobs et al., 2013; Chrastil et al., 2015; Diersch and Wolbers, 2019; Hartley et al., 2003).”
BackgroundCausal interactive links between presence and fear in virtual reality height exposure
Gromer, D., Reinke, M., Christner, I., & Pauli, P. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.00141 · Cited by 127 · cited 1×
“In contrast, VR has permitted assessment of common fears, such as height and public speaking, in the lab (Gromer et al., 2019; Stupar-Rutenfrans et al., 2017).”
BackgroundA multi-agent reinforcement learning model of common-pool resource appropriation
Perolat, J., Leibo, J. Z., Zambaldi, V., Beattie, C., Tuyls, K., & Graepel, T. (2017) · arXiv · doi:10.48550/arxiv.1707.06600 · Cited by 109 · cited 3×
“In other circumstances, they found the spontaneous emergence of exclusion behaviors and inequality (Perolat et al., 2017).”
BackgroundMethodSelf-citationMaintaining cooperation in complex social dilemmas using deep reinforcement learning
Lerer, A., & Peysakhovich, A. (2017) · arXiv · doi:10.48550/arxiv.1707.01068 · Cited by 101
Connecting neural codes with behavior in the auditory system of Drosophila
Clemens, J., Girardin, C. C., Coen, P., Guan, X.-J., Dickson, B. J., & Murthy, M. (2015) · Neuron · doi:10.1016/j.neuron.2015.08.014 · Cited by 69
Shiban, Y., Brütting, J., Pauli, P., & Mühlberger, A. (2015) · Journal of Behavior Therapy and Experimental Psychiatry · doi:10.1016/j.jbtep.2014.09.009 · Cited by 60
Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design
Dennis, M., Jaques, N., Vinitsky, E., Bayen, A., Russell, S., Critch, A., & Levine, S. (2020) · arXiv · doi:10.48550/arxiv.2012.02096 · Cited by 60 · cited 1×
“learning agents may allow virtual ecologies to be defined without experimenter input (Dennis et al., 2020).”
BackgroundSocial Diversity and Social Preferences in Mixed-Motive Reinforcement Learning
McKee, K. R., Gemp, I., McWilliams, B., Duèñez-Guzmán, E. A., Hughes, E., & Leibo, J. Z. (2020) · International Joint Conference on Autonomous Agents and Multiagent Systems · doi:10.65109/dnuf7773 · Cited by 33
Self-citation
Cited by
Predictive maps in rats and humans for spatial navigation
de Cothi, W., Nyberg, N., Griesbauer, E.-M., Ghanamé, C., Zisch, F., Lefort, J. M., Fletcher, L., Newton, C., Renaudineau, S., Bendor, D., et al. (2022) · Current Biology · doi:10.1016/j.cub.2022.06.090 · Cited by 109
Le Mau, T., Hoemann, K., Lyons, S. H., Fugate, J. M. B., Brown, E. N., Gendron, M., & Barrett, L. F. (2021) · Nature Communications · doi:10.1038/s41467-021-25352-6 · Cited by 74
A synergetic turn in cognitive neuroscience of brain diseases
Ibanez, A., Kringelbach, M. L., & Deco, G. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2023.12.006 · Cited by 65
Spurious normativity enhances learning of compliance and enforcement behavior in artificial agents
Köster, R., Hadfield-Menell, D., Everett, R., Weidinger, L., Hadfield, G. K., & Leibo, J. Z. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2106028118 · Cited by 37
“For instance, it may be important to study not only what choice agents make but also how they implement that choice (34).”
BackgroundSelf-citationEmergence of Collaborative Hunting via Multi-Agent Deep Reinforcement Learning
Tsutsui, K., Takeda, K., & Fujii, K. (2023) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-37660-3_15 · Cited by 5
Metadata
- Authors
- Mobbs, D., Wise, T., Suthana, N., Guzmán, N., Kriegeskorte, N., & Leibo, J. Z.
- Venue
- Neuron
- Publisher
- Elsevier BV
- Published
- 2021-07
- Volume
- 109
- Issue
- 14
- Pages
- 2224-2238
- ISSN
- 0896-6273 (ISSN-L)
- OpenAlex
- W3167903396
- Cited by
- 82
- References
- 49
- FWCI
- 6.71
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.