DeepBehavior: A Deep Learning Toolbox for Automated Analysis of Animal and Human Behavior Imaging Data
Open accessThe DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Abstract
Detailed behavioral analysis is key to understanding the brain-behavior relationship. Here, we present deep learning-based methods for analysis of behavior imaging data in mice and humans. Specifically, we use three different convolutional neural network architectures and five different behavior tasks in mice and humans and provide detailed instructions for rapid implementation of these methods for the neuroscience community. We provide examples of three dimensional (3D) kinematic analysis in the food pellet reaching task in mice, three-chamber test in mice, social interaction test in freely moving mice with simultaneous miniscope calcium imaging, and 3D kinematic analysis of two upper extremity movements in humans (reaching and alternating pronation/supination). We demonstrate that the transfer learning approach accelerates the training of the network when using images from these types of behavior video recordings. We also provide code for post-processing of the data after initial analysis with deep learning. Our methods expand the repertoire of available tools using deep learning for behavior analysis by providing detailed instructions on implementation, applications in several behavior tests, and post-processing methods and annotated code for detailed behavior analysis. Moreover, our methods in human motor behavior can be used in the clinic to assess motor function during recovery after an injury such as stroke.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields
Cao, Z., Simon, T., Wei, S.-E., & Sheikh, Y. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.143 · Cited by 7,539 · cited 2×
“We used OpenPose neural network architecture to detect the human poses in the videos (Cao et al., 2017).”
MethodOpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields
Cao, Z., Hidalgo, G., Simon, T., Wei, S.-E., & Sheikh, Y. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2019.2929257 · Cited by 5,536
DeepLabCut: markerless pose estimation of user-defined body parts with deep learning
Mathis, A., Mamidanna, P., Cury, K. M., Abe, T., Murthy, V. N., Mathis, M. W., & Bethge, M. (2018) · Nature Neuroscience · doi:10.1038/s41593-018-0209-y · Cited by 5,187 · cited 9×
“The other two approaches (Mathis et al., 2018; Pereira et al., 2018) showed successful implementation of the CNNs to behavior imaging data analysis, both of them focusing on body pose estimation in animals.”
BackgroundMethodResultInfluentialOn aims and methods of ethology
Tinbergen, N. (1963) · Zeitschrift für Tierpsychologie · doi:10.1111/j.1439-0310.1963.tb01161.x · Cited by 4,551
Wei, S.-E., Ramakrishna, V., Kanade, T., & Sheikh, Y. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.511 · Cited by 2,966
Moy, S. S., Nadler, J. J., Perez, A., Barbaro, R. P., Johns, J. M., Magnuson, T. R., Piven, J., & Crawley, J. N. (2004) · Genes, Brain and Behavior · doi:10.1111/j.1601-1848.2004.00076.x · Cited by 1,522
Neuroscience Needs Behavior: Correcting a Reductionist Bias
Krakauer, J. W., Ghazanfar, A. A., Gomez-Marin, A., MacIver, M. A., & Poeppel, D. (2017) · Neuron · doi:10.1016/j.neuron.2016.12.041 · Cited by 1,358 · cited 2×
“In contrast, the development of behavioral analysis has lagged, with indirect measurements and a reductionist approach (Krakauer et al., 2017).”
BackgroundMapping Sub-Second Structure in Mouse Behavior
Wiltschko, A. B., Johnson, M. J., Iurilli, G., Peterson, R. E., Katon, J. M., Pashkovski, S. L., Abraira, V. E., Adams, R. P., & Datta, S. R. (2015) · Neuron · doi:10.1016/j.neuron.2015.11.031 · Cited by 760
End-to-End People Detection in Crowded Scenes
Stewart, R., Andriluka, M., & Ng, A. Y. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.255 · Cited by 539 · cited 4×
“These “.json” files and the folders of corresponding raw images are then used as the training and test datasets for the GoogLeNet network (Stewart et al., 2016).”
BackgroundMethodInfluentialSkilled reaching relies on a V2a propriospinal internal copy circuit
Azim, E., Jiang, J., Alstermark, B., & Jessell, T. M. (2014) · Nature · doi:10.1038/nature13021 · Cited by 355
Discovery of brainwide neural-behavioral maps via multiscale unsupervised structure learning
Vogelstein, J. T., Park, Y., Ohyama, T., Kerr, R. A., Truman, J. W., Priebe, C. E., & Zlatic, M. (2014) · Science · doi:10.1126/science.1250298 · Cited by 218 · cited 1×
“Other studies have used machine vision techniques with unsupervised data analysis (Vogelstein et al., 2014; Robie et al., 2017).”
MethodRobotic Measurement of Arm Movements After Stroke Establishes Biomarkers of Motor Recovery
Krebs, H. I., Krams, M., Agrafiotis, D. K., DiBernardo, A., Chavez, J. C., Littman, G. S., Yang, E., Byttebier, G., Dipietro, L., Rykman, A., et al. (2014) · Stroke · doi:10.1161/strokeaha.113.002296 · Cited by 154 · cited 1×
“Various sensors, reflective markers, external devices, or robotics have been used to perform kinematic analyses (Krebs et al., 2014).”
Method
Cited by
Deep learning tools for the measurement of animal behavior in neuroscience
Mathis, M. W., & Mathis, A. (2020) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2019.10.008 · Cited by 450
A Primer on Motion Capture with Deep Learning: Principles, Pitfalls, and Perspectives
Mathis, A., Schneider, S., Lauer, J., & Mathis, M. W. (2020) · Neuron · doi:10.1016/j.neuron.2020.09.017 · Cited by 231
Abbas, W., & Masip Rodo, D. (2019) · Sensors · doi:10.3390/s19153274 · Cited by 21
“[119] Research code available at https://bit.”
BackgroundBalestra, N., Sharma, G., Riek, L. M., & Busza, A. (2021) · Digital Biomarkers · doi:10.1159/000516619 · Cited by 13
Metadata
- Authors
- Arac, A., Zhao, P., Dobkin, B. H., Carmichael, S. T., & Golshani, P.
- Venue
- Frontiers in Systems Neuroscience
- Publisher
- Frontiers Media SA
- Published
- 2019-05-07
- Volume
- 13
- Pages
- 20-20
- ISSN
- 1662-5137 (ISSN-L)
- OpenAlex
- W2944054712
- Cited by
- 113
- References
- 16
- FWCI
- 7.04
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.