Cited by 231Human Pose and Action RecognitionHuman Motion and AnimationMotor Control and Adaptation

A Primer on Motion Capture with Deep Learning: Principles, Pitfalls, and Perspectives

Open access
Mathis, Alexander, Schneider, Steffen, Lauer, Jessy, Mathis, Mackenzie Weygandt·2020·Neuron

Abstract

Extracting behavioral measurements non-invasively from video is stymied by the fact that it is a hard computational problem. Recent advances in deep learning have tremendously advanced our ability to predict posture directly from videos, which has quickly impacted neuroscience and biology more broadly. In this primer, we review the budding field of motion capture with deep learning. In particular, we will discuss the principles of those novel algorithms, highlight their potential as well as pitfalls for experimentalists, and provide a glimpse into the future.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.neuron.2020.09.017

Cites

202 of 202 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Various user-friendly toolboxes are available to extract the 2D posture of animals from videos, while the user can define which body parts should be estimated.

    “Various user-friendly toolboxes are available to extract the 2D posture of animals from videos (Fig. 1d, e), while the user can define which body parts should be estimated (reviewed in ref. 76).”

    From Perspectives in machine learning for wildlife conservation (2022) · cites this paper as [76]

  • Modern ML has rapidly changed laboratory-based non-invasive observation of animals.

    “In recent years, modern ML has rapidly changed laboratory-based non-invasive observation of animals76,105.”

    From Perspectives in machine learning for wildlife conservation (2022) · cites this paper as [76]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 9

  1. [17]17. Gary Kane, Gonçalo Lopes, Jonny L. Saunders, Alexander Mathis, and Mackenzie Mathis. Real-time deeplabcut for closed-loop feedback based on posture. bioRxiv , 2020.
  2. [39]39. Alexander Mathis, Mert Yüksekgönül, Byron Rogers, Matthias Bethge, and Mackenzie W Mathis. Pretraining boosts outof-domain robustness for pose estimation. arXiv preprint arXiv:1909.11229 , 2019. · arXiv
  3. [66]66. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, highperformance deep learning library. In Advances in neural information processing systems , pages 8026-8037, 2019.
  4. [95]95. Joshua H Siegle, Xiaoxuan Jia, Séverine Durand, Sam Gale, Corbett Bennett, Nile Graddis, Greggory Heller, Tamina K Ramirez, Hannah Choi, Jennifer A Luviano, et al. A survey of spiking activity reveals a functional hierarchy of mouse corticothalamic visual areas. bioRxiv , page 805010, 2019.
  5. [139]139. Alexander B Wiltschko, Matthew J Johnson, Giuliano Iurilli, Ralph E Peterson, Jesse M Katon, Stan L Pashkovski, Victoria E Abraira, Ryan P Adams, and Sandeep Robert Datta. Mapping subsecond structure in mouse behavior. Neuron , 88(6):1121-1135, 2015.
  6. [146]146. Kai J Sandbrink, Pranav Mamidanna, Claudio Michaelis, Mackenzie Weygandt Mathis, Matthias Bethge, and Alexander Mathis. Task-driven hierarchical deep neural networkmodels of the proprioceptive pathway. bioRxiv , 2020.
  7. [160]160. Mirco Ravanelli, Jianyuan Zhong, Santiago Pascual, Pawel Swietojanski, Joao Monteiro, Jan Trmal, and Yoshua Bengio. Multitask self-supervised learning for robust speech recognition. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 6989-6993. IEEE, 2020.
  8. [170]170. Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pretraining can improve model robustness and uncertainty. In ICML , 2019.
  9. [173]173. Cody Coleman, Deepak Narayanan, Daniel Kang, Tian Zhao, Jian Zhang, Luigi Nardi, Peter Bailis, Kunle Olukotun, Chris Ré, and Matei Zaharia. Dawnbench: An end-to-end deep learning benchmark and competition. Neural Information Processing Systems Workshops , 2017.

Cited by

Sorted by citations

Authors
Mathis, A., Schneider, S., Lauer, J., & Mathis, M. W.
Affiliations
Harvard University (US)
Corresponding author
  • Mathis, Alexander
  • Mathis, Mackenzie Weygandt
Venue
Neuron
Publisher
Elsevier BV
Published
2020-10
Volume
108
Issue
1
Pages
44-65
ISSN
0896-6273 (ISSN-L)
OpenAlex
W3081605501
Cited by
231
References
202
Citations by year
2026: 18 · 2025: 38 · 2024: 50 · 2023: 40 · 2022: 49 · 2021: 32 · 2020: 4
FWCI
2.54
Type
journal article
Language
en
Open access
Locations
  • Neuron (publishedVersion)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University)
Funders
Chan Zuckerberg Initiative
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
  • Physical Sciences › Engineering › Control and Systems Engineering › Human Motion and Animation
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Motor Control and Adaptation
Keywords
motion capture, deep learning, pose estimation, animal behavior
MeSH terms
Deep Learning, Algorithms, Animals, Humans, Motion, Motor Activity, Movement, Video Recording, Neural Networks, Computer

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review