CoChat
Sign Up
Cited by 167Human Pose and Action RecognitionNeurobiology and Insect Physiology ResearchCell Image Analysis Techniques

DeepFly3D, a deep learning-based approach for 3D limb and appendage tracking in tethered, adult Drosophila

Open access
Günel, Semih, Rhodin, Helge, Morales, Daniel, Campagnolo, João, Ramdya, Pavan, Fua, Pascal·2019·eLife
DOI 10.7554/elife.48571PMC PMC6828327

Published version of a preprint

Open this paper in CoChat — free account

Abstract

Studying how neural circuits orchestrate limbed behaviors requires the precise measurement of the positions of each appendage in three-dimensional (3D) space. Deep neural networks can estimate two-dimensional (2D) pose in freely behaving and tethered animals. However, the unique challenges associated with transforming these 2D measurements into reliable and precise 3D poses have not been addressed for small animals including the fly, Drosophila melanogaster. Here, we present DeepFly3D, a software that infers the 3D pose of tethered, adult Drosophila using multiple camera images. DeepFly3D does not require manual calibration, uses pictorial structures to automatically detect and correct pose estimation errors, and uses active learning to iteratively improve performance. We demonstrate more accurate unsupervised behavioral embedding using 3D joint angles rather than commonly used 2D pose data. Thus, DeepFly3D enables the automated acquisition of Drosophila behavioral measurements at an unprecedented level of detail for a variety of biological applications.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.7554/elife.48571

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Central Pattern Generators (CPGs) are neural circuits that generate rhythmic output without receiving rhythmic input.

    “Central Pattern Generators (CPGs) are neural circuits that generate rhythmic output without receiving rhythmic input 58 .”

    From NeuroMechFly v2: simulating embodied sensorimotor control in adult Drosophila (2024) · cites this paper as [58]

    The Results section discusses optogenetically elicited walking and grooming, while Materials and methods analyzes periodic limb motion. Neither defines central pattern generators or addresses whether they produce rhythmic output without rhythmic input. Searches for “central pattern generator” and “CPG rhythmic output” found no relevant passage in our copy; page numbers were not provided. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

  • Two prominent mechanisms have been proposed for insect locomotion: a centralized mechanism that depends on coupled CPGs, and a more decentralized approach whereby locomotion emerges from the application of sensory feedback-based rules that dictate for each leg when to lift, swing, or remain in stance phase.

    “Two prominent mechanisms have been proposed for insect locomotion. The first is a centralized mechanism that depends on coupled CPGs (i.e., circuits producing rhythmic motor output without rhythmic input 58 ). The second is a more decentralized approach whereby locomotion emerges from the application of sensory feedback-based rules that dictate for each leg when to lift, swing, or remain in stance”

    From NeuroMechFly v2: simulating embodied sensorimotor control in adult Drosophila (2024) · cites this paper as [58]

    Split into 2 atoms — 0 confirmed, 0 partially supported, 0 disputed, 0 misattributed, 2 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • Evidence for each of these control approaches has been found across species, motivating their application toward robotic motor control.

    “Evidence for each of these control approaches has been found across species 58,59,60,61 , motivating their application toward robotic motor control 62,63 .”

    From NeuroMechFly v2: simulating embodied sensorimotor control in adult Drosophila (2024) · cites this paper as [58]

    The Results section on behavioral classification describes tracking walking and grooming in tethered fruit flies, while the Discussion says the tracking method could be adapted to other animals. Neither passage provides evidence for the unspecified motor-control approaches across species. The robotic-motor-control portion is attached to different citations (62, 63). (LLM, conf=0.72)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 42

  1. Andriluka M Pishchulin L Gehler P Schiele B 2014 2d human pose estimation: new benchmark and state of the art analysis Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 3686 3693 10.1109/CVPR.2014.471 · DOI
  2. Bender JA Simpson EM Ritzmann RE 2010 Computer-assisted 3D kinematic analysis of all leg joints in walking insects PLOS ONE 5 e13617 10.1371/journal.pone.0013617 21049024 PMC2964314 · DOI
  3. Berman GJ Choi DM Bialek W Shaevitz JW 2014 Mapping the stereotyped behaviour of freely moving fruit flies Journal of the Royal Society Interface 11 20140672 10.1098/rsif.2014.0672 PMC4233753 25142523 · DOI · Paper page
  4. Bishop CM 2006 Pattern Recognition and Machine Learning Springer
  5. Casiez G Roussel N Vogel D 2012 1€ filter: a simple speed-based low-pass filter for noisy input in interactive systems Proceedings of the SIGCHI Conference on Human Factors in Computing Systems ACM 2527 2530
  6. Chavdarova T Baqué P Bouquet S Maksai A Jose C Lettry L Fua P Gool LV Fleuret F 2018 WILDTRACK: A Multi-Camera HD Dataset for Dense Unscripted Pedestrian Detection The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 5030 5039
  7. Chen CL Hermans L Viswanathan MC Fortun D Aymanns F Unser M Cammarato A Dickinson MH Ramdya P 2018 Imaging neural activity in the ventral nerve cord of behaving adult Drosophila Nature Communications 9 4390 10.1038/s41467-018-06857-z 30348941 PMC6197219 · DOI · Paper page
  8. Dombeck DA Khabbaz AN Collman F Adelman TL Tank DW 2007 Imaging large-scale neural activity with cellular resolution in awake, mobile mice Neuron 56 43 57 10.1016/j.neuron.2007.08.003 17920014 PMC2268027 · DOI · Paper page
  9. Elhayek A Aguiar E Jain A Tompson J Pishchulin L Andriluka M Bregler C Schiele B Theobalt C 2015 Efficient Convnet-Based Marker-Less motion capture in general scenes with a low number of cameras IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 10.1109/CVPR.2015.7299005 · DOI
  10. Feany MB Bender WW 2000 A Drosophila model of Parkinson's disease Nature 404 394 398 10.1038/35006074 10746727 · DOI
  11. Felzenszwalb PF Huttenlocher DP 2005 Pictorial structures for object recognition International Journal of Computer Vision 61 55 79 10.1023/B:VISI.0000042934.15159.49 · DOI
  12. Hartley R Zisserman A 2000 Multiple View Geometry in Computer Vision Cambridge University Press
  13. Hewitt VL Whitworth AJ 2017 Mechanisms of Parkinson's Disease: Lessons from Drosophila Current Topics in Developmental Biology 121 173 200 10.1016/bs.ctdb.2016.07.005 28057299 · DOI
  14. Ionescu C Li F Sminchisescu C 2011 Latent structured models for human pose estimation 2011 International Conference on Computer Vision IEEE 2220 2227 10.1109/ICCV.2011.6126500 · DOI
  15. Isakov A Buchanan SM Sullivan B Ramachandran A Chapman JK Lu ES Mahadevan L de Bivort B 2016 Recovery of locomotion after injury in Drosophila Melanogaster depends on proprioception The Journal of Experimental Biology 219 1760 1771 10.1242/jeb.133652 26994176 · DOI
  16. Kain J Stokes C Gaudry Q Song X Foley J Wilson R De Bivort B 1910 Leg-tracking and automated behavioural classification in Drosophila Nature Communications 2013 4 10.1038/ncomms2908 PMC3674277 23715269 · DOI
  17. Maaten LJP 2008 Visualizing High Dimensional Data Using t-SNE Journal of Machine Learning Research 2579 2605
  18. Martinez J Hossain R Romero J Little JJ 2017 A simple yet effective baseline for 3D human pose estimation ICCV
  19. Mathis A Mamidanna P Cury KM Abe T Murthy VN Mathis MW Bethge M 2018 DeepLabCut: markerless pose estimation of user-defined body parts with deep learning Nature Neuroscience 21 1281 1289 10.1038/s41593-018-0209-y 30127430 · DOI · Paper page
  20. McKellar CE Lillvis JL Bath DE Fitzgerald JE Cannon JG Simpson JH Dickson BJ 2019 Threshold-Based ordering of sequential actions during Drosophila courtship Current Biology 29 426 434 10.1016/j.cub.2018.12.019 30661796 · DOI · Paper page
  21. Mehta D Sridhar S Sotnychenko O Rhodin H Shafiei M Seidel H Xu W Casas D Theobalt C 2017 Vnect: Real-Time3D Human Pose Estimation with a Single RGB Camera SIGGRAPH
  22. Mendes CS Bartos I Akay T Márka S Mann RS 2013 Quantification of gait parameters in freely walking wild type and sensory deprived Drosophila Melanogaster eLife 2 e00231 10.7554/eLife.00231 23326642 PMC3545443 · DOI · Paper page
  23. Moeslund TB Granum E 2000 Multiple cues used in model-based human motion capture. Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580) 362 367 10.1109/AFGR.2000.840660 · DOI
  24. Moreno-noguer F 2017 3d human pose estimation from a single image via distance matrix regression CVPR
  25. Nath T Mathis A Chen AC Patel A Bethge M Mathis MW 2019 Using DeepLabCut for 3D markerless pose estimation across species and behaviors Nature Protocols 14 2152 2176 10.1038/s41596-019-0176-0 31227823 · DOI
  26. Pavlakos G Zhou X Derpanis K Konstantinos G Daniilidis K 2017a Coarse-To-Fine volumetric prediction for Single-Image 3D human pose CVPR
  27. Pavlakos G Zhou X Konstantinos KDG Kostas D 2017b Harvesting multiple views for Marker-Less 3D human pose annotations In: CVPR
  28. Pereira TD Aldarondo DE Willmore L Kislin M Wang SS Murthy M Shaevitz JW 2019 Fast animal pose estimation using deep neural networks Nature Methods 16 117 125 10.1038/s41592-018-0234-5 30573820 PMC6899221 · DOI
  29. Popa AI Zanfir M Sminchisescu C 2017 Deep multitask architecture for integrated 2D and 3D human sensing In: CVPR
  30. Rhodin H Robertini N Casas D Richardt C Seidel HP Theobalt C 2016 General automatic human shape and motion capture using volumetric contour cues ECCV
  31. Rogez G Weinzaepfel P Schmid C 2017 Lcr-Net: localization-classification-regression for human pose In: CVPR
  32. Seeds AM Ravbar P Chung P Hampel S Midgley FM Mensh BD Simpson JH 2014 A suppression hierarchy among competing motor programs drives sequential grooming in Drosophila eLife 3 e02951 10.7554/eLife.02951 25139955 PMC4136539 · DOI · Paper page
  33. Seelig JD Chiappe ME Lott GK Dutta A Osborne JE Reiser MB Jayaraman V 2010 Two-photon calcium imaging from head-fixed Drosophila during optomotor walking behavior Nature Methods 7 535 540 10.1038/nmeth.1468 20526346 PMC2945246 · DOI
  34. Simon T Joo H Matthews I Sheikh Y 2017 Hand keypoint detection in single images using multiview bootstrapping In: CVPR
  35. Sun X Shang J Liang S Wei Y 2017 Compositional human pose regression ICCV
  36. Takahashi K Mikami D Isogawa M Kimata H 2018 Human pose as calibration pattern; 3D human pose estimation with multiple unsynchronized and uncalibrated cameras The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops
  37. Tekin B Marquez-neila P Salzmann M Fua P 2017 Learning to fuse 2D and 3D image cues for monocular body pose estimation ICCV
  38. Todd JG Kain JS de Bivort BL 2017 Systematic exploration of unsupervised methods for mapping behavior Physical Biology 14 015002 10.1088/1478-3975/14/1/015002 28166059 · DOI
  39. Tome D Russell C Agapito L 2017 Lifting from the deep: convolutional 3D pose estimation from a single image arXiv https://arxiv.org/abs/1701.00295 · arXiv
  40. Triggs B Mclauchlan P Hartley R Fitzgibbon A 2000 Vision Algorithms: Theory and Practice Springer
  41. Uhlmann V Ramdya P Delgado-Gonzalo R Benton R Unser M 2017 FlyLimbTracker: an active contour based approach for leg segment tracking in unmarked, freely behaving Drosophila PLOS ONE 12 e0173433 10.1371/journal.pone.0173433 28453566 PMC5409058 · DOI
  42. Zhou X Huang Q Sun X Xue X Wei Y 2017 Weakly-supervised transfer for 3d human pose estimation in the wild IEEE International Conference on Computer Vision

Cited by

Sorted by citations

Authors
Günel, S., Rhodin, H., Morales, D., Campagnolo, J., Ramdya, P., & Fua, P.
Affiliations
  • Computer Vision Laboratory (CH)
  • University of British Columbia (CA)
  • École Polytechnique Fédérale de Lausanne (CH)
Corresponding author
  • Günel, Semih
  • Ramdya, Pavan
ORCID
Venue
eLife
Publisher
eLife Sciences Publications, Ltd
Published
2019-10-04
Volume
8
ISSN
2050-084X (ISSN-L)
DOI
10.7554/elife.48571
PMC
PMC6828327
OpenAlex
W2977639561
Cited by
167
References
7
Citations by year
2026: 14 · 2025: 14 · 2024: 19 · 2023: 28 · 2022: 32 · 2021: 33 · 2020: 24 · 2019: 3
FWCI
10.17
Type
journal article
Language
en
Open access
Licence
Locations
  • eLife (publishedVersion, cc-by)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Infoscience (Ecole Polytechnique Fédérale de Lausanne) (submittedVersion, cc-by-nc-nd)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Publikationen an der Universität Bielefeld (Universität Bielefeld) (submittedVersion)
  • PUB – Publications at Bielefeld University (Bielefeld University) (submittedVersion)
Funders
  • Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung (175667, 181239)
  • EPFL (iPhD)
  • Microsoft Research (JRC Project)
  • Swiss Government Excellence Postdoctoral Scholarship (2018.0483)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
  • Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neurobiology and Insect Physiology Research
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Biophysics › Cell Image Analysis Techniques
Keywords
pictorial structures, active learning (machine learning)
MeSH terms
Deep Learning, Animals, Behavior, Animal, Drosophila, Extremities, Movement, Software, Imaging, Three-Dimensional, Optical Imaging

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review