CoChat

ModDrop: Adaptive Multi-Modal Gesture Recognition

Open access
Neverova, Natalia, Wolf, Christian, Taylor, Graham, Nebout, Florian · 2016 · IEEE Transactions on Pattern Analysis and Machine Intelligence

The DOI matches its Crossref record; 71 of the 77 references it cites were found in a registry or an index; 6 of the 18 sources its claims cite were checked for support.

Abstract

We present a method for gesture detection and localisation based on multi-scale and multi-modal deep learning. Each visual modality captures spatial information at a particular spatial scale (such as motion of the upper body or a hand), and the whole system operates at three temporal scales. Key to our technique is a training strategy which exploits: i) careful initialization of individual modalities; and ii) gradual fusion involving random dropping of separate channels (dubbed ModDrop) for learning cross-modality correlations while preserving uniqueness of each modality-specific representation. We present experiments on the ChaLearn 2014 Looking at People Challenge gesture recognition track, in which we placed first out of 17 teams. Fusing multiple modalities at several spatial and temporal scales leads to a significant increase in recognition rates, allowing the model to compensate for errors of the individual classifiers as well as noise in the separate channels. Furthermore, the proposed ModDrop training technique ensures robustness of the classifier to missing signals in one or several channels to produce meaningful predictions from any number of available modalities. In addition, we demonstrate the applicability of the proposed fusion scheme to modalities of arbitrary nature by experiments on the same dataset augmented with audio.

Overview

CoChat Summary

TL;DR

The paper presents ModDrop, a multi-modal and multi-scale deep learning method for gesture detection and localization that placed first in the ChaLearn 2014 LAP Challenge track 3. ModDrop trains shared fusion layers with random modality dropping, improving recognition and making predictions robust to missing or corrupted channels.

Key findings

  • The method placed first out of 17 teams in the ChaLearn 2014 LAP Challenge track 3, with the submitted hybrid model scoring 0.850 Jaccard index and surpassing the second-best score by 1.61 percentage points; post-challenge improvements raised the score to 0.870.
  • On ChaLearn 2014 LAP augmented with audio, ModDrop improved robustness to missing channels: with mocap missing, accuracy rose from 38.41% to 92.82% and Jaccard from 0.306 to 0.859; with both hands missing, accuracy rose from 53.13% to 73.28% and Jaccard from 0.466 to 0.680.
  • On an MNIST multi-modal sanity check, ModDrop reduced test error under occlusion: 1 segment covered 2.30% vs 10.74%, 2 segments covered 7.19% vs 35.91%, and 3 segments covered 24.88% vs 68.03%.
  • Adding audio improved Jaccard by 1.3 points, and learned audio representations outperformed a phoneme-recognition baseline (F-measure 79.71% vs 56.50%; Jaccard 0.545 vs 0.256).

Methods

  • A multi-scale, multi-modal convolutional architecture uses parallel single-scale paths over dynamic poses of 5 frames sampled with a given temporal stride, integrating depth and intensity video, articulated pose (183-dimensional per frame; 915-dimensional dynamic pose), and optionally audio.
  • Training proceeds by per-modality pretraining, initialization of shared fusion layers to a normalized geometric mean of per-model outputs, and gradual fusion with ModDrop, which randomly drops separate modality channels during training.
  • Experiments use the ChaLearn 2014 Looking at People Challenge track 3 dataset (13,858 gesture instances, 20 gesture categories), augmented with audio from the ChaLearn 2013 Multi-modal Challenge, plus MNIST sanity checks; performance is measured mainly by Jaccard index.

Limitations

  • Stated: audio alone does not allow robust gesture start/end localization because of natural gesture-speech desynchronization; annotations are video-based while spoken words/phrases are typically shorter, and vocalization may be delayed or ahead of the gesture, yielding poor audio-only Jaccard scores.
  • Evident from the text: evaluation centers on the ChaLearn 2014 LAP dataset plus MNIST sanity checks and ChaLearn 2013 audio augmentation; hyperparameters and ModDrop/dropout rates were optimized on validation data, and hidden-unit dropout degraded performance due to the model’s compactness.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/tpami.2015.2461544

Cites

71 of 71 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Overview of our method on an example from the 2014 ChaLearn Looking at People (LAP) dataset.
Fig. 1 Overview of our method on an example from the 2014 ChaLearn Looking at People (LAP) dataset.From the PDF
Fig. 2 The ModDrop network operating at 3 temporal scales corresponding to 3 durations of dynamic poses.
Fig. 2 The ModDrop network operating at 3 temporal scales corresponding to 3 durations of dynamic poses.From the PDF
Image 3
From the PDF
Fig. 4 Mel-scaled spectrograms of two pairs of audio samples corresponding to two different gestures.
Fig. 4 Mel-scaled spectrograms of two pairs of audio samples corresponding to two different gestures.From the PDF
Fig. 5 On the left: architecture of shared hidden and output layers. On the right: structure of parameters of shared hidden and output layers (corresponds to the architecture on the left).
Fig. 5 On the left: architecture of shared hidden and output layers. On the right: structure of parameters of shared hidden and output layers (corresponds to the architecture on the left).From the PDF
Fig. 6 Toy network architecture and notations used for derivation of ModDrop regularization properties.
Fig. 6 Toy network architecture and notations used for derivation of ModDrop regularization properties.From the PDF
Fig. 7 'Multi-modal' setting for the MNIST dataset.
Fig. 7 'Multi-modal' setting for the MNIST dataset.From the PDF
Image 8
From the PDF
Image 9
From the PDF
Image 10
From the PDF
Image 11
From the PDF

Verification

Claims the paper makes

5 Valid · 4 Not valid · 13 Not checked · 74 Own results and interpretations

46 more claims

Create a free account to see all 96 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 34
  • Probably not a substantial claim. 19

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Research works printed but not linked: 3

  1. lee, 0, Julius—an open source real-time large vocabulary recognition engine, Proc INTERSPEECH
  2. bergstra, 0, Theano: A CPU and GPU math expression compiler, Proc Scipy Conf
  3. 10.1109/5.726791 · DOI · Paper page

Cited by

See all 420

Metadata

Authors
Neverova, N., Wolf, C., Taylor, G., & Nebout, F.
Affiliations
  • Centre National de la Recherche Scientifique (FR)
  • University of Guelph (CA)
Venue
IEEE Transactions on Pattern Analysis and Machine Intelligence
Published
2016-08-01
Volume
38
Issue
8
Pages
1692-1706
ISSN
  • 0162-8828 (ISSN-L)
  • 2160-9292
OpenAlex
W1533025524
Cited by
420
References
77
Citations by year
2026: 89 · 2025: 28 · 2024: 28 · 2023: 35 · 2022: 24 · 2021: 44 · 2020: 41 · 2019: 44 · 2018: 34 · 2017: 34 · 2016: 14
FWCI
55.12
Type
journal article
Language
en
Open access
Locations
  • IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • HAL (submittedVersion, other-oa)
  • arXiv (Cornell University)
Funders
  • Investissement's d’Avenir
  • Briques Génériques du Logiciel Embarqué
Topics
  • Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
Keywords
gesture recognition, multimodal fusion, multi-scale deep learning, spatial and temporal scales, missing modality robustness, upper body movement, hand movement

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.