Gimme Signals: Discriminative signal encoding for multimodal activity recognition
Open accessThe DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
We present a simple, yet effective and flexible method for action recognition supporting multiple sensor modalities. Multivariate signal sequences are encoded in an image and are then classified using a recently proposed EfficientNet CNN architecture. Our focus was to find an approach that generalizes well across different sensor modalities without specific adaptions while still achieving good results. We apply our method to 4 action recognition datasets containing skeleton sequences, inertial and motion capturing measurements as well as Wi-Fi fingerprints that range up to 120 action classes. Our method defines the current best CNN-based approach on the NTU RGB+D 120 dataset, lifts the state of the art on the ARIL Wi-Fi dataset by +6.8%, improves the UTD-MHAD inertial baseline by +14.4%, the UTD-MHAD skeleton baseline by +0.5% and achieves 96.1% on the Simitate motion capturing data (80/20 split). We further demonstrate experiments on both, modality fusion on a signal level and signal reduction to prevent the representation from overloading.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Recent advances in convolutional neural networks
Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., Liu, T., Wang, X., Wang, G., Cai, J., et al. (2018) · Pattern Recognition · doi:10.1016/j.patcog.2017.10.013 · Cited by 6,576
Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition
Yan, S., Xiong, Y., & Lin, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12328 · Cited by 5,410 · cited 1×
“lutional [41] networks.”
NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis
Shahroudy, A., Liu, J., Ng, T.-T., & Wang, G. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.115 · Cited by 3,148
A New Representation of Skeleton Sequences for 3D Action Recognition
Ke, Q., Bennamoun, M., An, S., Sohel, F., & Boussaid, F. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.486 · Cited by 793 · cited 1×
“Most action recognition approaches based on CNNs present custom architecture designs in their pipelines [34], [45].”
BackgroundA Comprehensive Survey of Vision-Based Human Action Recognition Methods
Zhang, H.-B., Zhang, Y.-X., Zhong, B., Lei, Q., Yang, L., Du, J.-X., & Chen, D.-S. (2019) · Sensors · doi:10.3390/s19051005 · Cited by 503 · cited 2×
“0% in [21] was falsely reported on a similar named dataset.”
BackgroundMFAS: Multimodal Fusion Architecture Search
Perez-Rua, J.-M., Vielzeuf, V., Pateux, S., Baccouche, M., & Jurie, F. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00713 · Cited by 193
Skeleton Image Representation for 3D Action Recognition Based on Tree Structure and Reference Joints
Caetano, C., Bremond, F., & Schwartz, W. R. (2019) · 2019 32nd SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI) · doi:10.1109/sibgrapi.2019.00011 · Cited by 149 · cited 2×
“[31], [32] represent a combination of reference joints and a tree-structured skeleton in images.”
BackgroundMethod
Cited by
Human Action Recognition From Various Data Modalities: A Review
Sun, Z., Ke, Q., Rahmani, H., Bennamoun, M., Wang, G., & Liu, J. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3183112 · Cited by 601
Boulahia, S. Y., Amamra, A., Madi, M. R., & Daikh, S. (2021) · Machine Vision and Applications · doi:10.1007/s00138-021-01249-8 · Cited by 236
Evaluation of an Audio-Video Multimodal Deepfake Dataset using Unimodal and Multimodal Detectors
Khalid, H., Kim, M., Tariq, S., & Woo, S. S. (2021) · Proceedings of the 1st Workshop on Synthetic Multimedia - Audiovisual Deepfake Generation and Detection · doi:10.1145/3476099.3484315 · Cited by 110
Javeed, M., Mudawi, N. A., Alabduallah, B. I., Jalal, A., & Kim, W. (2023) · Sensors · doi:10.3390/s23104716 · Cited by 18
Metadata
- Authors
- Memmesheimer, R., Theisen, N., & Paulus, D.
- Venue
- 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
- Publisher
- IEEE
- Published
- 2020-10-24
- Pages
- 10394-10401
- OpenAlex
- W3011747142
- Cited by
- 61
- References
- 8
- FWCI
- 3.24
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Gimme Signals: Discriminative signal encoding for multimodal activity recognition. https://app.cochat.ai/papers/10.1109/iros45743.2020.9341699
- The paper
- Memmesheimer, R., Theisen, N., & Paulus, D. (2020). Gimme Signals: Discriminative signal encoding for multimodal activity recognition. 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). https://doi.org/10.1109/iros45743.2020.9341699
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.