Synthesizing and Reconstructing Missing Sensory Modalities in Behavioral Context Recognition
Open accessThe DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
Detection of human activities along with the associated context is of key importance for various application areas, including assisted living and well-being. To predict a user’s context in the daily-life situation a system needs to learn from multimodal data that are often imbalanced, and noisy with missing values. The model is likely to encounter missing sensors in real-life conditions as well (such as a user not wearing a smartwatch) and it fails to infer the context if any of the modalities used for training are missing. In this paper, we propose a method based on an adversarial autoencoder for handling missing sensory features and synthesizing realistic samples. We empirically demonstrate the capability of our method in comparison with classical approaches for filling in missing values on a large-scale activity recognition dataset collected in-the-wild. We develop a fully-connected classification network by extending an encoder and systematically evaluate its multi-label classification performance when several modalities are missing. Furthermore, we show class-conditional artificial data generation and its visual and quantitative analysis on context classification task; representing a strong generative power of adversarial autoencoders.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep learning for sensor-based activity recognition: A survey
Wang, J., Chen, Y., Hao, S., Peng, X., & Hu, L. (2019) · Pattern Recognition Letters · doi:10.1016/j.patrec.2018.02.010 · Cited by 2,014
The prevention and handling of the missing data
Kang, H. (2013) · Korean Journal of Anesthesiology · doi:10.4097/kjae.2013.64.5.402 · Cited by 1,728 · cited 2×
“, if the number of missing values for different features are excessively unequal and vary over time) [9].”
BackgroundMiotto, R., Li, L., Kidd, B. A., & Dudley, J. T. (2016) · Scientific Reports · doi:10.1038/srep26094 · Cited by 1,710 · cited 1×
“[31] applied DAE to produce compressed embedding of patients’ characteristics from a very large and noisy set of electronic health records.”
MethodA survey on ambient-assisted living tools for older adults
Rashidi, P., & Mihailidis, A. (2013) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2012.2234129 · Cited by 1,128
Large-scale physical activity data reveal worldwide activity inequality
Althoff, T., Sosič, R., Hicks, J. L., King, A. C., Delp, S. L., & Leskovec, J. (2017) · Nature · doi:10.1038/nature23018 · Cited by 1,000
A survey of depth and inertial sensor fusion for human action recognition
Chen, C., Jafari, R., & Kehtarnavaz, N. (2017) · Multimedia Tools and Applications · doi:10.1007/s11042-015-3177-1 · Cited by 356
Feature Learning for Activity Recognition in Ubiquitous Computing
Plötz, T., Hammerla, N., & Olivier, P. (2011) · Newcastle University · doi:10.25405/ncl.34175319 · Cited by 337
Recognizing Detailed Human Context in the Wild from Smartphones and Smartwatches
Vaizman, Y., Ellis, K., & Lanckriet, G. (2017) · IEEE Pervasive Computing · doi:10.1109/mprv.2017.3971131 · Cited by 324
Learning deep physiological models of affect
Martinez, H. P., Bengio, Y., & Yannakakis, G. N. (2013) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2013.2247823 · Cited by 287 · cited 1×
“[32] applied a convolutional autoencoder on raw physiological signals to extract salient features for affect modeling of game players.”
BackgroundUniversum Autoencoder-Based Domain Adaptation for Speech Emotion Recognition
Deng, J., Xu, X., Zhang, Z., Fruhholz, S., & Schuller, B. (2017) · IEEE Signal Processing Letters · doi:10.1109/lsp.2017.2672753 · Cited by 125
Vaizman, Y., Weibel, N., & Lanckriet, G. (2018) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3161192 · Cited by 122
Cited by
Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances
Zhang, S., Li, Y., Zhang, S., Shahabi, F., Xia, S., Deng, Y., & Alshurafa, N. (2022) · Sensors · doi:10.3390/s22041476 · Cited by 625
“Usually, the considered corruption includes isotropic Gaussian Figure 4: Illustration of an autoencoder network [237].”
BackgroundMulti-task Self-Supervised Learning for Human Activity Detection
Saeed, A., Ozcelebi, T., & Lukkien, J. (2019) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3328932 · Cited by 346
“Deep learning has been increasingly used for end-to-end HAR with far superior performance that can be achieved through traditional machine learning methods [20, 44, 56, 61, 72].”
BackgroundMethodSelf-citationSpatial-Temporal Masked Autoencoder for Multi-Device Wearable Human Activity Recognition
Miao, S., Chen, L., & Hu, R. (2023) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3631415 · Cited by 23
Handling Missing Sensors in Topology-Aware IoT Applications with Gated Graph Neural Network
Liu, S., Yao, S., Huang, Y., Liu, D., Shao, H., Zhao, Y., Li, J., Wang, T., Wang, R., Yang, C., et al. (2020) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3411818 · Cited by 20
“Meanwhile, several neural network based general missing sensor handling algorithms (i.e., algorithms where one model handles all missing sensor situations) have been proposed [24, 29, 37, 47, 51].”
BackgroundMethodInfluential
Metadata
- Authors
- Saeed, A., Ozcelebi, T., & Lukkien, J.
- Venue
- Sensors
- Publisher
- MDPI AG
- Published
- 2018-09-06
- Volume
- 18
- Issue
- 9
- Pages
- 2967
- ISSN
- 1424-8220 (ISSN-L)
- OpenAlex
- W2890264806
- Cited by
- 22
- References
- 11
- FWCI
- 1.00
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.