Deep generative cross-modal on-body accelerometer data synthesis from videos
The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
Human activity recognition (HAR) based on wearable sensors has brought tremendous benefit to several industries ranging from healthcare to entertainment. However, to build reliable machine-learned models from wearables, labeled on-body sensor datasets obtained from real-world settings are needed. It is often prohibitively expensive to obtain large-scale, labeled on-body sensor datasets from real-world deployments. The lack of labeled datasets is a major obstacle in the wearable sensor-based activity recognition community. To overcome this problem, I aim to develop two deep generative cross-modal architectures to synthesize accelerometer data streams from video data streams. In the proposed approach, a conditional generative adversarial network (cGAN) is first used to generate sensor data conditioned on video data. Then, a conditional variational autoencoder (cVAE)-cGAN is proposed to further improve representation of the data. The effectiveness and efficacy of the proposed methods will be evaluated through two popular applications in HAR: eating recognition and physical activity recognition. Extensive experiments will be conducted on public sensor-based activity recognition datasets by building models with synthetic data and comparing the models against those trained from real sensor data. This work aims to expand labeled on-body sensor data, by generating synthetic on-body sensor data from video, which will equip the community with methods to transfer labels from video to on-body sensors.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Activity recognition using cell phone accelerometers
Kwapisz, J. R., Weiss, G. M., & Moore, S. A. (2011) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1964897.1964918 · Cited by 2,912 · cited 1×
“ple, for physical activity recognition, most commonly used public datasets only contain labeled sensor data ranging from 15 hours (in 29 subjects) [12] to 30 hours (in 10 subjects) [19], and it has been shown that increasing the amount of data used to train the 223”
Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition
Ordóñez, F., & Roggen, D. (2016) · Sensors · doi:10.3390/s16010115 · Cited by 2,761 · cited 1×
“Several widely used classifiers including Logistic Regression (LR), Random Forest (RF), Adaboost, Convolutional Neural Network (CNN), as well as activity recognition model DeepConvLSTM [18] will be chosen.”
MethodIntroducing a New Benchmarked Dataset for Activity Monitoring
Reiss, A., & Stricker, D. (2012) · 2012 16th International Symposium on Wearable Computers · doi:10.1109/iswc.2012.13 · Cited by 1,456 · cited 2×
“recognition datasets including PAMAP2 [21] and Opportunity [4].”
A Comprehensive Survey of Deep Learning for Image Captioning
Hossain, M. Z., Sohel, F., Shiratuddin, M. F., & Laga, H. (2019) · ACM Computing Surveys · doi:10.1145/3295748 · Cited by 879 · cited 2×
“Multi-modal representation learning can enable a number of applications across a variety of modalities, including image captaining [10], image generation from text [20],”
BackgroundThe Opportunity challenge: A benchmark database for on-body sensor-based activity recognition
Chavarriaga, R., Sagha, H., Calatroni, A., Digumarti, S. T., Tröster, G., Millán, J. D. R., & Roggen, D. (2013) · Pattern Recognition Letters · doi:10.1016/j.patrec.2012.12.014 · Cited by 742 · cited 2×
“recognition datasets including PAMAP2 [21] and Opportunity [4].”
Um, T. T., Pfister, F. M. J., Pichler, D., Endo, S., Lang, M., Hirche, S., Fietzek, U., & Kulić, D. (2017) · Proceedings of the 19th ACM International Conference on Multimodal Interaction · doi:10.1145/3136755.3136817 · Cited by 729 · cited 4×
“Sensor Data Synthesis: Besides the long standing problem for general time series data augmentation, recently there has been growing interest in sensor data augmentation techniques [2, 13, 17, 23, 24].”
BackgroundResultInfluentialRavi, D., Wong, C., Lo, B., & Yang, G.-Z. (2016) · 2016 IEEE 13th International Conference on Wearable and Implantable Body Sensor Networks (BSN) · doi:10.1109/bsn.2016.7516235 · Cited by 269 · cited 1×
“ple, for physical activity recognition, most commonly used public datasets only contain labeled sensor data ranging from 15 hours (in 29 subjects) [12] to 30 hours (in 10 subjects) [19], and it has been shown that increasing the amount of data used to train the 223”
SenseGen: A deep learning architecture for synthetic sensor data generation
Alzantot, M., Chakraborty, S., & Srivastava, M. (2017) · 2017 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops) · doi:10.1109/percomw.2017.7917555 · Cited by 153
Wang, J., Chen, Y., Gu, Y., Xiao, Y., & Pan, H. (2018) · 2018 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2018.8489106 · Cited by 92 · cited 2×
“Sensor Data Synthesis: Besides the long standing problem for general time series data augmentation, recently there has been growing interest in sensor data augmentation techniques [2, 13, 17, 23, 24].”
BackgroundMethod
Cited by
Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances
Zhang, S., Li, Y., Zhang, S., Shahabi, F., Xia, S., Deng, Y., & Alshurafa, N. (2022) · Sensors · doi:10.3390/s22041476 · Cited by 625
“Therefore, taking advantage of the cutting-edge deep generative models may contribute to addressing the wearable sensor data scarcity issue [269].”
BackgroundSelf-citationIMUGPT 2.0: Language-Based Cross Modality Transfer for Sensor-Based Human Activity Recognition
Leng, Z., Bhattacharjee, A., Rajasekhar, H., Zhang, L., Bruda, E., Kwon, H., & Plötz, T. (2024) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3678545 · Cited by 61
“The most notable ones are: (i) paucity of labeled data – the time-consuming and expensive nature of wearable data collection along with its inherent privacy concerns have led to datasets being relatively smaller in size; (ii) difficulty in data annotation – ambiguity in the target activities and…”
Liu, Y., Zhang, S., & Gowda, M. (2021) · Proceedings of the International Conference on Internet-of-Things Design and Implementation · doi:10.1145/3450268.3453537 · Cited by 39
“More recently, several innovative works [38, 55, 66, 74, 78] have explored the use of YouTube-like videos for training human activity recognition (HAR) on wearable sensors.”
BackgroundSelf-citationIf only we had more data!: Sensor-Based Human Activity Recognition in Challenging Scenarios
Plötz, T. (2023) · 2023 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops) · doi:10.1109/percomworkshops56833.2023.10150267 · Cited by 16
“Similar systems and variations thereof have been described in the literature since, e.g., [61]–[64].”
Background
Metadata
- Authors
- Zhang, S., & Alshurafa, N.
- Venue
- Adjunct Proceedings of the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2020 ACM International Symposium on Wearable Computers
- Publisher
- ACM
- Published
- 2020-09-10
- Pages
- 223-227
- OpenAlex
- W3084542156
- Cited by
- 21
- References
- 11
- FWCI
- 1.30
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Deep generative cross-modal on-body accelerometer data synthesis from videos. https://app.cochat.ai/papers/10.1145/3410530.3414329
- The paper
- Zhang, S., & Alshurafa, N. (2020). Deep generative cross-modal on-body accelerometer data synthesis from videos. Adjunct Proceedings of the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2020 ACM International Symposium on Wearable Computers. https://doi.org/10.1145/3410530.3414329
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.