The DOI matches its Crossref record; 9 of the 10 references it cites were found in a registry or an index.
Towards multimodal deep learning for activity recognition on mobile devices
Abstract
Current smartphones and smartwatches come equipped with a variety of sensors, from light sensor and inertial sensors to radio interfaces, enabling applications running on these devices to make sense of their surrounding environment. Rather than using sensors independently, combining their sensing capabilities facilitates more interesting and complex applications to emerge (e.g., user activity recognition). But differences between sensors ranging from sampling rate to data generation model (event triggered or continuous sampling) make integration of sensor streams challenging. Here we investigate the opportunity to use deep learning to perform this integration of sensor data from multiple sensors. The intuition is that neural networks can identify nonintuitive features largely from cross-sensor correlations which can result in a more accurate estimation. Initial results with a variant of a Restricted Boltzmann Machine (RBM), show better performance with this new approach compared to classic solutions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“In addition to training time and resources, real-world models may need to be small and compact to run on mobile devices [131].”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [131]
In the citing paper
“In this paper, we adopt a deep neural network architecture to recognize human activity from raw data input, which has shown promising results when adopted for interpreting human activities without complex data pre-processing for feature extraction [21,22,23].”
From Sensor Data Acquisition and Multimodal Sensor Fusion for Human Activity Recognition Using Deep Learning (2019) · cites this paper as [21]
In the citing paper
“Deep learning classification models are capable to give better inference results using raw sensor data than the feature data, where pre-processing transformations usually decrease the classification accuracy [21].”
From Sensor Data Acquisition and Multimodal Sensor Fusion for Human Activity Recognition Using Deep Learning (2019) · cites this paper as [21]
In the citing paper
“The recognition accuracy with this architecture on the Skoda dataset reached 81%, which is around 6% higher than the traditional classification method with the best performance (Random Forest).”
From Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances (2022) · cites this paper as [158]
In the citing paper
“In [158], a RBM is used to improve upon other methods of sensor fusion, as neural networks can identify non-intuitive predictive features largely from cross-sensor correlations and thus offer a more accurate estimation.”
From Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances (2022) · cites this paper as [158]
Citations
Cites
(2016) · Cited by 11,679
(2011) · Cited by 2,752
Stisen, A., Blunck, H., Bhattacharya, S., Prentow, T. S., Kjærgaard, M. B., Dey, A., Sonne, T., & Jensen, M. M. (2015) · Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems · doi:10.1145/2809695.2809718 · Cited by 714
Self-citationDeep, Convolutional, and Recurrent Models for Human Activity Recognition Using Wearables
Hammerla, N. Y., Halloran, S., & Ploetz, T. (2016) · arXiv · doi:10.48550/arxiv.1604.08880 · Cited by 606 · cited 1×
“This fast growing area of research has the potential to influence key topics like sensor data fusion, with study of this learning paradigm applied to mobile devices only recently begun [6, 4].”
BackgroundCan Deep Learning Revolutionize Mobile Sensing?
Lane, N. D., & Georgiev, P. (2015) · Proceedings of the 16th International Workshop on Mobile Computing Systems and Applications · doi:10.1145/2699343.2699349 · Cited by 282 · cited 1×
“This fast growing area of research has the potential to influence key topics like sensor data fusion, with study of this learning paradigm applied to mobile devices only recently begun [6, 4].”
BackgroundSelf-citationImproved Multimodal Deep Learning with Variation of Information
Sohn, K., Shang, W., & Lee, H. (2014) · neural information processing systems · Cited by 145
A semi-supervised learning approach for robust indoor-outdoor detection with smartphones
Radu, V., Katsikouli, P., Sarkar, R., & Marina, M. K. (2014) · ACM Conference on Embedded Networked Sensor Systems (SenSys) · doi:10.1145/2668332.2668347 · Cited by 125 · cited 1×
“These simple, numerous sensors provide the opportunity to help with more complex inference tasks by combining capabilities across complementary modalities; for example, as seen in [8].”
BackgroundSelf-citationMultimodal Emotion Recognition Using Multimodal Deep LearningPreprint
Liu, W., Zheng, W., & Lu, B. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.08225 · Cited by 76 · cited 1×
“Prior constructions of this variety of RBMs have been used to fuse pairs of text, video and audio data for the purpose of image captioning [10, 11], and speech [7] or emotion recognition [5].”
MethodSports video classification from multimodal information using deep neural networks
Sachan, D. S., Tekwani, U., & Sethi, A. (2013) · National Conference on Artificial Intelligence · Cited by 8
Research works printed but not linked: 1
- N. Srivastava and R. R. Salakhutdinov . 2012 . Multimodal Learning with Deep Boltzmann Machines. In In Proc. NIPS. N. Srivastava and R. R. Salakhutdinov. 2012. Multimodal Learning with Deep Boltzmann Machines. In In Proc. NIPS.
Cited by
Deep learning for sensor-based activity recognition: A survey
Wang, J., Chen, Y., Hao, S., Peng, X., & Hu, L. (2019) · Pattern Recognition Letters · doi:10.1016/j.patrec.2018.02.010 · Cited by 2,014
“For multi-modal sensors, (Radu et al., 2016) designed a multi-modal RBM where an RBM is constructed for each sensor modality, then the output of all the modalities are unified.”
BackgroundDeep Learning in Mobile and Wireless Networking: A Survey
Zhang, C., Patras, P., & Haddadi, H. (2019) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2019.2904897 · Cited by 1,768
Deep multimodal learning: a survey on recent advances and trends
Ramachandram, D., & Taylor, G. W. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2738401 · Cited by 1,139
Nweke, H. F., Teh, Y. W., Al-garadi, M. A., & Alo, U. R. (2018) · Expert Systems with Applications · doi:10.1016/j.eswa.2018.03.056 · Cited by 930
DeepSense: A Unified Deep Learning Framework for Time-Series Mobile Sensing Data Processing
Yao, S., Hu, S., Zhao, Y., Zhang, A., & Abdelzaher, T. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052577 · Cited by 667
“RBM [6] and MultiRBM [34] use Deep Boltzmann Machines and Multimodal DBMs to improve the performance of heterogeneous human activity recognition.”
MethodInfluential
Metadata
- Authors
- Radu, V., Lane, N. D., Bhattacharya, S., Mascolo, C., Marina, M. K., & Kawsar, F.
- Affiliations
- University of Edinburgh (GB)
- University College London (GB)
- Nokia Bell Labs
- University of Cambridge
- Venue
- Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct
- Publisher
- ACM
- Published
- 2016-09-12
- Pages
- 185-188
- OpenAlex
- W2519804223
- Cited by
- 147
- References
- 10
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 6 · 2023: 7 · 2022: 15 · 2021: 27 · 2020: 25 · 2019: 21 · 2018: 29 · 2017: 11 · 2016: 1
- FWCI
- 7.17
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) (publishedVersion)
- Cambridge University Engineering Department Publications Database (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Social Sciences › Transportation › Human Mobility and Location-Based Analysis
- Physical Sciences › Engineering › Electrical and Electronic Engineering › Indoor and Outdoor Localization Technologies
- Keywords
- multimodal sensor fusion, activity recognition, mobile devices, smartphone sensors, smartwatches, inertial sensors, restricted Boltzmann machine, deep learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Towards multimodal deep learning for activity recognition on mobile devices. https://app.cochat.ai/papers/10.1145/2968219.2971461
- The paper
- Radu, V., Lane, N. D., Bhattacharya, S., Mascolo, C., Marina, M. K., & Kawsar, F. (2016). Towards multimodal deep learning for activity recognition on mobile devices. Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct. https://doi.org/10.1145/2968219.2971461
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.