SADeepSense: Self-Attention Deep Learning Framework for Heterogeneous On-Device Sensors in Internet of Things Applications
The DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index.
Abstract
Deep neural networks are becoming increasingly popular in Internet of Things (IoT) applications. Their capabilities of fusing multiple sensor inputs and extracting temporal relationships can enhance intelligence in a wide range of applications. However, one key problem is the missing of adaptation to heterogeneous on-device sensors. These low-end sensors on IoT devices possess different accuracies, granularities, and amounts of information, whose sensing qualities are heterogeneous and vary over time. The existing deep learning frameworks for IoT applications usually treat every sensor input equally over time or increase model capacity in an ad-hoc manner, lacking the ability to identify and exploit the sensor heterogeneities. In this work, we propose SADeepSense, a deep learning framework that can automatically balance the contributions of multiple sensor inputs over time by exploiting their sensing qualities. SADeepSense makes two key contributions. First, SADeepSense employs the self-attention mechanism to learn the correlations among different sensors over time with no additional supervision. The correlations are then applied to infer the sensing qualities and to reassign model concentrations in multiple sensors over time. Second, instead of directly learning the sensing qualities and contributions, SADeepSense generates the residual concentrations that are deviated from the equal contributions, which helps to stabilize the training process. We demonstrate the effectiveness of SADeepSense with two representative IoT sensing tasks: heterogeneous human activity recognition with motion sensors and gesture recognition with the wireless signal. SADeepSense consistently outperforms the state-of-the-art methods by a clear margin. In addition, we show that SADeepSense only imposes little additional resource-consumption burden on embedded devices compared to the corresponding state-of-the-art framework.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
Dropout as a bayesian approximation: Representing model uncertainty in deep learning
Gal, Y., & Ghahramani, Z. (2015) · arXiv · doi:10.48550/arxiv.1506.02142 · Cited by 6,471
A Survey of Context-Aware Mobile Computing Research
Chen, G., & Kotz, D. F. (2000) · Cited by 1,749
Stisen, A., Blunck, H., Bhattacharya, S., Prentow, T. S., Kjærgaard, M. B., Dey, A., Sonne, T., & Jensen, M. M. (2015) · Proceedings of the 13th ACM Conference on Embedded Networked Sensor Systems · doi:10.1145/2809695.2809718 · Cited by 714
DeepSense: A Unified Deep Learning Framework for Time-Series Mobile Sensing Data Processing
Yao, S., Hu, S., Zhao, Y., Zhang, A., & Abdelzaher, T. (2017) · Proceedings of the 26th International Conference on World Wide Web · doi:10.1145/3038912.3052577 · Cited by 667 · cited 6×
“Great efforts have been made on designing unified structures for fusing multiple sensing inputs and extracting temporal relationships [19], [11], compressing neural network structures for reducing resource consumptions on low-end devices [24], [20], [4], providing well-calibrated uncertainty estimations for neural net…”
BackgroundMethodSelf-citationPreprocessing techniques for context recognition from accelerometer data
Figo, D., Diniz, P. C., Ferreira, D. R., & Cardoso, J. M. P. (2010) · Personal and Ubiquitous Computing · doi:10.1007/s00779-010-0293-9 · Cited by 578 · cited 1×
“It selects all popular time-domain and frequency domain features from [7] and ECDF features from [9].”
BackgroundA reliable and accurate indoor localization method using phone inertial sensors
Li, F., Zhao, C., Ding, G., Gong, J., Liu, C., & Zhao, F. (2012) · Proceedings of the 2012 ACM Conference on Ubiquitous Computing · doi:10.1145/2370216.2370280 · Cited by 574
DeepEar: robust smartphone audio sensing in unconstrained acoustic environments using deep learning
Lane, N. D., Georgiev, P., & Qendro, L. (2015) · Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing · doi:10.1145/2750858.2804262 · Cited by 310
Bhattacharya, S., & Lane, N. D. (2016) · Proceedings of the 14th ACM Conference on Embedded Networked Sensor Systems CD-ROM · doi:10.1145/2994551.2994564 · Cited by 236
Yao, S., Zhao, Y., Zhang, A., Su, L., & Abdelzaher, T. (2017) · Proceedings of the 15th ACM Conference on Embedded Network Sensor Systems · doi:10.1145/3131672.3131675 · Cited by 188
Self-citationYao, S., Zhao, Y., Shao, H., Liu, S., Liu, D., Su, L., & Abdelzaher, T. (2018) · Proceedings of the 16th ACM Conference on Embedded Networked Sensor Systems · doi:10.1145/3274783.3274840 · Cited by 146
Self-citationHammerla, N. Y., Kirkham, R., Andras, P., & Ploetz, T. (2013) · Proceedings of the 2013 International Symposium on Wearable Computers · doi:10.1145/2493988.2494353 · Cited by 138 · cited 1×
“It selects all popular time-domain and frequency domain features from [7] and ECDF features from [9].”
BackgroundDeep Learning for the Internet of Things
Yao, S., Zhao, Y., Zhang, A., Hu, S., Shao, H., Zhang, C., Su, L., & Abdelzaher, T. (2018) · Computer · doi:10.1109/mc.2018.2381131 · Cited by 132
Self-citationMusicalHeart: a hearty way of listening to music
Nirjon, S., Dickerson, R. F., Li, Q., Asare, P., Stankovic, J. A., Hong, D., Zhang, B., Jiang, X., Shen, G., & Zhao, F. (2012) · Proceedings of the 10th ACM Conference on Embedded Network Sensor Systems · doi:10.1145/2426656.2426662 · Cited by 105
Mathur, A., Zhang, T., Bhattacharya, S., Velickovic, P., Joffe, L., Lane, N. D., Kawsar, F., & Lio, P. (2018) · 2018 17th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN) · doi:10.1109/ipsn.2018.00048 · Cited by 70 · cited 3×
“We compare SADeepSense to the state-of-the-art DeepSense framework [19], the data-augmentation based solution for sensing heterogeneity problem [13], and other traditional machine learning algorithms.”
BackgroundMethodSenseGAN: Enabling Deep Learning for Internet of Things with a Semi-Supervised Framework
Yao, S., Zhao, Y., Shao, H., Zhang, C., Zhang, A., Hu, S., Liu, D., Liu, S., Su, L., & Abdelzaher, T. (2018) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3264954 · Cited by 54
Self-citationRDeepSense: Reliable Deep Mobile Computing Models with Uncertainty Estimations
Yao, S., Zhao, Y., Shao, H., Zhang, A., Zhang, C., Li, S., & Abdelzaher, T. (2018) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3161181 · Cited by 52
Self-citationRecursive Ground Truth Estimator for Social Data Streams
Yao, S., Amin, M. T., Su, L., Hu, S., Li, S., Wang, S., Zhao, Y., Abdelzaher, T., Kaplan, L., Aggarwal, C., et al. (2016) · 2016 15th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN) · doi:10.1109/ipsn.2016.7460719 · Cited by 38
Self-citationWisture: RNN-based Learning of Wireless Signals for Gesture Recognition in Unmodified SmartphonesPreprint
Haseeb, M. A. A., & Parasuraman, R. N. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.08569 · Cited by 19
Research works printed but not linked: 2
- 0, Weka for Android
- 0, Mobile TensorFlow
Cited by
Human activity recognition in artificial intelligence framework: a narrative review
Gupta, N., Gupta, S. K., Pathak, R. K., Jain, V., Rashidi, P., & Suri, J. S. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-021-10116-x · Cited by 337
A survey on deploying mobile deep learning applications: A systemic and technical perspective
Wang, Y., Wang, J., Zhang, W., Zhan, Y., Guo, S., Zheng, Q., & Wang, X. (2022) · Digital Communications and Networks · doi:10.1016/j.dcan.2021.06.001 · Cited by 110
Radar-Based Human Activity Recognition Using Hybrid Neural Network Model With Multidomain Fusion
Ding, W., Guo, X., & Wang, G. (2021) · IEEE Transactions on Aerospace and Electronic Systems · doi:10.1109/taes.2021.3068436 · Cited by 99
Attention-Based Deep Learning Framework for Human Activity Recognition With User Adaptation
Buffelli, D., & Vandin, F. (2021) · IEEE Sensors Journal · doi:10.1109/jsen.2021.3067690 · Cited by 68
Deep learning based multimodal complex human activity recognition using wearable devices
Chen, L., Liu, X., Peng, L., & Wu, M. (2020) · Applied Intelligence · doi:10.1007/s10489-020-02005-7 · Cited by 72
Metadata
- Authors
- Yao, S., Zhao, Y., Shao, H., Liu, D., Liu, S., Hao, Y., Piao, A., Hu, S., Lu, S., & Abdelzaher, T. F.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- University of Washington (US)
- IBM (United States) (US)
- Venue
- IEEE INFOCOM 2019 - IEEE Conference on Computer Communications
- Publisher
- IEEE
- Published
- 2019-04
- Pages
- 1243-1251
- OpenAlex
- W2918488306
- Cited by
- 52
- References
- 23
- Citations by year
- 2025: 8 · 2024: 11 · 2023: 11 · 2022: 6 · 2021: 9 · 2020: 7
- FWCI
- 1.79
- Type
- conference paper
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Conference on Computer Communications (INFOCOM) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Physical Sciences › Computer Science › Computer Networks and Communications › IoT and Edge/Fog Computing
- Physical Sciences › Computer Science › Information Systems › Internet of Things and AI
- Keywords
- heterogeneous sensors, self-attention, sensor fusion, residual concentration, human activity recognition, gesture recognition, wireless communication signals, embedded devices, Internet of Things
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.