Multi-modality Sensor Data Classification with Selective Attention
Open accessAbstract
Multimodel wearable sensor data classificationplays an important role in ubiquitous computingand has a wide range of applications in variousscenarios from healthcare to entertainment. How-ever, most of the existing work in this field em-ploys domain-specific approaches and is thus inef-fective in complex situations where multi-modalitysensor data is collected. Moreover, the wearablesensor data is less informative than the conven-tional data such as texts or images. In this paper,to improve the adaptability of such classificationmethods across different application contexts, weturn this classification task into a game and applya deep reinforcement learning scheme to dynami-cally deal with complex situations. We also intro-duce a selective attention mechanism into the rein-forcement learning scheme to focus on the crucialdimensions of the data. This mechanism helps tocapture extra information from the signal, and canthus significantly improve the discriminative powerof the classifier. We carry out several experimentson three wearable sensor datasets, and demonstratecompetitive performance of the proposed approachcompared to several state-of-the-art baselines.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Zhang et al. proposed combining reinforcement learning and LSTM to improve adaptability of different kinds of sensors, including EEG, RFID, and wearable IMU.
“In addition, Zhang et al. [249] proposed to combine reinforcement learning and LSTM model to improve the adaptability of different kinds of sensors, including EEG (EID dataset), RFID (RSSI dataset) [250], and wearable IMU (PAMAP2 dataset) [91].”
From Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances (2022) · cites this paper as [249]
One common way to improve robustness is to leverage the benefits of multiple types of sensors together to create multi-sensory systems.
“One common way to improve robustness is to leverage the benefits of multiple types of sensors together to create multi-sensory systems [249,310,311,312,313,314].”
From Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances (2022) · cites this paper as [249]
Multi-sensory networks in some cases are integrated with attention modules to learn the most representative and discriminative sensor modality to distinguish human activities.
“Multi-sensory networks in some cases are integrated with attention modules to learn the most representative and discriminative sensor modality to distinguish human activities [249].”
From Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances (2022) · cites this paper as [249]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357 · see citing works
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008 · see citing works
Dueling Network Architectures for Deep Reinforcement Learning
Wang, Z., Schaul, T., Hessel, M., van Hasselt, H., Lanctot, M., & de Freitas, N. (2015) · arXiv · doi:10.48550/arxiv.1511.06581 · Cited by 3,300 · see citing works
Fitting autoregressive models for prediction
Akaike, H. (1969) · Annals of the Institute of Statistical Mathematics · doi:10.1007/bf02532251 · Cited by 2,499 · see citing works
Learning to forget: continual prediction with LSTM
Gers, F. A. (1999) · doi:10.1049/cp:19991218 · Cited by 2,436 · see citing works
LSTM neural networks for language modeling
Sundermeyer, M., Schlüter, R., & Ney, H. (2012) · Interspeech 2012 · doi:10.21437/interspeech.2012-65 · Cited by 1,963 · see citing works
The visual hull concept for silhouette-based image understanding
Laurentini, A. (1994) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.273735 · Cited by 1,470 · see citing works
The PREP pipeline: standardized preprocessing for large-scale EEG analysis
Bigdely-Shamlo, N., Mullen, T. R., Kothe, C. A., Su, K., & Robbins, K. A. (2015) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2015.00016 · Cited by 1,360 · see citing works
Learning to Diagnose with LSTM Recurrent Neural Networks
Lipton, Z. C., Kale, D. C., Elkan, C., & Wetzel, R. (2015) · arXiv · doi:10.48550/arxiv.1511.03677 · Cited by 879 · see citing works
LSTM recurrent networks learn simple context-free and context-sensitive languages
Gers, F. A., & Schmidhuber, E. (2001) · IEEE Transactions on Neural Networks · doi:10.1109/72.963769 · Cited by 724 · see citing works
Time Series Classification Using Multi-Channels Deep Convolutional Neural Networks
Zheng, Y., Liu, Q., Chen, E., Ge, Y., & Zhao, J. L. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-08010-9_33 · Cited by 689 · see citing works
Attention-Based Motion Perception
Cavanagh, P. (1992) · Science · doi:10.1126/science.1523411 · Cited by 478 · see citing works
LSTM-based Deep Learning Models for Non-factoid Answer SelectionPreprint
Tan, M., dos Santos, C., Xiang, B., & Zhou, B. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.04108 · Cited by 373 · see citing works
Self-citationSilhouette scores for assessment of SNP genotype clusters
Lovmar, L., Ahlford, A., Jonsson, M., & Syvänen, A.-C. (2005) · BMC Genomics · doi:10.1186/1471-2164-6-35 · Cited by 204 · see citing works
Land cover classification accuracy as a function of sensor spatial resolution
Markham, B. L., & Townshend, J. R. G. (1981) · NASA Technical Reports Server (NASA) · Cited by 119 · see citing works
Converting Your Thoughts to Texts: Enabling Brain Typing via Deep Feature Learning of EEG Signals
Zhang, X., Yao, L., Sheng, Q. Z., Kanhere, S. S., Gu, T., & Zhang, D. (2018) · IEEE International Conference on Pervasive Computing and Communications · doi:10.1109/percom.2018.8444575 · Cited by 117 · see citing works
Self-citationCompressive Representation for Device-Free Activity Recognition with Passive RFID Signal Strength
Yao, L.-N., Sheng, Q. Z., Li, X., Gu, T., Tan, M., Wang, X.-Z., Wang, S., & Ruan, W. (2018) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2017.2706282 · Cited by 105 · see citing works
Self-citationIntent Recognition in Smart Living Through Deep Recurrent Neural Networks
Zhang, X., Yao, L., Huang, C., Sheng, Q. Z., & Wang, X. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-70096-0_76 · Cited by 77 · see citing works
Self-citationZhang, D., Yao, L., Zhang, X., Wang, S., Chen, W., & Boots, R. J. (2017) · Cited by 52 · see citing works
Self-citationA Framework for Real Time Processing of Sensor Data in the Cloud
Kamburugamuve, S., Christiansen, L., & Fox, G. (2015) · Journal of Sensors · doi:10.1155/2015/468047 · Cited by 42 · see citing works
Fida, B., Bibbo, D., Bernabucci, I., Proto, A., Conforto, S., & Schmid, M. (2015) · Wireless Mobile Communication and Healthcare · doi:10.4108/eai.14-10-2015.2261695 · Cited by 17 · see citing works
From Shared Subspaces to Shared Landmarks: A Robust Multi-Source Classification Approach
Erfani, S., Baktashmotlagh, M., Moshtaghi, M., Nguyen, V., Leckie, C. A., Bailey, J., & Ramamohanarao, K. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10870 · Cited by 14 · see citing works
Flagged first, then by citations
3 of the 26 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- Alok Chowdhury, Dian Tjon- dronegoro, Vinod Chandran, and Stewart Trost. Physical activity recognition using posterior-adapted class-based fusion of multi-accelerometers data. IEEE Journal of Biomedical and Health Informatics, 2017.
- Tokic, 2010] Michel Tokic. Adaptive ε-greedy exploration in reinforcement learning based on value differences. In Annual Conference on Artificial Intelligence, pages 203- 210. Springer, 2010.
Cited by
Sorted by citations
Sarker, I. H. (2021) · SN Computer Science · doi:10.1007/s42979-021-00815-1 · Cited by 2,890 · see citing works
Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances
Zhang, S., Li, Y., Zhang, S., Shahabi, F., Xia, S., Deng, Y., & Alshurafa, N. (2022) · Sensors · doi:10.3390/s22041476 · Cited by 625 · see citing works
“Nowadays, multi-sensor based HAR and sensor fusion has become a trend to improve the robustness and generalization of recognition systems [4, 122, 131, 212, 244, 331].”
BackgroundMethodDeep Learning for Sensor-based Human Activity Recognition: Overview, Challenges and Opportunities
Chen, K., Zhang, D., Yao, L., Guo, B., Yu, Z., & Liu, Y. (2020) · arXiv · doi:10.48550/arxiv.2001.07416 · Cited by 314 · see citing works
Self-citationUnsupervised Deep Anomaly Detection for Multi-Sensor Time-Series Signals
Zhang, Y., Chen, Y., Wang, J., & Pan, Z. (2021) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3102110 · Cited by 322 · see citing works
A survey on deep learning-based non-invasive brain signals: recent advances and new frontiers
Zhang, X., Yao, L., Wang, X., Monaghan, J., McAlpine, D., & Zhang, Y. (2021) · Journal of Neural Engineering · doi:10.1088/1741-2552/abc902 · Cited by 296 · see citing works
Self-citationKumar, P., Chauhan, S., & Awasthi, L. K. (2023) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-023-09986-x · Cited by 220 · see citing works
Ren, L., Mo, T., & Cheng, X. (2023) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2023.3264111 · Cited by 103 · see citing works
Zhang, X., Yao, L., Zhang, S., Kanhere, S. S., Sheng, Q. Z., & Liu, Y. (2018) · IEEE Internet of Things Journal · doi:10.1109/jiot.2018.2877786 · Cited by 93 · see citing works
Self-citationAn Attention Based CNN-LSTM Approach for Sleep-Wake Detection With Heterogeneous Sensors
Chen, Z., Wu, M., Cui, W., Liu, C., & Li, X. (2020) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2020.3006145 · Cited by 59 · see citing works
MVFFNet: Multi-view feature fusion network for imbalanced ship classification
Liang, M., Zhan, Y., & Liu, R. W. (2021) · Pattern Recognition Letters · doi:10.1016/j.patrec.2021.07.024 · Cited by 53 · see citing works
SEPN: A Sequential Engagement Based Academic Performance Prediction Model
Song, X., Li, J., Sun, S., Yin, H., Dawson, P., & Doss, R. (2020) · IEEE Intelligent Systems · doi:10.1109/mis.2020.3006961 · Cited by 54 · see citing works
Multi-agent Attentional Activity Recognition
Chen, K., Yao, L., Zhang, D., Guo, B., & Yu, Z. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/186 · Cited by 44 · see citing works
Self-citationA multi-view CNN-based acoustic classification system for automatic animal species identification
Xu, W., Zhang, X., Yao, L., Xue, W., & Wei, B. W. (2020) · Ad Hoc Networks · doi:10.1016/j.adhoc.2020.102115 · Cited by 44 · see citing works
Self-citationAdversarial Multi-view Networks for Activity Recognition
Bai, L., Yao, L., Wang, X., Kanhere, S. S., Guo, B., & Yu, Z. (2020) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3397323 · Cited by 40 · see citing works
Self-citationXia, S., Chu, L., Pei, L., Zhang, Z., Yu, W., & Qiu, R. C. (2021) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2021.3111996 · Cited by 40 · see citing works
MDLdroid: a ChainSGD-reduce Approach to Mobile Deep Learning for Personal Mobile Sensing
Zhang, Y., Gu, T., & Zhang, X. (2020) · 2020 19th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN) · doi:10.1109/ipsn48710.2020.00-45 · Cited by 23 · see citing works
Self-citationEnergy Harvesting-Based Smart Transportation Mode Detection System via Attention-Based LSTM
Xu, W., Feng, X., Wang, J., Luo, C., Li, J., & Ming, Z. (2019) · IEEE Access · doi:10.1109/access.2019.2918555 · Cited by 23 · see citing works
Know Your Mind: Adaptive Cognitive Activity Recognition with Reinforced CNN
Zhang, X., Yao, L., Wang, X., Zhang, W., Zhang, S., & Liu, Y. (2019) · 2019 IEEE International Conference on Data Mining (ICDM) · doi:10.1109/icdm.2019.00100 · Cited by 22 · see citing works
Self-citationZhu, Y., Luo, H., Chen, R., & Zhao, F. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3285547 · Cited by 21 · see citing works
Salgueiro, A. D. J. P., Shichkina, Y. A., García, A. G., & Rodríguez, L. G. (2021) · Procedia Computer Science · doi:10.1016/j.procs.2021.04.175 · Cited by 20 · see citing works
Differentially Private Collaborative Coupling Learning for Recommender Systems
Zhang, Y., Bai, G., Zhong, M., Li, X., & Ko, R. K. L. (2020) · IEEE Intelligent Systems · doi:10.1109/mis.2020.3005930 · Cited by 19 · see citing works
Deep reinforcement learning imbalanced credit risk of SMEs in supply chain finance
Zhang, W., Yan, S., Li, J., Peng, R., & Tian, X. (2024) · Annals of Operations Research · doi:10.1007/s10479-024-05921-w · Cited by 9 · see citing works
LIMUNet: A Lightweight Neural Network for Human Activity Recognition Using Smartwatches
Lin, L., Wu, J., An, R., Ma, S., Zhao, K., & Ding, H. (2024) · Applied Sciences · doi:10.3390/app142210515 · Cited by 9 · see citing works
Radhika, R., Bhuvaneswari, A., & Kalpana, G. (2021) · Wireless Personal Communications · doi:10.1007/s11277-021-09033-7 · Cited by 8 · see citing works
Zhang, X., Yao, L., Wang, X., Zhang, W., Zhang, S., & Liu, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.03996 · Cited by 8 · see citing works
Self-citationMcDaniel, C., & Quinn, S. P. (2018) · Proceedings of the Python in Science Conferences · doi:10.25080/majora-4af1f417-005 · Cited by 5 · see citing works
Predicting Citywide Passenger Demand via Reinforcement Learning from Spatio-Temporal Dynamics
Ning, X., Yao, L., Wang, X., Benatallah, B., Salim, F. D., & Haghighi, P. D. (2018) · EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services (MobiQuitous) · doi:10.1145/3286978.3286991 · Cited by 5 · see citing works
Self-citationMOSS—Multi-Modal Best Subset Modeling in Smart Manufacturing
Wang, L., Du, P., & Jin, R. (2021) · Sensors · doi:10.3390/s21010243 · Cited by 5 · see citing works
Radhika, R., Bhuvaneswari, A., & Kalpana, G. (2021) · Research Square · doi:10.21203/rs.3.rs-277610/v1 · Cited by 4 · see citing works
Jindal, M., Bajal, E., & Kazim, A. (2023) · Wiley eBooks · doi:10.1002/9781119857655.ch2 · Cited by 3 · see citing works
Li, Z., Tao, J., Liu, X., & Shou, D. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3581226 · Cited by 1 · see citing works
Hu, S. J., Zeng, J., Lin, X., Zhong, X., Liang, F., Chen, P., & Bai, X. (2025) · International Symposium on Computer Applications and Information Systems (ISCAIS) · doi:10.1109/iscait64916.2025.11010470 · Cited by 1 · see citing works
MDLdroid: a ChainSGD-reduce Approach to Mobile Deep Learning for Personal Mobile SensingPreprint
Zhang, Y., Gu, T., & Zhang, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.02897 · Cited by 1 · see citing works
Self-citationSensing-HH: A Deep Hybrid Attention Model for Footwear Recognition
Yao, Y., Wen, Y., & Wang, J. (2020) · Electronics · doi:10.3390/electronics9091552
Okulov, J. (2023) · Technikzukünfte, Wissenschaft und Gesellschaft · doi:10.1007/978-3-658-37641-3_11
Deep Learning Approach to Real-World Application: An Empirical Analysis
Kiranmayee, B. V., Sukheja, D., Gurujala, C. D., Kilaru, A., Rohith, D., Pallavulu, S., & More, R. (2024) · Data-intensive research · doi:10.1007/978-981-99-9179-2_45
EEG-Based Cognitive Activity Recognition
Sheng, Q. Z., Liu, Y., & Yao, L. (2026) · Springer eBooks · doi:10.1007/978-3-032-16502-2_7
Self-citationBrain-Computer Interface for IoT Cognitive Interactivity
Sheng, Q. Z., Liu, Y., & Yao, L. (2026) · Springer eBooks · doi:10.1007/978-3-032-16502-2_10
Self-citationWearable Sensor-Based Physical Activity Recognition
Sheng, Q. Z., Liu, Y., & Yao, L. (2026) · Springer eBooks · doi:10.1007/978-3-032-16502-2_6
Self-citationSheng, Q. Z., Liu, Y., & Yao, L. (2026) · Springer eBooks · doi:10.1007/978-3-032-16502-2_1
Self-citationCTRL: Cross-Modal Transformer With Reinforcement Learning for Climate Causal Discovery
Liu, J., Wang, T., Lu, Y., & Ji, J. (2026) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2026.3688211
- Authors
- Zhang, X., Yao, L., Huang, C., Wang, S., Tan, M., Long, G., & Wang, C.
- Affiliations
- UNSW Sydney (AU)
- Griffith University (AU)
- South China University of Technology (CN)
- University of Technology Sydney (AU)
- Venue
- Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence
- Published
- 2018-07
- Pages
- 3111-3117
- OpenAlex
- W2964082633
- Cited by
- 41
- References
- 26
- Citations by year
- 2026: 5 · 2025: 2 · 2024: 3 · 2023: 5 · 2022: 1 · 2021: 9 · 2020: 9 · 2019: 3 · 2018: 4
- FWCI
- 1.88
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International Joint Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- UNSWorks (University of New South Wales, Sydney, Australia) (submittedVersion, cc-by-nc-nd)
- Griffith Research Online (submittedVersion)
- UTS ePRESS (University of Technology Sydney) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Physical Sciences › Computer Science › Signal Processing › Time Series Analysis and Forecasting
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- wearable sensors, selective attention, deep reinforcement learning, ubiquitous computing, healthcare monitoring
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
