Transferable multi-modal fusion in knee angles and gait phases for their continuous prediction
Open accessAbstract
Objective. The gait phase and joint angle are two essential and complementary components of kinematics during normal walking, whose accurate prediction is critical for lower-limb rehabilitation, such as controlling the exoskeleton robots. Multi-modal signals have been used to promote the prediction performance of the gait phase or joint angle separately, but it is still few reports to examine how these signals can be used to predict both simultaneously. Approach. To address this problem, we propose a new method named transferable multi-modal fusion (TMMF) to perform a continuous prediction of knee angles and corresponding gait phases by fusing multi-modal signals. Specifically, TMMF consists of a multi-modal signal fusion block, a time series feature extractor, a regressor, and a classifier. The multi-modal signal fusion block leverages the maximum mean discrepancy to reduce the distribution discrepancy across different modals in the latent space, achieving the goal of transferable multi-modal fusion. Subsequently, by using the long short-term memory-based network, we obtain the feature representation from time series data to predict the knee angles and gait phases simultaneously. To validate our proposal, we design an experimental paradigm with random walking and resting to collect data containing multi-modal biomedical signals from electromyography, gyroscopes, and virtual reality. Main results. Comprehensive experiments on our constructed dataset demonstrate the effectiveness of the proposed method. TMMF achieves a root mean square error of 0.090 ± 0.022 s in knee angle prediction and a precision of 83.7 ± 7.7 % in gait phase prediction. Significance. We demonstrate the feasibility and validity of using TMMF to predict lower-limb kinematics continuously from multi-modal biomedical signals. This proposed method represents application potential in predicting the motor intent of patients with different pathologies.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
QDA uses quadratic decision boundaries with a heterogeneous covariance assumption, is capable of modeling complex distributions, and is used for dynamic muscle activation pattern recognition via EMG.
“| QDA | Quadratic decision boundaries, heterogeneous | Capable of modeling complex | Dynamic muscle activation pattern recognition |”
From Recent advances in intelligent wearable systems: from multiscale biomechanical features towards human motion intent prediction (2026) · cites this paper as [262]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
SciPy 1.0: fundamental algorithms for scientific computing in Python
Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., et al. (2020) · Nature Methods · doi:10.1038/s41592-019-0686-2 · Cited by 38,430 · see citing works
Statistical Comparisons of Classifiers over Multiple Data Sets
Demšar, J. (2006) · doi:10.5555/1248547.1248548 · Cited by 11,966 · see citing works
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893 · see citing works
Multi-scale context aggregation by dilated convolutions
Yu, F., & Koltun, V. (2015) · arXiv · doi:10.48550/arxiv.1511.07122 · Cited by 7,738 · see citing works
What is a support vector machine?
Noble, W. S. (2006) · Nature Biotechnology · doi:10.1038/nbt1206-1565 · Cited by 4,307 · see citing works
Domain adaptation for medical image analysis: A survey
Guan, H., & Liu, M. (2022) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2021.3117407 · Cited by 1,057 · see citing works
Electromechanical delay in human skeletal muscle under concentric and eccentric contractions
Cavanagh, P. R., & Komi, P. V. (1979) · European Journal of Applied Physiology · doi:10.1007/bf00431022 · Cited by 805 · see citing works
Autonomous exoskeleton reduces metabolic cost of human walking during load carriage
Mooney, L. M., Rouse, E. J., & Herr, H. (2014) · Journal of NeuroEngineering and Rehabilitation · doi:10.1186/1743-0003-11-80 · Cited by 466 · see citing works
Hybrid decision tree and naïve Bayes classifiers for multi-class classification tasks
Farid, D. M., Zhang, L., Rahman, C. M., Hossain, M. A., & Strachan, R. (2013) · Expert Systems with Applications · doi:10.1016/j.eswa.2013.08.089 · Cited by 369 · see citing works
Effect of timing of hip extension assistance during loaded walking with a soft exosuit
Ding, Y., Panizzolo, F. A., Siviy, C., Malcolm, P., Galiana, I., Holt, K. G., & Walsh, C. J. (2016) · Journal of NeuroEngineering and Rehabilitation · doi:10.1186/s12984-016-0196-8 · Cited by 250 · see citing works
How robust is human gait to muscle weakness?
van der Krogt, M. M., Delp, S. L., & Schwartz, M. (2012) · Gait & Posture · doi:10.1016/j.gaitpost.2012.01.017 · Cited by 240 · see citing works
Fleming, A., Stafford, N. E., Huang, S., Hu, X., Ferris, D. P., & Huang, H. (2021) · Journal of Neural Engineering · doi:10.1088/1741-2552/ac1176 · Cited by 221 · see citing works
Panwar, M., Biswas, D., Bajaj, H., Jobges, M., Turk, R., Maharatna, K., & Acharyya, A. (2019) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2019.2899927 · Cited by 158 · see citing works
FS-HGR: Few-Shot Learning for Hand Gesture Recognition via Electromyography
Rahimian, E., Zabihi, S., Asif, A. A., Farina, D., Atashzar, S. F., & Mohammadi, A. (2021) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2021.3077413 · Cited by 135 · see citing works
From sensing to control of lower limb exoskeleton: a systematic review
Sun, Y., Tang, Y., Jia, Z., Dong, D., Chen, X., & Bai, L. (2022) · Annual Reviews in Control · doi:10.1016/j.arcontrol.2022.04.003 · Cited by 135 · see citing works
Deep learning-based BCI for gait decoding from EEG with LSTM recurrent neural network
Tortora, S., Ghidoni, S., Chisari, C., Micera, S., & Artoni, F. (2020) · Journal of Neural Engineering · doi:10.1088/1741-2552/ab9842 · Cited by 134 · see citing works
Durandau, G., Farina, D., Asín-Prieto, G., Dimbwadyo-Terrer, I., Lara, S. L., Pons, J. L., Moreno, J. C., & Sartori, M. (2019) · Journal of NeuroEngineering and Rehabilitation · doi:10.1186/s12984-019-0559-z · Cited by 121 · see citing works
Li, Y., Pan, J., Long, J., Yu, T., Wang, F., Yu, Z., & Wu, W. (2016) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2469106 · Cited by 117 · see citing works
Self-citationLower-Limb Joint Torque Prediction Using LSTM Neural Networks and Transfer Learning
Zhang, L., Soselia, D., Wang, R., & Gutierrez-Farewik, E. M. (2022) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2022.3156786 · Cited by 108 · see citing works
Lu, Y., Wang, H., Zhou, B., Wei, C., & Xu, S. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117340 · Cited by 104 · see citing works
Gait training using a robotic hip exoskeleton improves metabolic gait efficiency in the elderly
Martini, E., Crea, S., Parri, A., Bastiani, L., Faraguna, U., McKinney, Z., Molino-Lova, R., Pratali, L., & Vitiello, N. (2019) · Scientific Reports · doi:10.1038/s41598-019-43628-2 · Cited by 99 · see citing works
Ma, C., Lin, C., Samuel, O. W., Xu, L., & Li, G. (2020) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2020.102024 · Cited by 89 · see citing works
An empirical comparison of neural networks and machine learning algorithms for EEG gait decoding
Nakagome, S., Luu, T. P., He, Y., Ravindran, A. S., & Contreras-Vidal, J. L. (2020) · Scientific Reports · doi:10.1038/s41598-020-60932-4 · Cited by 81 · see citing works
Hybrid Human-Machine Interface for Gait Decoding Through Bayesian Fusion of EEG and EMG Classifiers
Tortora, S., Tonin, L., Chisari, C., Micera, S., Menegatti, E., & Artoni, F. (2020) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2020.582728 · Cited by 74 · see citing works
Lee, J., Hong, W., & Hur, P. (2021) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2021.3098689 · Cited by 73 · see citing works
Machine-Learning-Based Prediction of Gait Events From EMG in Cerebral Palsy Children
Christian Morbidoni, B., Cucchiarelli, A., Agostini, V., Knaflitz, M., Fioretti, S., & Nardo, F. D. (2021) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2021.3076366 · Cited by 68 · see citing works
Biomechanics and muscle function during gait
Brunner, R., & Rutz, E. (2013) · Journal of Children s Orthopaedics · doi:10.1007/s11832-013-0508-5 · Cited by 67 · see citing works
A CNN-Attention Network for Continuous Estimation of Finger Kinematics from Surface Electromyography
Geng, Y., Yu, Z., Long, Y., Qin, L., Chen, Z., Li, Y., Guo, X., & Li, G. (2022) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2022.3169448 · Cited by 66 · see citing works
Inertial Sensing for Gait Event Detection and Transfemoral Prosthesis Control Strategy
Ledoux, E. D. (2018) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2018.2813999 · Cited by 63 · see citing works
Ma, X., Liu, Y., Song, Q., & Wang, C. (2020) · Sensors · doi:10.3390/s20174966 · Cited by 61 · see citing works
Gautam, A., Panwar, M., Wankhede, A., Arjunan, S. P., Naik, G. R., Acharyya, A., & Kumar, D. (2020) · IEEE Journal of Translational Engineering in Health and Medicine · doi:10.1109/jtehm.2020.3023898 · Cited by 61 · see citing works
Gait phase recognition of lower limb exoskeleton system based on the integrated network model
Zhang, Z., Wang, Z., Han, L., & Gu, W. (2022) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2022.103693 · Cited by 59 · see citing works
Inter-Subject Domain Adaptation for CNN-Based Wrist Kinematics Estimation Using sEMG
Bao, T., Zaidi, S. A. R., Xie, S., Yang, P., & Zhang, Z. (2021) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2021.3086401 · Cited by 56 · see citing works
Continuous Prediction of Lower-Limb Kinematics From Multi-Modal Biomedical Signals
Yi, C., Jiang, F., Zhang, S., Guo, H. Y., Yang, C., Ding, Z., Wei, B., Lan, X., & Zhou, H. (2021) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2021.3071461 · Cited by 51 · see citing works
Molazadeh, V., Zhang, Q., Bao, X., & Sharma, N. (2021) · IEEE Transactions on Control Systems Technology · doi:10.1109/tcst.2021.3089885 · Cited by 51 · see citing works
A Review of Computational Musculoskeletal Analysis of Human Lower Extremities
Alamdari, A., & Krovi, V. (2016) · Elsevier eBooks · doi:10.1016/b978-0-12-803137-7.00003-3 · Cited by 43 · see citing works
Iterative Refinement for Multi-source Visual Domain Adaptation
Wu, H., Yan, Y., Lin, G., Yang, M., Ng, M. K., & Wu, Q. (2020) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2020.3014697 · Cited by 39 · see citing works
Self-citationSilva, L., & Stergiou, N. (2020) · Elsevier eBooks · doi:10.1016/b978-0-12-813372-9.00007-5 · Cited by 37 · see citing works
Deep Learning for Accelerometric Data Assessment and Ataxic Gait Monitoring
Procházka, A., Dostál, O., Cejnar, P., Mohamed, H. I., Pavelek, Z., Vališ, M., & Vyšata, O. (2021) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2021.3051093 · Cited by 36 · see citing works
Knowledge Preserving and Distribution Alignment for Heterogeneous Domain Adaptation
Wu, H., Wu, Q., & Ng, M. K. (2021) · ACM Transactions on Information Systems · doi:10.1145/3469856 · Cited by 34 · see citing works
Self-citationZhang, H., Duong, T. T. H., Rao, A. K., Mazzoni, P., Agrawal, S. K., Guo, Y. P., & Zanotto, D. (2022) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2022.3143094 · Cited by 32 · see citing works
Heterogeneous Domain Adaptation by Information Capturing and Distribution Matching
Wu, H., Zhu, H., Yan, Y., Wu, J., Zhang, Y., & Ng, M. K. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3094137 · Cited by 29 · see citing works
Self-citationZhang, Q., Li, F., Zhang, Q., & Xiong, C. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.05.113 · Cited by 29 · see citing works
Real-Time Voluntary Motion Prediction and Parkinson’s Tremor Reduction Using Deep Neural Networks
Ibrahim, A. H., Zhou, Y., Jenkins, M. E., Trejos, A. L., & Naish, M. D. (2021) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2021.3097007 · Cited by 28 · see citing works
Wu, H., Long, J., Li, N., Yu, D., & Ng, M. K. (2022) · ACM Transactions on Information Systems · doi:10.1145/3544105 · Cited by 26 · see citing works
Self-citationA Novel sEMG-Based Gait Phase-Kinematics-Coupled Predictor and Its Interaction With Exoskeletons
Wei, B., Ding, Z., Yi, C., Guo, H. Y., Wang, Z., Zhu, J., & Jiang, F. (2021) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2021.704226 · Cited by 23 · see citing works
Adaptive neural fuzzy reasoning method for recognizing human movement gait phase
Song, J., Zhu, A., Tu, Y., Mao, H., & Zhang, X. (2022) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2022.104087 · Cited by 23 · see citing works
Shi, K., Mu, F., Huang, R., Huang, K., Peng, Z., Zou, C., Yang, X. L., & Cheng, H. (2022) · Frontiers in Neuroscience · doi:10.3389/fnins.2022.796290 · Cited by 18 · see citing works
Foot and footwear biomechanics and gait
Price, C., Schmeltzpfenning, T., Nester, C., & Brauner, T. (2021) · Elsevier eBooks · doi:10.1016/b978-0-12-821606-4.00013-2 · Cited by 16 · see citing works
Multiple Graphs and Low-Rank Embedding for Multi-Source Heterogeneous Domain Adaptation
Wu, H., & Ng, M. K. (2022) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3492804 · Cited by 14 · see citing works
Self-citationEstimation of knee and ankle angles during walking using thigh and shank angles
Eslamy, M., & Schilling, A. F. (2021) · Bioinspiration & Biomimetics · doi:10.1088/1748-3190/ac245f · Cited by 10 · see citing works
Vaswani (2017) · doi:10.5555/3295222.3295349
Flagged first, then by citations
Cited by
Sorted by citations
Shaping high-performance wearable robots for human motor and sensory reconstruction and enhancement
Xia, H., Zhang, Y., Rajabi, N., Taleb, F., Yang, Q., Kragić, D., & Li, Z. (2024) · Nature Communications · doi:10.1038/s41467-024-46249-0 · Cited by 137 · see citing works
A Review of Gait Analysis Using Gyroscopes and Inertial Measurement Units
Lin, S., Evans, K. A., Hartley, D., Morrison, S., McDonald, S., Veidt, M., & Wang, G. (2025) · Sensors · doi:10.3390/s25113481 · Cited by 20 · see citing works
Chen, S., Peng, C., Yang, B., Lin, J., Zhou, L., Jiang, Z., Liu, Z., Liu, Y., & Tang, L. (2026) · npj Artificial Intelligence · doi:10.1038/s44387-026-00083-5 · Cited by 9 · see citing works
Online Transfer Learning With Pseudo Label for Gait Phase Prediction
Guo, Z., Wu, H., Fang, J., Zhang, J., & Long, J. (2024) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2024.3480203 · Cited by 5 · see citing works
Self-citationYe, Q., Yang, X., Zhao, R., Ji, Y., Cai, X., Zheng, Q. S., & Fan, Y. (2025) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2025.3646165 · Cited by 4 · see citing works
Zhou, S., Ma, X., Lv, Y., Luo, C., Luo, C., & Zhou, H. (2026) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2026.109978 · Cited by 4 · see citing works
Zhang, M., & Sun, R. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.106192 · Cited by 3 · see citing works
A Multi-sensor Gait Dataset Collected Under Non-standardized Dual-Task Conditions
Liao, Y., Cao, J., Yu, L., Xiang, J., & Zhao, Y. (2025) · Scientific Data · doi:10.1038/s41597-025-05458-y · Cited by 3 · see citing works
Huang, Y., Tang, Y., Zhang, Z., Shen, L., Guo, Z., Chen, Z., & Long, J. (2026) · Journal of NeuroEngineering and Rehabilitation · doi:10.1186/s12984-025-01871-x · Cited by 2 · see citing works
Self-citationBeyond Gait: Seamless Knee Angle Prediction for Lower Limb Prosthesis in Multiple Scenarios
Wang, P., Chen, Y., Su, W., Wang, J., Ma, T., & Yu, H. (2024) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2024.3506220 · Cited by 2 · see citing works
An EEG-sEMG Asynchronous Time–Frequency Progressive Fusion Model for Hand Trajectory Estimation
Duan, S., Wu, L., Liu, A., Qian, R., & Chen, X. (2025) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2025.3636906 · Cited by 1 · see citing works
Review of gait prediction of lower extremity exoskeleton robot
Geng, H., Guo, X., Lin, H., Hao, Y., & Zhang, G. (2024) · Progress in Medical Devices · doi:10.61189/673672yizrwd
Saoud, L. S., & Hussain, I. (2025) · IEEE Access · doi:10.1109/access.2025.3582745
Research on Continuous Motion Intent Perception Method Based on Network Layer FusionPreprint
Wang, W., Dong, B., Yuan, X., & Liu, J. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5368931
Lv, Y., Zhang, X., Zhou, R., Zhang, W., Song, C., Chen, C., Lu, C., & Xu, J. (2025) · IEEE Transactions on Medical Robotics and Bionics · doi:10.1109/tmrb.2025.3646757
Wang, W., Liu, C., Shen, L., Li, Z., & Yuan, X. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.134011
Wang, A., Teng, R., Dong, J., Yue, X., & Zhang, X. (2026) · Journal of Robotics and Mechatronics · doi:10.20965/jrm.2026.p1059
Continuous knee joint angle prediction for exoskeleton applications using sEMG–IMU fusion and PGQNet
Lin, J., Zheng, G., Jia, H., Tang, J., Shi, L., & Zhou, D. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-01155-x
- Authors
- Guo, Z., Zheng, H., Wu, H., Zhang, J., Zhou, G., & Long, J.
- Affiliations
- Jinan University (CN)
- Guangdong University of Technology (CN)
- Corresponding author
- Long, Jinyi
- Venue
- Journal of Neural Engineering
- Publisher
- IOP Publishing
- Published
- 2023-05-24
- Volume
- 20
- Issue
- 3
- Pages
- 036019
- ISSN
- 1741-2552 (ISSN-L)
- 1741-2560
- OpenAlex
- W4365504023
- Cited by
- 18
- References
- 53
- Citations by year
- 2026: 6 · 2025: 7 · 2024: 5
- FWCI
- 1.42
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- Locations
- Journal of Neural Engineering (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Guangdong Provincial Natural Science Foundation of China (No. 2019A1515012175)
- Outstanding Youth Project of Guangdong Natural Science Foundation of China (No. 2021B1515020076)
- China Postdoctoral Science Foundation (No. 2022M721343)
- Science and Technology Project in Guangzhou (No. 202201010498)
- Guangdong Provincial Key Laboratory of Traditional Chinese Medicine Informatization (No. 2021B1212040007)
- National Natural Science Foundation of China (No. 62106084)
- Fundamental Research Funds for Central Universities (No. 21621026)
- Topics
- Physical Sciences › Engineering › Biomedical Engineering › Prosthetics and Rehabilitation Robotics
- Physical Sciences › Engineering › Biomedical Engineering › Muscle activation and electromyography studies
- Physical Sciences › Engineering › Biomedical Engineering › Gait Recognition and Analysis
- Keywords
- maximum mean discrepancy, lower limb kinematics, electromyography, gyroscope, virtual reality, LSTM, lower limb rehabilitation, exoskeleton control
- MeSH terms
- Biomechanical Phenomena, Electromyography, Gait, Humans, Walking, Lower Extremity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
