Deep Temporal Multimodal Fusion for Medical Procedure Monitoring Using Wearable Sensors
The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
Process monitoring and verification have a wide range of uses in the medical and healthcare fields. Currently, such tasks are often carried out by a trained specialist, which makes them expensive, inefficient, and time-consuming. Recent advances in automated video- and multimodal-data-based action and activity recognition have made it possible to reduce the extent of manual intervention required to effectively carry out process supervision tasks. In this paper, we propose algorithms for automated egocentric human action and activity recognition from multimodal data, with a target application of monitoring and assisting a user perform a multistep medical procedure. We propose a supervised deep multimodal fusion framework that relies on concurrent processing of motion data acquired with wearable sensors and video data acquired with an egocentric or body-mounted camera. We demonstrate the effectiveness of the algorithm on a public multimodal dataset and conclude that automated process monitoring via the use of multiple heterogeneous sensors is a viable alternative to its manual counterpart. Furthermore, we demonstrate that the application of previously proposed adaptive sampling schemes to the video processing branch of the multimodal framework results in significant performance improvements.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition
Ordóñez, F., & Roggen, D. (2016) · Sensors · doi:10.3390/s16010115 · Cited by 2,761 · cited 1×
“A recent example of a deep multimodal data fusion framework for action recognition that does not rely on hand-engineered features is [68]; although the proposed method includes temporal modeling of the data, the fusion is performed with atemporal feedforward networks, more specifically CNNs, so the temporal”
MethodDense Trajectories and Motion Boundary Descriptors for Action Recognition
Wang, H., Kläser, A., Schmid, C., & Liu, C.-L. (2013) · International Journal of Computer Vision · doi:10.1007/s11263-012-0594-8 · Cited by 1,669
Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects
Lahat, D., Adali, T., & Jutten, C. (2015) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2460697 · Cited by 1,296 · cited 1×
“For more extensive treatments on multimodal data fusion, we refer the reader to [15] and [58].”
BackgroundMultimodal fusion for multimedia analysis: a survey
Atrey, P. K., Hossain, M. A., El Saddik, A., & Kankanhalli, M. S. (2010) · Multimedia Systems · doi:10.1007/s00530-010-0182-0 · Cited by 1,279
Learning to recognize objects in egocentric activities
Fathi, A., Ren, X., & Rehg, J. M. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995444 · Cited by 557 · cited 1×
“Interest in analysis of egocentric video data [29] with applications including action and activity recognition [30]–[34], object recognition and scene understanding [35]–[38], and event summarization [16], [39], [40] has recently surged.”
Background
Research works printed but not linked: 70
- 10.1109/ICASSP.2013.6638947 · DOI · Paper page
- gers, 2003, Learning precise timing with LSTM recurrent networks, J Mach Learn Res, 3, 115
- 10.1109/72.279181 · DOI · Paper page
- 10.1145/1291233.1291311 · DOI · Paper page
- 10.1109/ASRU.2013.6707742 · DOI · Paper page
- 10.1109/CVPR.2013.350 · DOI · Paper page
- sak, 2014, Long short-term memory based recurrent neural network architectures for large vocabulary speech recognition, CoRR
- lee, 0, Discovering important people and objects for egocentric video summarization, Proc IEEE Conf Comput Vis Pattern Recog, 1346
- 10.1145/1553374.1553510 · DOI · Paper page
- 10.1007/978-3-642-33718-5_23 · DOI · Paper page
- 10.1007/978-3-642-33786-4_7 · DOI
- 10.1109/ICCV.2011.6126269 · DOI · Paper page
- 10.1109/CVPR.2013.352 · DOI · Paper page
- 10.1109/CVPR.2010.5540074 · DOI · Paper page
- ren, 0, Egocentric recognition of handled objects: Benchmark and analysis, Proc IEEE Conf Comput Vis and Pattern Recog, 1
- 10.1109/CVPR.2012.6248010 · DOI · Paper page
- 10.1109/EIT.2014.6871782 · DOI
- 10.1109/ICSENS.2010.5690746 · doi:10.1109/icsens.2010.5690746
- 10.1109/ISWC.2006.286350 · DOI · Paper page
- 10.1109/MFI-2003.2003.1232672 · doi:10.1109/mfi-2003.2003.1232672
- baccouche, 0, Sequential deep learning for human action recognition, Proc 2nd Int Conf Human Behavior Understanding, 29
- 10.1109/ISMAR.2003.1240693 · DOI
- 10.1109/CVPR.2015.7298872 · DOI · Paper page
- 10.1088/0957-0233/22/11/114025 · DOI
- sohn, 0, Improved multimodal deep learning with variation of information, Proc Adv Neur Inf Process Syst, 2141
- 10.1109/JPROC.2012.2200554 · DOI
- 10.1109/ICASSP.2011.5947690 · DOI
- 10.21236/ADA623249 · DOI · Paper page
- 2004, Insulin administration, Diabetes Care, 27, 106s, 10.2337/diacare.27.2007.S106 · DOI
- srivastava, 2014, Multimodal learning with deep Boltzmann machines, J Mach Learn Res, 15, 2949
- 10.1109/CVPR.2011.5995496 · DOI · Paper page
- dosovitskiy, 2014, Discriminative unsupervised feature learning with convolutional neural networks, Proc Adv Neural Inform Process Syst 27 (NIPS)
- taylor, 0, Convolutional learning of spatio-temporal features, Proc 11th European Conf Computer Vision Part V, 140
- chen, 0, Deep learning of invariant spatio-temporal features from video, Proc NIPS 2010 Workshop Deep Learn Unsupervised Feature Learn
- 10.1109/CVPR.2014.223 · DOI · Paper page
- simonyan, 2014, Two-stream convolutional networks for action recognition in videos, Proc Adv Neural Inform Process Syst 27 (NIPS)
- 10.1007/s11036-007-0017-1 · DOI · Paper page
- 10.4236/jst.2011.12004 · DOI
- 10.1007/978-3-642-32320-1_3 · DOI
- ye, 0, Robust late fusion with rank minimization, Proc IEEE Conf Comput Vis Pattern Recog, 3021
- 10.1186/1471-2105-11-309 · DOI
- 10.1093/bioinformatics/bth294 · DOI · Paper page
- lanckriet, 2004, Learning the kernel matrix with semidefinite programming, J Mach Learn Res, 5, 27
- 10.1155/2014/490824 · DOI · Paper page
- gelogo, 2015, Integration of wearable monitoring device and android smartphone apps for u-healthcare monitoring system, Int J Software Eng Appl, 9, 195
- 10.1109/TPAMI.2011.158 · DOI · Paper page
- ke, 0, Efficient visual event detection using volumetric features, Proc 10th IEEE Int Conf Comput Vis, 1, 166
- 10.1109/CVPR.2015.7298836 · DOI
- 10.1109/CVPR.2012.6247806 · DOI · Paper page
- cheng, 2015, Advances in human action recognition: A survey, CoRR
- 10.1109/CVPRW.2009.5204354 · DOI · Paper page
- 10.1109/CVPRW.2015.7301344 · DOI
- srivastava, 0, Learning representations for multimodal data with deep belief nets, Proc ICML Workshop on Representation Learning
- ngiam, 2011, Multimodal deep learning, Proc Int Conf Mach Learn
- 10.1590/S0104-11692009000400007 · doi:10.1590/s0104-11692009000400007
- surendranath, 2012, A study to assess the knowledge and attitude of insulin self administration among patients with diabetes mellitus, Asian Journal of Pharmaceutical and Clinical Research, 5
- 10.4212/cjhp.v63i4.937 · doi:10.4212/cjhp.v63i4.937
- 10.1007/s11096-004-2270-8 · DOI · Paper page
- 10.1109/CVPR.2008.4587756 · DOI · Paper page
- 10.1109/VSPETS.2005.1570899 · DOI · Paper page
- 10.1109/WAINA.2009.192 · doi:10.1109/waina.2009.192
- 10.1007/s11263-005-1838-7 · DOI · Paper page
- 10.1016/j.amepre.2012.11.007 · DOI · Paper page
- 10.1016/j.amepre.2012.11.004 · DOI · Paper page
- 10.1109/ICCV.2015.530 · DOI
- 10.1016/j.amepre.2012.11.005 · DOI
- 10.5244/C.27.30 · DOI
- 10.1007/978-3-540-73549-6_25 · DOI
- 10.1109/CVPRW.2012.6239188 · DOI
- 10.1109/CVPRW.2014.87 · DOI
Cited by
Ali, F., El-Sappagh, S., Islam, S. M. R., Kwak, D., Ali, A., Imran, M., & Kwak, K.-S. (2020) · Information Fusion · doi:10.1016/j.inffus.2020.06.008 · Cited by 818
A Comprehensive Survey of the Internet of Things (IoT) and AI-Based Smart Healthcare
Alshehri, F., & Muhammad, G. (2021) · IEEE Access · doi:10.1109/access.2020.3047960 · Cited by 348
A comprehensive survey on multimodal medical signals fusion for smart healthcare systems
Muhammad, G., Alshehri, F., Karray, F., Saddik, A. E., Alsulaiman, M., & Falk, T. H. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.06.007 · Cited by 287
Arikumar, K., Prathiba, S., Alazab, M., Gadekallu, T., Pandya, S., Khan, J., & Moorthy, R. (2022) · Sensors · doi:10.3390/s22041377 · Cited by 168
“In [15], a supervised deep multimodal aggregation architecture for observing and authenticating the medical and healthcare industries, through concurrent processes of multiple motion data collected with a portable sensor, and video data captured by a body-mounted camera, is described.”
BackgroundRetractedImproved Accuracy in Predicting the Best Sensor Fusion Architecture for Multiple Domains
Molino-Minero-Re, E., Aguileta, A. A., Brena, R. F., & Garcia-Ceja, E. (2021) · Sensors · doi:10.3390/s21217007 · Cited by 11
Metadata
- Authors
- Bernal, E. A., Yang, X., Li, Q., Kumar, J., Madhvanath, S., Ramesh, P., & Bala, R.
- Venue
- IEEE Transactions on Multimedia
- Published
- 2018-01
- Volume
- 20
- Issue
- 1
- Pages
- 107-118
- ISSN
- 1520-9210 (ISSN-L)
- 1941-0077
- OpenAlex
- W2734834614
- Cited by
- 74
- References
- 7
- FWCI
- 2.81
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Deep Temporal Multimodal Fusion for Medical Procedure Monitoring Using Wearable Sensors. https://app.cochat.ai/papers/10.1109/tmm.2017.2726187
- The paper
- Bernal, E. A., Yang, X., Li, Q., Kumar, J., Madhvanath, S., Ramesh, P., & Bala, R. (2018). Deep Temporal Multimodal Fusion for Medical Procedure Monitoring Using Wearable Sensors. IEEE Transactions on Multimedia. https://doi.org/10.1109/tmm.2017.2726187
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.