CoChat

Deep Temporal Multimodal Fusion for Medical Procedure Monitoring Using Wearable Sensors

Bernal, Edgar A., Yang, Xitong, Li, Qun, Kumar, Jayant, Madhvanath, Sriganesh, Ramesh, Palghat, Bala, Raja · 2018 · IEEE Transactions on Multimedia
Cited by 74

The DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.

Abstract

Process monitoring and verification have a wide range of uses in the medical and healthcare fields. Currently, such tasks are often carried out by a trained specialist, which makes them expensive, inefficient, and time-consuming. Recent advances in automated video- and multimodal-data-based action and activity recognition have made it possible to reduce the extent of manual intervention required to effectively carry out process supervision tasks. In this paper, we propose algorithms for automated egocentric human action and activity recognition from multimodal data, with a target application of monitoring and assisting a user perform a multistep medical procedure. We propose a supervised deep multimodal fusion framework that relies on concurrent processing of motion data acquired with wearable sensors and video data acquired with an egocentric or body-mounted camera. We demonstrate the effectiveness of the algorithm on a public multimodal dataset and conclude that automated process monitoring via the use of multiple heterogeneous sensors is a viable alternative to its manual counterpart. Furthermore, we demonstrate that the application of previously proposed adaptive sampling schemes to the video processing branch of the multimodal framework results in significant performance improvements.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tmm.2017.2726187

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 7 references

Research works printed but not linked: 70

  1. 10.1109/ICASSP.2013.6638947 · DOI · Paper page
  2. gers, 2003, Learning precise timing with LSTM recurrent networks, J Mach Learn Res, 3, 115
  3. 10.1109/72.279181 · DOI · Paper page
  4. 10.1145/1291233.1291311 · DOI · Paper page
  5. 10.1109/ASRU.2013.6707742 · DOI · Paper page
  6. 10.1109/CVPR.2013.350 · DOI · Paper page
  7. sak, 2014, Long short-term memory based recurrent neural network architectures for large vocabulary speech recognition, CoRR
  8. lee, 0, Discovering important people and objects for egocentric video summarization, Proc IEEE Conf Comput Vis Pattern Recog, 1346
  9. 10.1145/1553374.1553510 · DOI · Paper page
  10. 10.1007/978-3-642-33718-5_23 · DOI · Paper page
  11. 10.1007/978-3-642-33786-4_7 · DOI
  12. 10.1109/ICCV.2011.6126269 · DOI · Paper page
  13. 10.1109/CVPR.2013.352 · DOI · Paper page
  14. 10.1109/CVPR.2010.5540074 · DOI · Paper page
  15. ren, 0, Egocentric recognition of handled objects: Benchmark and analysis, Proc IEEE Conf Comput Vis and Pattern Recog, 1
  16. 10.1109/CVPR.2012.6248010 · DOI · Paper page
  17. 10.1109/EIT.2014.6871782 · DOI
  18. 10.1109/ICSENS.2010.5690746 · doi:10.1109/icsens.2010.5690746
  19. 10.1109/ISWC.2006.286350 · DOI · Paper page
  20. 10.1109/MFI-2003.2003.1232672 · doi:10.1109/mfi-2003.2003.1232672
  21. baccouche, 0, Sequential deep learning for human action recognition, Proc 2nd Int Conf Human Behavior Understanding, 29
  22. 10.1109/ISMAR.2003.1240693 · DOI
  23. 10.1109/CVPR.2015.7298872 · DOI · Paper page
  24. 10.1088/0957-0233/22/11/114025 · DOI
  25. sohn, 0, Improved multimodal deep learning with variation of information, Proc Adv Neur Inf Process Syst, 2141
  26. 10.1109/JPROC.2012.2200554 · DOI
  27. 10.1109/ICASSP.2011.5947690 · DOI
  28. 10.21236/ADA623249 · DOI · Paper page
  29. 2004, Insulin administration, Diabetes Care, 27, 106s, 10.2337/diacare.27.2007.S106 · DOI
  30. srivastava, 2014, Multimodal learning with deep Boltzmann machines, J Mach Learn Res, 15, 2949
  31. 10.1109/CVPR.2011.5995496 · DOI · Paper page
  32. dosovitskiy, 2014, Discriminative unsupervised feature learning with convolutional neural networks, Proc Adv Neural Inform Process Syst 27 (NIPS)
  33. taylor, 0, Convolutional learning of spatio-temporal features, Proc 11th European Conf Computer Vision Part V, 140
  34. chen, 0, Deep learning of invariant spatio-temporal features from video, Proc NIPS 2010 Workshop Deep Learn Unsupervised Feature Learn
  35. 10.1109/CVPR.2014.223 · DOI · Paper page
  36. simonyan, 2014, Two-stream convolutional networks for action recognition in videos, Proc Adv Neural Inform Process Syst 27 (NIPS)
  37. 10.1007/s11036-007-0017-1 · DOI · Paper page
  38. 10.4236/jst.2011.12004 · DOI
  39. 10.1007/978-3-642-32320-1_3 · DOI
  40. ye, 0, Robust late fusion with rank minimization, Proc IEEE Conf Comput Vis Pattern Recog, 3021
  41. 10.1186/1471-2105-11-309 · DOI
  42. 10.1093/bioinformatics/bth294 · DOI · Paper page
  43. lanckriet, 2004, Learning the kernel matrix with semidefinite programming, J Mach Learn Res, 5, 27
  44. 10.1155/2014/490824 · DOI · Paper page
  45. gelogo, 2015, Integration of wearable monitoring device and android smartphone apps for u-healthcare monitoring system, Int J Software Eng Appl, 9, 195
  46. 10.1109/TPAMI.2011.158 · DOI · Paper page
  47. ke, 0, Efficient visual event detection using volumetric features, Proc 10th IEEE Int Conf Comput Vis, 1, 166
  48. 10.1109/CVPR.2015.7298836 · DOI
  49. 10.1109/CVPR.2012.6247806 · DOI · Paper page
  50. cheng, 2015, Advances in human action recognition: A survey, CoRR
  51. 10.1109/CVPRW.2009.5204354 · DOI · Paper page
  52. 10.1109/CVPRW.2015.7301344 · DOI
  53. srivastava, 0, Learning representations for multimodal data with deep belief nets, Proc ICML Workshop on Representation Learning
  54. ngiam, 2011, Multimodal deep learning, Proc Int Conf Mach Learn
  55. 10.1590/S0104-11692009000400007 · doi:10.1590/s0104-11692009000400007
  56. surendranath, 2012, A study to assess the knowledge and attitude of insulin self administration among patients with diabetes mellitus, Asian Journal of Pharmaceutical and Clinical Research, 5
  57. 10.4212/cjhp.v63i4.937 · doi:10.4212/cjhp.v63i4.937
  58. 10.1007/s11096-004-2270-8 · DOI · Paper page
  59. 10.1109/CVPR.2008.4587756 · DOI · Paper page
  60. 10.1109/VSPETS.2005.1570899 · DOI · Paper page
  61. 10.1109/WAINA.2009.192 · doi:10.1109/waina.2009.192
  62. 10.1007/s11263-005-1838-7 · DOI · Paper page
  63. 10.1016/j.amepre.2012.11.007 · DOI · Paper page
  64. 10.1016/j.amepre.2012.11.004 · DOI · Paper page
  65. 10.1109/ICCV.2015.530 · DOI
  66. 10.1016/j.amepre.2012.11.005 · DOI
  67. 10.5244/C.27.30 · DOI
  68. 10.1007/978-3-540-73549-6_25 · DOI
  69. 10.1109/CVPRW.2012.6239188 · DOI
  70. 10.1109/CVPRW.2014.87 · DOI

Cited by

See all 74

Metadata

Authors
Bernal, E. A., Yang, X., Li, Q., Kumar, J., Madhvanath, S., Ramesh, P., & Bala, R.
Venue
IEEE Transactions on Multimedia
Published
2018-01
Volume
20
Issue
1
Pages
107-118
ISSN
  • 1520-9210 (ISSN-L)
  • 1941-0077
OpenAlex
W2734834614
Cited by
74
References
7
FWCI
2.81
Type
journal article
Language
en
Open access
closed

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Deep Temporal Multimodal Fusion for Medical Procedure Monitoring Using Wearable Sensors. https://app.cochat.ai/papers/10.1109/tmm.2017.2726187
The paper
Bernal, E. A., Yang, X., Li, Q., Kumar, J., Madhvanath, S., Ramesh, P., & Bala, R. (2018). Deep Temporal Multimodal Fusion for Medical Procedure Monitoring Using Wearable Sensors. IEEE Transactions on Multimedia. https://doi.org/10.1109/tmm.2017.2726187

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.