Jointly Learning Energy Expenditures and Activities Using Egocentric Multimodal Signals
Open accessThe DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
Physiological signals such as heart rate can provide valuable information about an individuals state and activity. However, existing work on computer vision has not yet explored leveraging these signals to enhance egocentric video understanding. In this work, we propose a model for reasoning on multimodal data to jointly predict activities and energy expenditures. We use heart rate signals as privileged self-supervision to derive energy expenditure in a training stage. A multitask objective is used to jointly optimize the two tasks. Additionally, we introduce a dataset that contains 31 hours of egocentric video augmented with heart rate and acceleration signals. This study can lead to new applications such as a visual calorie counter.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Self-citation2011 Compendium of Physical Activities: A Second Update of Codes and MET Values
AINSWORTH, B. E., HASKELL, W. L., HERRMANN, S. D., MECKES, N., BASSETT, D. R., TUDOR-LOCKE, C., GREER, J. L., VEZINA, J., WHITT-GLOVER, M. C., & LEON, A. S. (2011) · Medicine & Science in Sports & Exercise · doi:10.1249/mss.0b013e31821ece12 · Cited by 6,317
Action Recognition with Improved Trajectories
Wang, H., & Schmid, C. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.441 · Cited by 3,407
Activity Recognition from User-Annotated Acceleration Data
Bao, L., & Intille, S. S. (2004) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-24646-6_1 · Cited by 3,147
Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition
Ordóñez, F., & Roggen, D. (2016) · Sensors · doi:10.3390/s16010115 · Cited by 2,761 · cited 1×
“[65, 32] use recurrent networks for single triaxial accelerometer data, [41] for a heart rate controller, and [50] for multimodal wearable sensors.”
BackgroundActivityNet: A large-scale video benchmark for human activity understanding
Heilbron, F. C., Escorcia, V., Ghanem, B., & Niebles, J. C. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298698 · Cited by 2,578
Introducing a New Benchmarked Dataset for Activity Monitoring
Reiss, A., & Stricker, D. (2012) · 2012 16th International Symposium on Wearable Computers · doi:10.1109/iswc.2012.13 · Cited by 1,456 · cited 1×
“4 7 320×240 Head X - - Indoor Activity PAMAP2 [52] 9 - - - - X X -”
End-to-End Learning of Action Detection from Frame Glimpses in Videos
Yeung, S., Russakovsky, O., Mori, G., & Fei-Fei, L. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.293 · Cited by 585 · cited 1×
“uses recurrent neural network-based models for dense activity recognition [66] and action detection from frame glimpses [67].”
Self-citationLearning to recognize objects in egocentric activities
Fathi, A., Ren, X., & Rehg, J. M. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995444 · Cited by 557 · cited 2×
“0 40 640×480 Head X - - Kitchen Activity GTEA [21] 4 28 0.”
BackgroundFirst Person Action Recognition Using Deep Learned Descriptors
Singh, S., Arora, C., & Jawahar, C. V. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.287 · Cited by 192
Research works printed but not linked: 57
- 10.1249/MSS.0b013e31825e825a · DOI · Paper page
- 10.1109/CVPR.2015.7298625 · DOI · Paper page
- 10.1109/CVPR.2011.5995406 · DOI · Paper page
- 10.1109/TITB.2010.2051955 · DOI · Paper page
- 10.1109/CVPR.2014.223 · DOI · Paper page
- 10.1109/JPROC.2012.2200554 · DOI
- 10.1109/ICCV.2013.399 · DOI · Paper page
- lee, 2012, Discovering important people and objects for egocentric video summarization, CVPR
- 10.1109/TSMCC.2009.2023380 · DOI · Paper page
- 10.1038/sj.ejcn.1602264 · DOI
- 10.1109/CVPR.2015.7298594 · DOI · Paper page
- 10.1109/ICCV.2015.510 · DOI · Paper page
- tokui, 2015, Chainer: a next-generation open source framework for deep learning, Workshop on Machine Learning Systems at NIPS
- 10.1109/TCSVT.2008.2005594 · DOI · Paper page
- hiilloskorpi, 2003, Use of heart rate to predict energy expenditure from low to high activity levels, International Journal of Sports Medicine, 24, 332, 10.1055/s-2003-40701 · DOI · Paper page
- 10.4108/phat.1.1.e2 · DOI
- 10.1016/j.patrec.2008.08.002 · DOI · Paper page
- yeung, 2015, Every moment counts Dense detailed labeling of actions in complex videos
- idrees, 2016, The THUMOS challenge on action recognition for videos “in the wild”
- 10.1109/CVPR.2016.288 · DOI
- 10.1145/1922649.1922653 · DOI · Paper page
- abdullah, 2012, Towards population scale activity recognition: a framework for handling data diversity, AAAI
- 10.1006/cviu.1998.0716 · DOI · Paper page
- 10.1109/CVPR.2016.138 · DOI · Paper page
- hamed, 2012, Detecting activities of daily living in first-person camera views, CVPR
- 10.4108/icst.mobihealth.2014.257219 · DOI
- 10.1109/CVPR.2014.325 · DOI
- 10.1109/CVPRW.2009.5204354 · DOI · Paper page
- soomro, 2012, Ucf101 A Dataset of 101 Human Actions Classes from Videos in the Wild
- 10.1016/j.patcog.2016.07.031 · DOI
- 10.1109/CVPR.2015.7298691 · DOI
- 10.1109/CVPR.2013.352 · DOI · Paper page
- 10.1109/CVPR.2010.5540074 · DOI · Paper page
- 10.1079/BJN19890107 · DOI · Paper page
- 10.1152/japplphysiol.00818.2005 · DOI · Paper page
- liu, 2012, Multisensor data fusion for physical activity assessment, IEEE Trans Biomedical Engineering, 59, 687, 10.1109/TBME.2011.2178070 · DOI · Paper page
- damen, 2014, You-do, I-learn: discovering task relevant objects and their modes of interaction from multiuser egocentric video, BMVC
- dauphin, 2015, RMSProp and equilibrated adaptive learning rates for non-convex optimization
- 10.1109/CVPR.2015.7298878 · DOI · Paper page
- 10.1088/0967-3334/35/11/2191 · DOI · Paper page
- 10.1109/ICCV.2011.6126269 · DOI · Paper page
- 10.1109/CVPR.2012.6247805 · DOI · Paper page
- fathi, 2012, Learning to recognize daily actions using gaze, ECCV
- 10.1109/ICCV.2015.376 · DOI
- 10.1109/ICCV.2015.226 · DOI · Paper page
- bambach, 2015, A survey on recent advances of computer vision algorithms for egocentric video
- 10.1109/TCSVT.2015.2409731 · DOI · Paper page
- 10.1109/CVPR.2016.338 · DOI
- 10.1249/MSS.0b013e3181a24536 · DOI · Paper page
- mozer, 1989, A focused backpropagation algorithm for temporal pattern recognition, Complex Systems, 3, 349
- 10.1109/BSN.2013.6575520 · DOI
- 10.3390/s16010072 · DOI
- 10.1109/EMBC.2015.7318411 · DOI
- 10.1109/CVPR.2013.350 · DOI · Paper page
- 10.1016/j.cmpb.2016.02.009 · doi:10.1016/j.cmpb.2016.02.009
- 10.1007/978-3-642-12654-3_15 · DOI
- 10.1109/CVPR.2016.209 · DOI · Paper page
Cited by
Ego4D: Around the World in 3,000 Hours of Egocentric Video
Grauman, K., Westbury, A., Byrne, E., Chavis, Z., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Liu, M., Liu, X., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01842 · Cited by 630
Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances
Zhang, S., Li, Y., Zhang, S., Shahabi, F., Xia, S., Deng, Y., & Alshurafa, N. (2022) · Sensors · doi:10.3390/s22041476 · Cited by 625
“If high-fidelity synthetic wearable sensor data can be generated from the available real-world video datasets (such as Stanford-ECM dataset [294]) or online video corpus, it may help alleviate the in-the-field data scarcity issue.”
BackgroundToward Driving Scene Understanding: A Dataset for Learning Driver Behavior and Causal Reasoning
Ramanishka, V., Chen, Y.-T., Misu, T., & Saenko, K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00803 · Cited by 314
Yadav, S. K., Tiwari, K., Pandey, H. M., & Akbar, S. A. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.106970 · Cited by 311
Multimodal Federated Learning: A Survey
Che, L., Wang, J., Zhou, Y., & Ma, F. (2023) · doi:10.20944/preprints202307.1420.v1 · Cited by 127
Barut, O., Zhou, L., & Luo, Y. (2020) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.2996578 · Cited by 70
“In this article, we propose a similar approach to [10] by introducing the single LSTM-based deep neural network on both physical activity intensity estimation and activity classification problems.”
BackgroundMethodInfluentialMultimodal Attention Network for Continuous-Time Emotion Recognition Using Video and EEG Signals
Choi, D. Y., Kim, D.-H., & Song, B. C. (2020) · IEEE Access · doi:10.1109/access.2020.3036877 · Cited by 39
“1, recent multimodal deep learning networks are based on a fully-connected (FC) layer structure, adopting an approach of concatenating input modalities or intermediate features for delivery to the next layer [18].”
BackgroundMethodContinual Egocentric Activity Recognition With Foreseeable-Generalized Visual–IMU Representations
He, C., Cheng, S., Qiu, Z., Xu, L., Meng, F., Wu, Q., & Li, H. (2024) · IEEE Sensors Journal · doi:10.1109/jsen.2024.3371975 · Cited by 11
Metadata
- Authors
- Nakamura, K., Yeung, S., Alahi, A., & Fei-Fei, L.
- Venue
- 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2017-07
- Pages
- 6817-6826
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2737602873
- Cited by
- 67
- References
- 11
- FWCI
- 1.70
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.