Combining embedded accelerometers with computer vision for recognizing food preparation activities
The DOI matches its Crossref record; 32 of the 33 references it cites were found in a registry or an index.
Abstract
This paper introduces a publicly available dataset of complex activities that involve manipulative gestures. The dataset captures people preparing mixed salads and contains more than 4.5 hours of accelerometer and RGB-D video data, detailed annotations, and an evaluation protocol for comparison of activity recognition algorithms. Providing baseline results for one possible activity recognition task, this paper further investigates modality fusion methods at different stages of the recognition pipeline: (i) prior to feature extraction through accelerometer localization, (ii) at feature level via feature concatenation, and (iii) at classification level by combining classifier outputs. Empirical evaluation shows that fusing information captured by these sensor types can considerably improve recognition performance.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The 50Salads dataset contains 50 videos with 17 action classes.”
From MS-TCN++: Multi-Stage Temporal Convolutional Network for Action Segmentation (2023) · cites this paper as [25]
In the citing paper
“Our model achieves state-of-the-art results on three challenging datasets: 50Salads, Georgia Tech Egocentric Activities (GTEA), and the Breakfast dataset.”
From MS-TCN++: Multi-Stage Temporal Convolutional Network for Action Segmentation (2023) · cites this paper as [25]
Citations
Cites
A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
Kohavi, R. (1995) · Zenodo · doi:10.5281/zenodo.19712698 · Cited by 10,909
Kittler, J., Hatef, M., Duin, R. P. W., & Matas, J. (1998) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.667881 · Cited by 5,290 · cited 2×
“For classifier fusion we tried the sum-rule and the product-rule [11] as well as Random Decision Forests as a non-linear combination method.”
MethodRecognizing human actions: A local SVM approach
Schüldt, C., Laptev, I., & Barbara Caputo, B. (2004) · Cited by 2,906
Laptev, I. (2005) · International Journal of Computer Vision · doi:10.1007/s11263-005-1838-7 · Cited by 2,629
Action recognition by dense trajectories
Wang, H., Klaser, A., Schmid, C., & Liu, C.-L. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995407 · Cited by 2,154 · cited 3×
“In addition to yielding low recognition performance, extracting local descriptors around densely sampled locations as proposed in [26] is computationally demanding in real-time, e.”
BackgroundMethodHuman activity analysis: A review
Aggarwal, J. K., & Ryoo, M. S. (2011) · ACM Computing Surveys · doi:10.1145/1922649.1922653 · Cited by 2,056
Marszalek, M., Laptev, I., & Schmid, C. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206557 · Cited by 1,168 · cited 3×
“, KTH [23], YouTube [14], Hollywood2 [15] and URADL [17]).”
BackgroundRecognizing realistic actions from videos “in the wild”
Liu, J., Luo, J., & Shah, M. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206744 · Cited by 1,075
Collecting complex activity datasets in highly rich networked sensor environments
Roggen, D., Calatroni, A., Rossi, M., Holleczek, T., Forster, K., Troster, G., Lukowicz, P., Bannach, D., Pirkl, G., Ferscha, A., et al. (2010) · 2010 Seventh International Conference on Networked Sensing Systems (INSS) · doi:10.1109/inss.2010.5573462 · Cited by 788
Criminisi, A. (2011) · now publishers, Inc. eBooks · doi:10.1561/9781601985415 · Cited by 703
Research works printed but not linked: 6
- 10.5555/2029024 · doi:10.5555/2029024
- 10.5555/2283516.2283683 · doi:10.5555/2283516.2283683
- M. E. Pollack . Intelligent technology for an aging population: The use of AI to assist elders with cognitive impairment . AI Magazine , 26 ( 2 ), 2005 . M. E. Pollack. Intelligent technology for an aging population: The use of AI to assist elders with cognitive impairment. AI Magazine, 26(2), 2005.
- 10.5555/2354409.2354909 · doi:10.5555/2354409.2354909
- 10.5555/1018429.1020906 · doi:10.5555/1018429.1020906
- 10.5555/1786014.1786017 · doi:10.5555/1786014.1786017
Cited by
ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes
Dai, A., Chang, A. X., Savva, M., Halber, M., Funkhouser, T., & Niessner, M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.261 · Cited by 4,368
Temporal Convolutional Networks for Action Segmentation and Detection
Lea, C., Flynn, M. D., Vidal, R., Reiter, A., & Hager, G. D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.113 · Cited by 2,238
“We use MERL Shopping which was designed for action detection, Georgia Tech Egocentric Activities which was designed for action segmentation, and 50 Salads which has been used for both.”
BackgroundMethodInfluentialSUN RGB-D: A RGB-D scene understanding benchmark suite
Song, S., Lichtenberg, S. P., & Xiao, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298655 · Cited by 1,943
“There are many existing RGB-D datasets available [54] [47] [1] [25] [44] [60] [49] [45] [29] [66] [57] [52] [46] [16] [21] [73] [35] [67] [3] [41] [10] [63] [42] [64] [65] [48] [12] [33] [8] [50] [26] [6].”
Temporal Convolutional Networks: A Unified Approach to Action Segmentation
Lea, C., Vidal, R., Reiter, A., & Hager, G. D. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-49409-8_7 · Cited by 924
“University of Dundee 50 Salads [18] contains 50 sequences of users making a salad.”
BackgroundScaling Egocentric Vision: The "Equation missing" Dataset
Damen, D., Doughty, H., Farinella, G. M., Fidler, S., Furnari, A., Kazakos, E., Moltisanti, D., Munro, J., Perrett, T., Price, W., et al. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01225-0_44 · Cited by 504
MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation
Farha, Y. A., & Gall, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00369 · Cited by 673
“For evaluation, we use five-fold cross-validation and report the average as in [25].”
MethodTowards Automatic Learning of Procedures From Web Instructional Videos
Zhou, L., Xu, C., & Corso, J. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12342 · Cited by 526
“Table 1 shows the comparison between YouCookII and other commonly-used instructional video datasets, e.g., YouCook (Das et al. 2013), MPII (Rohrbach et al. 2012), 50Salads (Stein and McKenna 2013), Breakfast (Kuehne, Arslan, and Serre 2014) and Coffee (Alayrac et al. 2016).”
MethodThe Language of Actions: Recovering the Syntax and Semantics of Goal-Directed Human Activities
Kuehne, H., Arslan, A., & Serre, T. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.105 · Cited by 571
“Some of the most recent action recognition datasets with this level of annotation include the CMU-MMAC dataset [19], the ICPR-KSCGR dataset [18], the GTEA Gaze+Dataset [7], the MPII Cooking dataset [17] as well as the 50 Salads dataset [20].”
MethodScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
Dai, A., Chang, A. X., Savva, M., Halber, M., Funkhouser, T. A., & Nießner, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.04405 · Cited by 412
Vision-based human activity recognition: a survey
Beddiar, D. R., Nini, B., Sabokrou, M., & Hadid, A. (2020) · Multimedia Tools and Applications · doi:10.1007/s11042-020-09004-3 · Cited by 505
“(b) 50 Salads dataset: It was created at the University of Dundee [188].”
Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100
Damen, D., Doughty, H., Farinella, G. M., Furnari, A., Kazakos, E., Ma, J., Moltisanti, D., Munro, J., Perrett, T., Price, W., et al. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-021-01531-2 · Cited by 400
Min, W., Jiang, S., Liu, L., Rui, Y., & Jain, R. (2020) · ACM Computing Surveys · doi:10.1145/3329168 · Cited by 341
A survey of depth and inertial sensor fusion for human action recognition
Chen, C., Jafari, R., & Kehtarnavaz, N. (2017) · Multimedia Tools and Applications · doi:10.1007/s11042-015-3177-1 · Cited by 356
“The 50 salads dataset 5 [58] is a publicly available dataset of complex activities that involve manipulative gestures.”
BackgroundMS-TCN++: Multi-Stage Temporal Convolutional Network for Action Segmentation
Li, S., Farha, Y. A., Liu, Y., Cheng, M.-M., & Gall, J. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.3021756 · Cited by 349
“Our approach achieves state-of-the-art results on three challenging benchmarks for action segmentation: 50Salads [15], Georgia Tech”
MethodScaling Egocentric Vision: The EPIC-KITCHENS Dataset
Damen, D., Doughty, H., Farinella, G. M., Fidler, S., Furnari, A., Kazakos, E., Moltisanti, D., Munro, J., Perrett, T., Price, W., et al. (2018) · arXiv · doi:10.48550/arxiv.1804.02748 · Cited by 274
“0M 433 3078 50 0 0 52 18 50 Salads [25] × × × 2013 0.”
MethodCOIN: A Large-Scale Dataset for Comprehensive Instructional Video Analysis
Tang, Y., Ding, D., Rao, Y., Zheng, Y., Zhang, D., Zhao, L., Lu, J., & Zhou, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00130 · Cited by 253
“MPII [35] 9h,48m 44 5,609 cooking activities self-collected - 2012 YouCook [14] 2h,20m 88 cooking activities YouTube - 2013 50Salads [40] 5h,20m 50 966 cooking activities self-collected - 2013 Breakfast [28] 77h 1,989 8,456 cooking activities self-collected 10 2014 "5 tasks" [10] 5h 150 comprehensive tasks YouTube 5 2…”
BackgroundLearning Video Representations using Contrastive Bidirectional Transformer
Sun, C., Baradel, F., Murphy, K. P., & Schmid, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.05743 · Cited by 192
“Following [23, 1], we use the five-fold cross-validation resulting in an average of 722 training and 180 test examples.”
BackgroundMethodSegmental Spatiotemporal CNNs for Fine-Grained Action Segmentation
Lea, C., Reiter, A., Vidal, R., & Hager, G. D. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46487-9_3 · Cited by 246
“University of Dundee 50 Salads: Stein and McKenna introduced 50 Salads [19] for evaluating fine-grained action recognition in the cooking domain.”
BackgroundMethodThe EPIC-KITCHENS Dataset: Collection, Challenges and Baselines
Damen, D., Doughty, H., Farinella, G. M., Fidler, S., Furnari, A., Kazakos, E., Moltisanti, D., Munro, J., Perrett, T., Price, W., et al. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2020.2991965 · Cited by 221
“five third-person activity-recognition datasets [18], [19], [26], [27], [28] that focus on object-interaction activities.”
Multi-Modal Domain Adaptation for Fine-Grained Action Recognition
Munro, J., & Damen, D. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00020 · Cited by 210
Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities
Sener, F., Chatterjee, D., Shelepov, D., He, K., Singhania, D., Wang, R., & Yao, A. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.02042 · Cited by 213
“However, they are either small [12, 47] or have little ordering variations [22].”
BackgroundMethodInfluentialTemporal Action Detection Using a Statistical Language Model
Richard, A., & Gall, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.341 · Cited by 215
“Finally, we conduct experiments on 50 Salads [24], a dataset originally designed for hierarchical activity recognition.”
MethodAnticipative Video Transformer
Girdhar, R., & Grauman, K. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01325 · Cited by 211
“While well explored in thirdperson video [2, 26, 37, 38, 46, 48, 79, 87], it has recently gained in popularity for first-person (egocentric) videos [13, 14, 16, 24, 55, 62, 74], due to its applicability on wearable computing platforms.”
BackgroundMethodResultInfluentialTemporal Deformable Residual Networks for Action Segmentation in Videos
Lei, P., & Todorovic, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00705 · Cited by 202
“We conduct experiments on three challenging action segmentation datasets, including the University of Dundee 50 Salads (50Salads) [40], Georgia Tech Egocentric Activities (GTEA) [9] and the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS) [12].”
MethodAn end-to-end generative framework for video segmentation and recognition
Kuehne, H., Gall, J., & Serre, T. (2016) · 2016 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2016.7477701 · Cited by 188
“Figure 4: Sample frames from the datasets used for performance evaluation: a) ADL [14], b) Olympic [16], c) ToyAssembly [32], d) CMU-MMAC [29], e) MPIICooking [23], f) 50Salads [30], g) Breakfast [12], and h) CRIM13 [2].”
MethodRecognizing Fine-Grained and Composite Activities Using Hand-Centric Features and Script Data
Rohrbach, M., Rohrbach, A., Regneri, M., Amin, S., Andriluka, M., Pinkal, M., & Schiele, B. (2016) · International Journal of Computer Vision · doi:10.1007/s11263-015-0851-8 · Cited by 143
“Similarly to (Farhadi et al. 2010; Fathi et al. 2011; Stein and McKenna 2013) the camera view mainly shows hands and manipulated cooking ingredients.”
BackgroundWhen will you do what? - Anticipating Temporal Occurrences of Activities
Farha, Y. A., Richard, A., & Gall, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00560 · Cited by 180
Alleviating Over-segmentation Errors by Detecting Action Boundaries
Ishikawa, Y., Kasai, S., Aoki, Y., & Kataoka, H. (2021) · 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv48630.2021.00237 · Cited by 171
“For evaluation, we use 5-fold cross-validation on 50 Salads dataset and 4-fold cross-validation on the others as in [41, 22, 9].”
MethodASFormer: Transformer for Action Segmentation
Yi, F., Wen, H., & Jiang, T. (2021) · Proceedings of the British Machine Vision Conference 2021 · doi:10.5244/c.35.49 · Cited by 170
Vision-based action recognition of construction workers using dense trajectories
Yang, J., Shi, Z., & Wu, Z. (2016) · Advanced Engineering Informatics · doi:10.1016/j.aei.2016.04.009 · Cited by 166
“This varies from previous studies, which focused either on coarse-grained actions [51,50] or fine-grained actions [21,22] only.”
BackgroundMethodHOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction
Liu, Y., Liu, Y., Jiang, C., Lyu, K., Wan, W., Shen, H., Liang, B., Fu, Z., Wang, H., & Yi, L. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.02034 · Cited by 477
“Recent advances in video action segmentation have accomplished promising achievements in coarse-level segmentation on the Breakfast [21], 50 Salad [39], and GTEA datasets [1].”
BackgroundInfluentialImproving Action Segmentation via Graph-Based Temporal Reasoning
Huang, Y., Sugano, Y., & Sato, Y. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01404 · Cited by 226
“To test the effectiveness of our proposed model on other general cases, we also test our model performance on the 50Salads [54] and Breakfast [32] datasets.”
MethodInfluentialRGBD Datasets: Past, Present and Future
Firman, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2016.88 · Cited by 147
“Others are more niche: 50 Salads [96] features over 4 hours of people preparing mixed salads.”
BackgroundWeakly-Supervised Action Segmentation with Iterative Soft Boundary Assignment
Xu, C., & Ding, L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00681 · Cited by 104
Boundary-Aware Cascade Networks for Temporal Action Segmentation
Wang, Z., Gao, Z., Wang, L., Li, Z., & Wu, G. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58595-2_3 · Cited by 199
“We evaluate our framework on three challenging datasets for action segmentation: 50Salads [26], GTEA [5] and Breakfast dataset [11].”
MethodRGB-D datasets using microsoft kinect or similar sensors: a survey
Cai, Z., Han, J., Liu, L., & Shao, L. (2016) · Multimedia Tools and Applications · doi:10.1007/s11042-016-3374-6 · Cited by 173
“3 50 salads dataset School of Computing in University of Dundee created 50 Salads dataset [88] that involves manipulative objects in 2013.”
BackgroundA review of computational approaches for evaluation of rehabilitation exercises
Liao, Y., Vakanski, A., Xian, M., Paul, D. R., & Baker, R. T. (2020) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2020.103687 · Cited by 172
Temporal Aggregate Representations for Long-Range Video Understanding
Şener, F., Singhania, D., & Yao, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58517-4_10 · Cited by 200
“We experiment on Breakfast [18], 50Salads [34] and EPIC-Kitchens [5].”
MethodNeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learning
Richard, A., Kuehne, H., Iqbal, A., & Gall, J. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00771 · Cited by 161
Unsupervised Learning of Action Classes With Continuous Temporal Embedding
Kukleva, A., Kuehne, H., Şener, F., & Gall, J. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01234 · Cited by 168
Weakly-supervised action segmentation with iterative soft boundary assignment
Ding, L., & Xu, C. (2018) · arXiv · doi:10.48550/arxiv.1803.10699 · Cited by 124
“Surpassing [15, 18], these two datasets are among the largest datasets for evaluating action segmentation.”
BackgroundTemporal Action Segmentation: An Analysis of Modern Techniques
Ding, G., Şener, F., & Yao, A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3327284 · Cited by 213
“50Salads [27] is composed of 50 recorded videos of 25 participants making two different mixed salads.”
BackgroundInfluentialAction Segmentation With Joint Self-Supervised Temporal Domain Adaptation
Chen, M.-H., Li, B., Bao, Y., AlRegib, G., & Kira, Z. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00947 · Cited by 152
Damen, D., Doughty, H., Farinella, G. M., Furnari, A., Kazakos, E., Ma, J., Moltisanti, D., Munro, J., Perrett, T., Price, W., et al. (2020) · arXiv (Cornell University) · doi:10.5523/bris.2g1n6qdydwa9u22shpxqzp0t8m · Cited by 97
Time-Conditioned Action Anticipation in One Shot
Ke, Q., Fritz, M., & Schiele, B. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01016 · Cited by 141
“The results on the Epic-Kitchen Dataset and the 50Salads Dataset are shown in Figure 5.”
InfluentialContrastive Bidirectional Transformer for Temporal Representation Learning
Sun, C., Baradel, F., Murphy, K. P., & Schmid, C. (2019) · Cited by 117
Learning convolutional action primitives for fine-grained action recognition
Lea, C., Vidal, R., & Hager, G. D. (2016) · 2016 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2016.7487305 · Cited by 106
“action primitives per class achieves 79% precision and 79% recall compared to 65% and 67% using the state of the art approach [13].”
BackgroundMethodResultInfluential(2016) · MPG.PuRe (Max Planck Society) · Cited by 107
A Comprehensive Survey on Deep Learning Methods in Human Activity Recognition
Kaseris, M., Kostavelis, I., & Malassiotis, S. (2024) · Machine Learning and Knowledge Extraction · doi:10.3390/make6020040 · Cited by 128
“✓ Weizmann [128] ✓ Stanford 40 [129] ✓ IXMAS [130] ✓ VISOR [122] ✓ MCAD [123] ✓ MSR Daily Activity 3D [124] ✓ ✓ ✓ 50 Salads [131] ✓ ✓ UCF50 [125] ✓ ✓ ETISEO [132] ✓ ✓ Olympic Sports [133] ✓ ✓ UT-Interaction [134] ✓ ✓ UT-Tower [135] ✓ ✓ ✓ ActivityNet [136] ✓ ✓ ✓ Kinetics [137] ✓ ✓ ✓ HMDB-51 [126] ✓…”
Ego-Topo: Environment Affordances From Egocentric Video
Nagarajan, T., Li, Y., Feichtenhofer, C., & Grauman, K. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00024 · Cited by 98
Deep Learning-Based Action Detection in Untrimmed Videos: A Survey
Vahdani, E., & Tian, Y. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3193611 · Cited by 120
“5 No MultiTHUMOS [52] Sports 413 65 212 97 Yes Breakfast [11] Cooking 1,712 48 162 6 No 50Salads [129] Cooking 50 17 384 20 No MPII cooking 2 [130] Cooking 273 59 356 51.”
Behrmann, N., Golestaneh, S. A., Kolter, J. Z., Gall, J., & Noroozi, M. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19833-5_4 · Cited by 155
Modeling Temporal Structure with LSTM for Online Action Detection
Geest, R. D., & Tuytelaars, T. (2018) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2018.00173 · Cited by 109
“The 50 Salads dataset [33] clearly has strong dependencies between the actions, but it is quite small with only a few hours of data.”
BackgroundWeakly Supervised Energy-Based Learning for Action Segmentation
Li, J., Lei, P., & Todorović, S. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00634 · Cited by 117
“50Salads [24] has 50 very long videos showing 17 classes of human manipulative gestures.”
BackgroundMethodInfluentialMulti-shot Temporal Event Localization: a Benchmark
Liu, X., Hu, Y., Bai, S., Ding, F., Bai, X., & Torr, P. H. S. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01241 · Cited by 79
Vision and Inertial Sensing Fusion for Human Action Recognition: A Review
Majumder, S., & Kehtarnavaz, N. (2020) · IEEE Sensors Journal · doi:10.1109/jsen.2020.3022326 · Cited by 109
“25, 966 Single AR/AD [141] D,I/(AR/AD) >=49% for all activities”
BackgroundTemporal Action Segmentation from Timestamp Supervision
Li, Z., Farha, Y. A., & Gall, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00826 · Cited by 116
“We evaluate our approach on four datasets: 50Salads [38], Breakfast [16], BEOID [7], and Georgia Tech Egocentric Activities (GTEA) [10].”
MethodTemporally-Weighted Hierarchical Clustering for Unsupervised Action Segmentation
Sarfraz, M. S., Murray, N., Sharma, V., Diba, A., Gool, L. V., & Stiefelhagen, R. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01107 · Cited by 125
Activity Recognition for Ambient Assisted Living with Videos, Inertial Units and Ambient Sensors
Ranieri, C. M., MacLeod, S., Dragone, M., Vargas, P. A., & Romero, R. A. F. (2021) · Sensors · doi:10.3390/s21030768 · Cited by 101
“50 Salads [63] Microsoft Kinect: 1 Accelerometers: 11 30 Hz 50 Hz 25 51 966”
MethodVideo benchmarks of human action datasets: a review
Singh, T., & Vishwakarma, D. K. (2018) · Artificial Intelligence Review · doi:10.1007/s10462-018-9651-1 · Cited by 89
“Something (Goyal et al. 2018) 2018 RGB Human–object interaction Table 2 RGB-D (3D) human action datasets Dataset Year Modality Applications domain i3DPost Multi-View (Gkalelis et al. 2009) 2009 RBG, Depth Human action/interaction and behaviour MSR Action 3D (Li et al. 2010) 2010 Depth+skeleton Sports Gesture recogniti…”
BackgroundInfluentialBridge-Prompt: Towards Ordinal Action Understanding in Instructional Videos
Li, M., Chen, L., Duarr, Y., Hu, Z., Feng, J., Zhou, J., & Lu, J. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01926 · Cited by 116
“50Salads [37] contains 50 top-view 30-fps instructional videos regarding salad preparation.”
BackgroundA Perceptual Prediction Framework for Self Supervised Event Segmentation
Aakur, S. N., & Sarkar, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00129 · Cited by 79
Refining Action Segmentation with Hierarchical Video Representations
Ahn, H., & Lee, D. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01599 · Cited by 92
Global2Local: Efficient Structure Search for Video Action Segmentation
Gao, S., Han, Q., Li, Z., Peng, P., Wang, L., & Cheng, M. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01653 · Cited by 103
“Following [5, 12, 40, 65], we evaluate our proposed method on three popular action segmentation datasets: Breakfast [31], 50Salads [60], and GTEA [15].”
MethodUnsupervised Action Segmentation by Joint Representation Learning and Online Clustering
Kumar, S., Haresh, S., Ahmed, A., Konin, A., Zia, M. Z., & Tran, Q.-H. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01954 · Cited by 104
Unobtrusive and Wearable Systems for Automatic Dietary Monitoring
Prioleau, T., Moore, E., & Ghovanloo, M. (2017) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2016.2631246 · Cited by 70
“A few public datasets relevant for dietary monitoring include kitchen and food preparation datasets [95]–[97], the Pittsburgh Fast-food Image Dataset [93], iEatSet [98], and iHEARu-EAT [99].”
BackgroundArtificial intelligence for phase recognition in complex laparoscopic cholecystectomy
Golany, T., Aides, A., Freedman, D., Rabani, N., Liu, Y., Rivlin, E., Corrado, G. S., Matias, Y., Khoury, W., Kashtan, H., et al. (2022) · Surgical Endoscopy · doi:10.1007/s00464-022-09405-5 · Cited by 72
“For the temporal aggregation stage, we utilize a temporal convolution network [22, 23] which recently achieved stateof-the-art results on temporal action segmentation tasks in general videos [24, 25] as well as on surgical phase detection in the Cholec80 dataset [26].”
MethodFuture Transformer for Long-term Action Anticipation
Gong, D., Lee, J., Kim, M., Ha, S. J., & Cho, M. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00306 · Cited by 120
Temporal Convolutional Networks for Action Segmentation and DetectionPreprint
Lea, C., Flynn, M. D., Vidal, R., Reiter, A., & Hager, G. D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.05267 · Cited by 136
“We use MERL Shopping which was designed for action detection, Georgia Tech Egocentric Activities which was designed for action segmentation, and 50 Salads which has been used for both.”
BackgroundMethodInfluentialIntra- and Inter-Action Understanding via Temporal Action Parsing
Shao, D., Zhao, Y., Dai, B., & Lin, D. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00081 · Cited by 92
Synthesizing Pose Sequences from 3D Assets for Vision-Based Activity Analysis
Calderon, W. T., Roberts, D., & Golparvar‐Fard, M. (2020) · Journal of Computing in Civil Engineering · doi:10.1061/(asce)cp.1943-5487.0000937 · Cited by 65
Generic Event Boundary Detection: A Benchmark for Event Segmentation
Shou, M. Z., Lei, S. W., Wang, W., Ghadiyaram, D., & Feiszli, M. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00797 · Cited by 101
“Some popular benchmarks are 50Salads [49], GTEA [29], Breakfast [24, 25], MERL Shopping [47], etc.”
BackgroundLearning Semantics-Preserving Attention and Contextual Interaction for Group Activity Recognition
Tang, Y., Lu, J., Wang, Z., Yang, M., & Zhou, J. (2019) · IEEE Transactions on Image Processing · doi:10.1109/tip.2019.2914577 · Cited by 64
“in GTEA dataset [62] and 50 Salads dataset [63] evaluated”
MethodC2F-TCN: A Framework for Semi- and Fully-Supervised Temporal Action Segmentation
Singhania, D., Rahaman, R., & Yao, A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3284080 · Cited by 74
“50Salads [80] features 25 people making 2 mixed salads.”
BackgroundA survey of transformer-based multimodal pre-trained modals
Han, X., Wang, Y.-T., Feng, J.-L., Deng, C., Chen, Z.-H., Huang, Y.-A., Su, H., Hu, L., & Hu, P.-W. (2023) · Neurocomputing · doi:10.1016/j.neucom.2022.09.136 · Cited by 59
Continuous Human Action Recognition for Human-machine Interaction: A Review
Gammulle, H., Ahmedt‐Aristizabal, D., Denman, S., Tychsen-Smith, L., Petersson, L., & Fookes, C. (2023) · ACM Computing Surveys · doi:10.1145/3587931 · Cited by 70
“Widely used datasets in the current literature include Break-fast [68], 50Salads [59], MPII cooking activities dataset [3], MPII cooking 2 dataset [119], EPIC-KITCHENS-100 [60], GTEA [67], ActivityNet [120], THUMOS15 [121], Toyota Smart-home Untrimmed dataset [122], and FineGym [123].”
MethodRF-Next: Efficient Receptive Field Search for Convolutional Neural Networks
Gao, S., Li, Z., Qi, H., Cheng, M., & Wang, L. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3183829 · Cited by 81
“mentation datasets: Breakfast [113], 50Salads [112], and GTEA [52].”
Niewiadomski, R., Ceccaldi, E., Huisman, G., Volpe, G., & Mancini, M. (2019) · Frontiers in Robotics and AI · doi:10.3389/frobt.2019.00119 · Cited by 65
“For this reason, they present work in which activity recognition is carried out by performing a training on a limited amount of data, collected in the publicly available 50 Salads dataset presented in Stein and McKenna (2013a) and Chen et al. (2017).”
MethodJoint Visual-Temporal Embedding for Unsupervised Learning of Actions in Untrimmed Sequences
VidalMata, R. G., Scheirer, W., Kukleva, A., Cox, D., & Kuehne, H. (2020) · IEEE Workshop/Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00128 · Cited by 69
“rame. We evaluate the proposed system on three challenging standard benchmark datasets, the Breakfast dataset [13], the INRIA YouTube Instructional Videos (YTI) dataset [4], and the 50 Salads dataset [36], achieving good temporal segmentation compared to state-of-the-art approaches. Figure 1 shows qualitative examples…”
MethodInfluentialSelf-Regulated Learning for Egocentric Video Activity Anticipation
Qi, Z., Wang, S., Su, C., Su, L., Huang, Q., & Tian, Q. C. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3059923 · Cited by 89
“For 50 Salads and Breakfast, considering the dense anticipation protocol, we design a new training instance generation method.”
BackgroundMethodInfluentialPredicting the future from first person (egocentric) vision: A survey
Rodin, I., Furnari, A., Mavroeidis, D., & Farinella, G. M. (2021) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2021.103252 · Cited by 72
VideoGraph: Recognizing Minutes-Long Human Activities in VideosPreprint
Hussein, N., Gavves, E., & Smeulders, A. W. M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.05143 · Cited by 92
“Cooking-related activities are good examples, as in YouCook [33], Breakfast [1], Epic-Kitchens [10], MPII Cooking [34] or 50-Salads [35].”
BackgroundToyota Smarthome Untrimmed: Real-World Untrimmed Videos for Activity Detection
Dai, R., Das, S., Sharma, S., Minciullo, L., Garattoni, L., Brémond, F., & Francesca, G. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3169976 · Cited by 87
“TABLE 1 Untrimmed Dataset Comparison Along the Seven Real-World Challenges Dataset Spontaneous Camera Object-based Multi- Composite Concurrent Var. activity Temporal View Video behaviour framing activities view activities activities duration annotation type type MEVA [17] High Low Yes No No No Low Precise Monitoring S…”
BackgroundInfluentialTemporal Relational Modeling with Self-Supervision for Action Segmentation
Wang, D., Hu, D., Li, X., & Dou, D. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i4.16377 · Cited by 64
The KIT Bimanual Manipulation Dataset
Krebs, F., Meixner, A., Patzer, I., & Asfour, T. (2021) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/humanoids47582.2021.9555788 · Cited by 81
How Much Temporal Long-Term Context is Needed for Action Segmentation?
Bahrami, E., Francesca, G., & Gall, J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00950 · Cited by 92
Handling annotation uncertainty in human activity recognition
Kwon, H., Abowd, G. D., & Plötz, T. (2019) · Proceedings of the 23rd International Symposium on Wearable Computers · doi:10.1145/3341163.3347744 · Cited by 52
“For 50salads [30], we followed the accelerometer-only protocol from the original paper as suggested by the authors.”
MethodLopes, A., de Souza, R. M., & Pedrini, H. (2022) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2022.103489 · Cited by 69
“…purposes in this sub-category, such as action recognition based on a first-person view (no torso / head parts available in video) (Tang et al., 2017), salad preparation (Stein and McKenna, 2013), hand-pose information (Tompson et al., 2014), and sign language recognition (Wang et al., 2012a).”
BackgroundTowards Structured Analysis of Broadcast Badminton Videos
Ghosh, A., Singh, S., & Jawahar, C. V. (2018) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2018.00039 · Cited by 68
The Multi-Agent Behavior Dataset: Mouse Dyadic Social Interactions
Sun, J. J., Karigo, T., Chakraborty, D., Mohanty, S. P., Wild, B., Sun, Q., Chen, C., Anderson, D. J., Perona, P., Yue, Y., et al. (2021) · PubMed · doi:10.48550/arxiv.2104.02710 · Cited by 82
“[58] Sebastian Stein and Stephen J McKenna.”
BackgroundUncertainty-Aware Anticipation of Activities
Farha, Y. A., & Gall, J. (2019) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2019.00151 · Cited by 55
“For evaluation, we use five-fold cross-validation and report the average as in [23].”
MethodTricorNet: A Hybrid Temporal Convolutional and Recurrent Network for Video Action SegmentationPreprint
Ding, L., & Xu, C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.07818 · Cited by 66
“University of Dundee 50 Salads [22] dataset captures 25 people preparing mixed salad two times each, and has annotated accelerometer and RGB-D video data.”
MethodPrediction of Manipulation Actions
Fermuller, C. M., Wang, F., Yang, Y., Zampogiannis, K., Zhang, Y., Barranco, F., & Pfeiffer, M. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-017-0992-z · Cited by 57
“plify the comparison, we chose the second coarsest level using ten different action labels, as defined in (Stein and McKenna, 2013).”
MethodRecent Data Sets on Object Manipulation: A Survey
Huang, Y., Bianchi, M., Liarokapis, M. V., & Sun, Y. (2016) · Big Data · doi:10.1089/big.2016.0042 · Cited by 57
“For the objects, the dataset includes 2D rotational velocity, which is unavailable in 50 Salad.”
BackgroundMethodInfluentialA dataset of daily interactive manipulation
Huang, Y., & Sun, Y. (2019) · The International Journal of Robotics Research · doi:10.1177/0278364919849091 · Cited by 47
“50 Salad dataset Stein and McKenna (2013) includes 3-axis acceleration of more cooking utensils than Slice & Dice which are involved in salad preparation.”
BackgroundMac, K.-N., Joshi, D., Yeh, R., Xiong, J., Feris, R., & Do, M. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00638 · Cited by 43
“50 Salads Dataset [25]: This dataset contains 50 salad making videos from multiple sensors.”
BackgroundMethodSpotting Temporally Precise, Fine-Grained Events in Video
Hong, J., Zhang, H., Gharbi, M., Fisher, M., & Fatahalian, K. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19833-5_3 · Cited by 82
Fast and Unsupervised Action Boundary Detection for Action Segmentation
Du, Z., Wang, X., Zhou, G., & Wang, Q. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00332 · Cited by 68
“Tremendous progress has recently been made on supervised action segmentation due to the introduction of large-scale datasets and the development of deep neural networks [12, 21, 26, 43].”
BackgroundMethodSkeleTR: Towards Skeleton-based Action Recognition in the Wild
Duan, H., Xu, M., Shuai, B., Modolo, D., Tu, Z., Tighe, J., & Bergamo, A. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01254 · Cited by 78
“Mean-while, other areas of interest include group activity recognition [24, 9], action localization [17, 25, 70], or action segmentation [27, 50].”
BackgroundDeep Learning for Vision-based Prediction: A SurveyPreprint
Rasouli, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.00095 · Cited by 62
“2013 50Salads [284] Cooking (ego) RGBD, Activity, Temporal seg.”
Method
Metadata
- Authors
- Stein, S., & McKenna, S. J.
- Affiliations
- University of Dundee (GB)
- Venue
- Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing
- Publisher
- ACM
- Published
- 2013-09-08
- Pages
- 729-738
- OpenAlex
- W2109698606
- Cited by
- 448
- References
- 33
- Citations by year
- 2026: 26 · 2025: 41 · 2024: 56 · 2023: 56 · 2022: 54 · 2021: 50 · 2020: 37 · 2019: 49 · 2018: 26 · 2017: 15 · 2016: 24 · 2015: 7 · 2014: 5 · 2013: 2
- FWCI
- 8.08
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) (publishedVersion)
- Discovery Research Portal (University of Dundee) (submittedVersion)
- ENLIGHTEN (Jurnal Bimbingan dan Konseling Islam) (submittedVersion)
- Funders
- Research Councils UK
- Engineering and Physical Sciences Research Council (EP/G066019/1)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Keywords
- activity recognition, embedded accelerometers, RGB-D video, multimodal fusion, cooking, salad preparation, node localization, feature-level fusion, annotated dataset
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.