Combining embedded accelerometers with computer vision for recognizing food preparation activities
The DOI matches its Crossref record; 32 of the 33 references it cites were found in a registry or an index.
Abstract
This paper introduces a publicly available dataset of complex activities that involve manipulative gestures. The dataset captures people preparing mixed salads and contains more than 4.5 hours of accelerometer and RGB-D video data, detailed annotations, and an evaluation protocol for comparison of activity recognition algorithms. Providing baseline results for one possible activity recognition task, this paper further investigates modality fusion methods at different stages of the recognition pipeline: (i) prior to feature extraction through accelerometer localization, (ii) at feature level via feature concatenation, and (iii) at classification level by combining classifier outputs. Empirical evaluation shows that fusing information captured by these sensor types can considerably improve recognition performance.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The 50Salads dataset contains 50 videos with 17 action classes.”
From MS-TCN++: Multi-Stage Temporal Convolutional Network for Action Segmentation (2023) · cites this paper as [25]
In the citing paper
“Our model achieves state-of-the-art results on three challenging datasets: 50Salads, Georgia Tech Egocentric Activities (GTEA), and the Breakfast dataset.”
From MS-TCN++: Multi-Stage Temporal Convolutional Network for Action Segmentation (2023) · cites this paper as [25]
Citations
Cites
A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection
Kohavi, R. (1995) · Zenodo · doi:10.5281/zenodo.19712698 · Cited by 10,909
Kittler, J., Hatef, M., Duin, R. P. W., & Matas, J. (1998) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.667881 · Cited by 5,290 · cited 2×
“For classifier fusion we tried the sum-rule and the product-rule [11] as well as Random Decision Forests as a non-linear combination method.”
MethodRecognizing human actions: A local SVM approach
Schüldt, C., Laptev, I., & Barbara Caputo, B. (2004) · Cited by 2,906
Laptev, I. (2005) · International Journal of Computer Vision · doi:10.1007/s11263-005-1838-7 · Cited by 2,629
Action recognition by dense trajectories
Wang, H., Klaser, A., Schmid, C., & Liu, C.-L. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995407 · Cited by 2,154 · cited 3×
“In addition to yielding low recognition performance, extracting local descriptors around densely sampled locations as proposed in [26] is computationally demanding in real-time, e.”
BackgroundMethodHuman activity analysis: A review
Aggarwal, J. K., & Ryoo, M. S. (2011) · ACM Computing Surveys · doi:10.1145/1922649.1922653 · Cited by 2,056
Marszalek, M., Laptev, I., & Schmid, C. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206557 · Cited by 1,168 · cited 3×
“, KTH [23], YouTube [14], Hollywood2 [15] and URADL [17]).”
BackgroundRecognizing realistic actions from videos “in the wild”
Liu, J., Luo, J., & Shah, M. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206744 · Cited by 1,075
Collecting complex activity datasets in highly rich networked sensor environments
Roggen, D., Calatroni, A., Rossi, M., Holleczek, T., Forster, K., Troster, G., Lukowicz, P., Bannach, D., Pirkl, G., Ferscha, A., et al. (2010) · 2010 Seventh International Conference on Networked Sensing Systems (INSS) · doi:10.1109/inss.2010.5573462 · Cited by 788
Criminisi, A. (2011) · now publishers, Inc. eBooks · doi:10.1561/9781601985415 · Cited by 703
Criminisi, A., Shotton, J., & Konukoglu, E. (2012) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000035 · Cited by 677 · cited 3×
“These characteristics make this classifier favorable compared to the popular support vector machine (SVM) [3].”
BackgroundPreprocessing techniques for context recognition from accelerometer data
Figo, D., Diniz, P. C., Ferreira, D. R., & Cardoso, J. M. P. (2010) · Personal and Ubiquitous Computing · doi:10.1007/s00779-010-0293-9 · Cited by 578
A database for fine grained activity detection of cooking activities
Rohrbach, M., Amin, S., Andriluka, M., & Schiele, B. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6247801 · Cited by 521 · cited 4×
“While these methods show good performance on after-the-fact (offline) classification of articulated full body motion and actions in movies, Rohrbach et al. [22] show empirically that this approach is not well suited for recognizing food preparation activities.”
BackgroundMethodInfluentialActivity recognition using the velocity histories of tracked keypoints
Messing, R., Pal, C., & Kautz, H. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459154 · Cited by 455 · cited 3×
“, KTH [23], YouTube [14], Hollywood2 [15] and URADL [17]).”
BackgroundDiscovery of activity patterns using topic models
Huynh, T., Fritz, M., & Schiele, B. (2008) · Proceedings of the 10th international conference on Ubiquitous computing · doi:10.1145/1409635.1409638 · Cited by 442 · cited 4×
“, grasp) [21], simple, repetitive, articulated full body motions [23], fine-grained hand gestures [18], or complex inter-actions of multiple objects [10].”
BackgroundInfluentialApples-to-apples in cross-validation studies: pitfalls in classifier performance measurement
Forman, G., & Scholz, M. (2010) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1882471.1882479 · Cited by 368 · cited 1×
“…and recall over all classes and cross-validation partitions are computed by first summing TP, FP and FN over all partitions for each class separately, then applying the formulas for precision and recall on the sums and finally estimating the arithmetic mean over all classes, as argued in [6].”
MethodFeature Learning for Activity Recognition in Ubiquitous Computing
Plötz, T., Hammerla, N., & Olivier, P. (2011) · Newcastle University · doi:10.25405/ncl.34175319 · Cited by 337
Activity Recognition from On-Body Sensors: Accuracy-Power Trade-Off by Dynamic Sensor Selection
Zappi, P., Lombriser, C., Stiefmeier, T., Farella, E., Roggen, D., Benini, L., & Tröster, G. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-77690-1_2 · Cited by 329 · cited 2×
“, car assembly-line checkpoint [29] or food preparation [18]).”
BackgroundA Scalable Approach to Activity Recognition based on Object Use
Wu, J., Osuntogun, A., Choudhury, T., Philipose, M., & Rehg, J. M. (2007) · 2007 IEEE 11th International Conference on Computer Vision · doi:10.1109/iccv.2007.4408865 · Cited by 324
Elliott, D. (2010) · Cited by 312
Guide to the Carnegie Mellon University Multimodal Activity (CMU-MMAC) Database
Torre, F. D. L., Hodgins, J. K., Bargteil, A. W., et al. (2018) · KiltHub Repository · doi:10.1184/r1/6555020.v1 · Cited by 236
Hoey, J., Poupart, P., Bertoldi, A. V., Craig, T., Boutilier, C., & Mihailidis, A. (2010) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2009.06.008 · Cited by 229
Guide to the Carnegie Mellon University Multimodal Activity (CMU-MMAC) Database
Torre, F. D. L., Hodgins, J. K., Bargteil, A. W., Martin, X., Macey, J., Collado, A., & Beltran, P. (2018) · KiltHub Repository · doi:10.1184/r1/6555020 · Cited by 228
Tenorth, M., Bandouch, J., & Beetz, M. (2009) · 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops · doi:10.1109/iccvw.2009.5457583 · Cited by 214 · cited 4×
“, Darmstadt Daily Routines [10], AmbientKitchen [18], TUM Kitchen [25], CMU-MACC [4], Opportunity [21], ICPR-KSCGR1 and MPII-Cooking [22]).”
BackgroundInfluentialRecognizing realistic actions from videos “in the wild”
Liu, J., Luo, J., & Shah, M. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvprw.2009.5206744 · Cited by 179
Learning Multi-modal SimilarityPreprint
McFeeBrian, & LanckrietGert (2010) · arXiv (Cornell University) · doi:10.5555/1953048.1953063 · Cited by 132 · cited 1×
“Prominent approaches include multiple kernel learning as a feature combination method for discriminative classification [16], and the sum-rule and product-rule for classifier combination [11].”
MethodActivity Recognition in Pervasive Intelligent Environments
Chen, L., Hoey, J., Biswas, J., & Nugent, C. (2011) · Atlantis ambient and pervasive intelligence · doi:10.2991/978-94-91216-05-3 · Cited by 120 · cited 1×
“In pervasive intelligent environments it is therefore common practice to embed sensors in objects involved in an interaction instead [2].”
BackgroundRapid specification and automated generation of prompting systems to assist people with dementia
Hoey, J., Plötz, T., Jackson, D., Monk, A. F., Pham, C., & Olivier, P. L. (2010) · Pervasive and Mobile Computing · doi:10.1016/j.pmcj.2010.11.007 · Cited by 116 · cited 1×
“Ubiquitous computing has great potential in addressing the challenges of an aging population by providing automatic situated support for people with cognitive impairments [7, 8, 20].”
BackgroundSlice&Dice: Recognizing Food Preparation Activities Using Embedded Accelerometers
Pham, C., & Olivier, P. L. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-05408-2_4 · Cited by 84 · cited 9×
“, car assembly-line checkpoint [29] or food preparation [18]).”
BackgroundMethodInfluentialProblems people with dementia have with kitchen tasks: The challenge for pervasive computing
Wherton, J. P., & Monk, A. (2010) · Interacting with computers · doi:10.1016/j.intcom.2010.03.004 · Cited by 62 · cited 1×
“, dementia to stay longer in their homes and to perform activities of daily living (ADL) independently of a social carer arguably increases their perceived quality of life [27] and significantly reduces the social cost associated with cognitive impairments.”
BackgroundAn Accelerometer Based Approach for Indoor Localization
Hsu, C., & Yu, C.-H. (2009) · Symposia and Workshops on Ubiquitous, Autonomic and Trusted Computing (UIC-ATC) · doi:10.1109/uic-atc.2009.90 · Cited by 33 · cited 1×
“Video and IMU data from mobile phones are combined in [9] for indoor localization.”
BackgroundAccelerometer Localization in the View of a Stationary Camera
Stein, S., & McKenna, S. J. (2012) · Canadian Conference on Computer and Robot Vision · doi:10.1109/crv.2012.22 · Cited by 10 · cited 2×
“In order to use visual accelerometer localization for activity recognition we adapt the approach proposed in [24].”
MethodSelf-citation
Research works printed but not linked: 6
- 10.5555/2029024 · doi:10.5555/2029024
- 10.5555/2283516.2283683 · doi:10.5555/2283516.2283683
- M. E. Pollack . Intelligent technology for an aging population: The use of AI to assist elders with cognitive impairment . AI Magazine , 26 ( 2 ), 2005 . M. E. Pollack. Intelligent technology for an aging population: The use of AI to assist elders with cognitive impairment. AI Magazine, 26(2), 2005.
- 10.5555/2354409.2354909 · doi:10.5555/2354409.2354909
- 10.5555/1018429.1020906 · doi:10.5555/1018429.1020906
- 10.5555/1786014.1786017 · doi:10.5555/1786014.1786017
Cited by
ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes
Dai, A., Chang, A. X., Savva, M., Halber, M., Funkhouser, T., & Niessner, M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.261 · Cited by 4,368
Temporal Convolutional Networks for Action Segmentation and Detection
Lea, C., Flynn, M. D., Vidal, R., Reiter, A., & Hager, G. D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.113 · Cited by 2,238
“We use MERL Shopping which was designed for action detection, Georgia Tech Egocentric Activities which was designed for action segmentation, and 50 Salads which has been used for both.”
BackgroundMethodInfluentialSUN RGB-D: A RGB-D scene understanding benchmark suite
Song, S., Lichtenberg, S. P., & Xiao, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298655 · Cited by 1,943
“There are many existing RGB-D datasets available [54] [47] [1] [25] [44] [60] [49] [45] [29] [66] [57] [52] [46] [16] [21] [73] [35] [67] [3] [41] [10] [63] [42] [64] [65] [48] [12] [33] [8] [50] [26] [6].”
Temporal Convolutional Networks: A Unified Approach to Action Segmentation
Lea, C., Vidal, R., Reiter, A., & Hager, G. D. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-49409-8_7 · Cited by 924
“University of Dundee 50 Salads [18] contains 50 sequences of users making a salad.”
BackgroundScaling Egocentric Vision: The "Equation missing" Dataset
Damen, D., Doughty, H., Farinella, G. M., Fidler, S., Furnari, A., Kazakos, E., Moltisanti, D., Munro, J., Perrett, T., Price, W., et al. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01225-0_44 · Cited by 504
Metadata
- Authors
- Stein, S., & McKenna, S. J.
- Affiliations
- University of Dundee (GB)
- Venue
- Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing
- Publisher
- ACM
- Published
- 2013-09-08
- Pages
- 729-738
- OpenAlex
- W2109698606
- Cited by
- 448
- References
- 33
- Citations by year
- 2026: 26 · 2025: 41 · 2024: 56 · 2023: 56 · 2022: 54 · 2021: 50 · 2020: 37 · 2019: 49 · 2018: 26 · 2017: 15 · 2016: 24 · 2015: 7 · 2014: 5 · 2013: 2
- FWCI
- 8.08
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) (publishedVersion)
- Discovery Research Portal (University of Dundee) (submittedVersion)
- ENLIGHTEN (Jurnal Bimbingan dan Konseling Islam) (submittedVersion)
- Funders
- Research Councils UK
- Engineering and Physical Sciences Research Council (EP/G066019/1)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
- Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Keywords
- activity recognition, embedded accelerometers, RGB-D video, multimodal fusion, cooking, salad preparation, node localization, feature-level fusion, annotated dataset
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.