CoChat
Sign Up
Cited by 400Human Pose and Action RecognitionHand Gesture Recognition SystemsContext-Aware Activity Recognition Systems

Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100

Open access
Damen, Dima, Doughty, Hazel, Farinella, Giovanni Maria, Furnari, Antonino, Kazakos, Evangelos, Ma, Jian, Moltisanti, Davide, Munro, Jonathan +3 more·2022·International Journal of Computer Vision
DOI 10.1007/s11263-021-01531-2
Open this paper in CoChat — free accountOpen PDF

Abstract

This paper introduces the pipeline to extend the largest dataset in egocentric vision, EPIC-KITCHENS. The effort culminates in EPIC-KITCHENS-100, a collection of 100 hours, 20M frames, 90K actions in 700 variable-length videos, capturing long-term unscripted activities in 45 environments, using head-mounted cameras. Compared to its previous version (Damen in Scaling egocentric vision: ECCV, 2018), EPIC-KITCHENS-100 has been annotated using a novel pipeline that allows denser (54% more actions per minute) and more complete annotations of fine-grained actions (+128% more action segments). This collection enables new challenges such as action detection and evaluating the “test of time”—i.e. whether models trained on data collected in 2018 can generalise to new footage collected two years later. The dataset is aligned with 6 challenges: action recognition (full and weak supervision), action detection, action anticipation, cross-modal retrieval (from captions), as well as unsupervised domain adaptation for action recognition. For each challenge, we define the task, provide baselines and evaluation metrics.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s11263-021-01531-2

Cites

25 of 25 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • V-JEPA 2 action anticipation performance increases consistently with model size on EK100.

    “Using the Epic-Kitchens-100 (EK100) benchmark (Damen et al., 2022), we demonstrate that V-JEPA 2 action anticipation performance increases consistently with model size.”

    From V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning (2025) · cites this paper as Damen et al., 2022

  • EK100 action anticipation task involves predicting noun, verb, and action from a video clip that occurs before the start timestamp of an action segment, with anticipation time set to 1 second by default.

    “The EK100 action anticipation task involves predicting noun, verb, and action (i.e., predicting verb and noun jointly) from a video clip, referred to as context, that occurs before the start timestamp of an action segment. The interval between the end of the context and the beginning of the action segment is the anticipation time, which is set to 1 second by default.”

    From V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning (2025) · cites this paper as Damen et al., 2022

  • Mean-class recall-at-5 is used as the metric to measure performance on EK100 action anticipation.

    “Given that different future actions may be possible from a given context, mean-class recall-at-5 is used as the metric to measure performance (Damen et al., 2022).”

    From V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning (2025) · cites this paper as Damen et al., 2022

  • EK100 dataset comprises 100 hours of cooking activities recorded from an egocentric perspective across 45 kitchen environments.

    “The EK100 dataset is comprised of 100 hours of cooking activities recorded from an egocentric perspective across 45 kitchen environments.”

    From V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning (2025) · cites this paper as Damen et al., 2022

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 92

  1. Bearman A, Russakovsky O, Ferrari V, & Fei-Fei L (2016) What's the point: semantic segmentation with point supervision. In ECCV
  2. Bhattacharyya A, Fritz M, & Schiele B (2019) Bayesian prediction of future street scenes using synthetic likelihoods. In ICLR
  3. Bojanowski P, Lajugie R, Bach F, Laptev I, Ponce J, Schmid C, & Sivic J (2014) Weakly supervised action labeling in videos under ordering constraints. In ECCV
  4. Cao Y, Long M, Wang J, & Yu P (2017) Correlation hashing network for efficient cross-modal retrieval. In BMVC
  5. Caputo B, Müller H, Martinez-Gomez J, Villegas M, Acar B, Patricia N, Marvasti N, Üsküdarlı S, Paredes R, Cazorla M, et al. (2014) Imageclef 2014: Overview and analysis of the results. In: Inter- national Conference of the Cross-Language Evaluation Forum for European Languages, Springer 192-211
  6. Carlevaris-Bianco, N., Ushani, A. K., & Eustice, R. M. (2016). Univer- sity of Michigan North Campus long-term vision and lidar dataset. Int J Robotics Res, 35(9), 1023-1035.
  7. Carreira J, Noland E, Hillier C, & Zisserman A (2019) A short note on the Kinetics-700 human action dataset. arXiv
  8. Chang C, Huang DA, Sui Y, Fei-Fei L, & Niebles JC (2019) D3TW: Discriminative differentiable dynamic time warping for weakly supervised action alignment and segmentation. In CVPR
  9. Chen D, & Dolan, W (2011) Collecting highly parallel data for para- phrase evaluation. In NAACL-HLT
  10. Chen MH, Kira Z, AlRegib G, Yoo J, Chen R, & Zheng J (2019) Tem- poral attentive alignment for large-scale video domain adaptation. In ICCV
  11. Chéron G, Alayrac J, Laptev I, & Schmid C (2018) A flexible model for training action localization with varying levels of supervision. In NeurIPS
  12. Cordts M, Omran M, Ramos S, Rehfeld T, Enzweiler M, Benenson R, Franke U, Roth S, & Schiele B (2016) The cityscapes dataset for semantic urban scene understanding. In CVPR
  13. Damen D, Doughty H, Farinella GM, Fidler S, Furnari A, Kazakos E, Moltisanti D, Munro J, Perrett T, Price W, & Wray M (2018) Scal- ing egocentric vision: The EPIC-KITCHENS dataset. In ECCV
  14. Damen D, Leelasawassuk T, Haines O, Calway A, & Mayol-Cuevas W (2014). You-do, I-learn: Discovering task relevant objects and their modes of interaction from multi-user egocentric video. In BMVC
  15. De Geest R, Gavves E, Ghodrati A, Li Z, Snoek C, & Tuytelaars T (2016) Online action detection. In ECCV
  16. Deng J, Dong W, Socher R, Li LJ, Li K, & Fei-Fei L (2009) Imagenet: A large-scale hierarchical image database. In CVPR
  17. Ding L, & Xu C (2018) Weakly-supervised action segmentation with iterative soft boundary assignment. In CVPR
  18. Fathi A, Li Y, & Rehg J (2012) Learning to recognize daily actions using gaze. In ECCV
  19. Furnari A, & Farinella GM (2020) Rolling-unrolling LSTMs for action anticipation from first-person video. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI)
  20. Ganin Y, Ustinova E, Ajakan, H, Germain P, Larochelle H, Laviolette F, Marchand M, & Lempitsky V (2016) Domain-adversarial training of neural networks. JMLR
  21. Geiger A, Lenz P, & Urtasun R (2012) Are we ready for autonomous driving? The KITTI vision benchmark suite. In CVPR
  22. Gu C, Sun C, Ross DA, Vondrick C, Pantofaru C, Li Y, Vijaya- narasimhan S, Toderici G, Ricco S, Sukthankar R, Schmid C, & Malik J (2018) AVA: A video dataset of spatio-temporally local- ized atomic visual actions. In CVPR
  23. Gupta S, & Malik J (2016) Visual semantic role labeling. In CVPR
  24. Gygli M, & Ferrari V (2019) Efficient object annotation via speaking and pointing. IJCV
  25. He K, Girshick R, & Dollár P (2019) Rethinking ImageNet pre-training. In ICCV
  26. He K, Gkioxari G, Dollár P, & Girshick R (2017) Mask R-CNN. In ICCV
  27. Heilbron FC, Lee JY, Jin H, & Ghanem B (2018) What do i annotate next?, An empirical study of active learning for action localization. In ECCV
  28. Ioffe S, & Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. In ICML
  29. Jamal A, Namboodiri VP, Deodhare D, & Venkatesh K (2018) Deep domain adaptation in action space. In BMVC
  30. Kay W, Carreira J, Simonyan K, Zhang B, Hillier C, Vijayanarasimhan S, Viola F, Green T, Back T, Natsev P, Suleyman M, & Zisserman A (2017) The Kinetics human action video dataset. arXiv
  31. Kazakos E, Nagrani A, Zisserman A, & Damen D (2019) EPIC-Fusion: Audio-visual temporal binding for egocentric action recognition. In ICCV
  32. Kingma DP, & Ba J (2014) Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 · arXiv
  33. Koppula HS, & Saxena A (2016) Anticipating human activities using object affordances for reactive robotic response. TPAMI
  34. Krishna R, Hata K, Ren F, Fei-Fei L, & Niebles JC (2017) Dense- captioning events in videos. In ICCV
  35. Lee N, Choi W, Vernaza P, Choy C, Torr PHS, & Chandraker M (2017) DESIRE: Distant future prediction in dynamic scenes with inter- acting agents. In CVPR
  36. Li J, Lei P, & Todorovic S (2019) Weakly supervised energy-based learning for action segmentation. In ICCV
  37. Li Y, Ye Z, & Rehg JM (2015) Delving into egocentric actions. In CVPR
  38. Lin J, Gan C, & Han S (2019) TSM: Temporal shift module for efficient video understanding. In ICCV
  39. Lin T, Liu X, Li X, Ding E, & Wen S (2019) BMN: Boundary-matching network for temporal action proposal generation. In ICCV
  40. Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, & Zitnick CL (2014) Microsoft COCO: Common objects in context. In ECCV
  41. Liu D, Jiang T, & Wang Y (2019) Completeness modeling and context separation for weakly supervised temporal action localization. In CVPR
  42. Mahdisoltani F, Berger G, Gharbieh W, Fleet D, & Memisevic R (2018) On the effectiveness of task granularity for transfer learning. arXiv
  43. Marszalek M, Laptev I, & Schmid C (2009) Actions in context. In CVPR
  44. Mettes P, Van Gemert JC, & Snoek CG (2016) Spot on: Action local- ization from pointly-supervised proposals. In ECCV
  45. Moltisanti D, Fidler S, & Damen D (2019). Action recognition from single timestamp supervision in untrimmed videos. In CVPR
  46. Moltisanti D, Wray M, Mayol-Cuevas W, & Damen D (2017) Tres- passing the boundaries: Labeling temporal bounds for object interactions in egocentric video. In ICCV
  47. Monfort M, Vondrick C, Oliva A, Andonian A, Zhou B, Ramakrishnan K, Bargal SA, Yan T, Brown L, Fan Q, & Gutfreund D (2020) Moments in Time dataset: One million videos for event under- standing. TPAMI
  48. Munro J, & Damen D (2020) Multi-modal domain adaptation for fine- grained action recognition. In CVPR
  49. Narayan S, Cholakkal H, Khan F, & Shao L (2019) 3C-Net: Category count and center loss for weakly-supervised action localization. In ICCV
  50. Nguyen P, Liu T, Prasad G, & Han B (2018). Weakly supervised action localization by sparse temporal pooling network. In CVPR
  51. Nguyen P, Ramanan D, & Fowlkes C (2019) Weakly-supervised action localization with background modeling. In ICCV
  52. Noroozi M, & Favaro P (2016) Unsupervised learning of visual repre- sentations by solving jigsaw puzzles. In ECCV
  53. Oberdiek P, Rottmann M, & Fink GA (2020) Detection and Retrieval of Out-of-Distribution Objects in Semantic Segmentation. In Com- puter Vision and Pattern Recognition Workshops
  54. Pan B, Cao Z, Adeli E, & Niebles JC (2020) Adversarial cross-domain action recognition with co-attention. In AAAI
  55. Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L, Desmaison A, Kopf A, Yang E, DeVito Z, Raison M, Tejani A, Chilamkurthy S, Steiner B, Fang L, Bai J, & Chintala S (2019). Pytorch: An imperative style, high- performance deep learning library. In Wallach H, Larochelle H, Beygelzimer A, dÁlché-Buc F, Fox E, & Garnett R, eds. Advances in Neural Information Processing Systems 32.
  56. Patron-Perez A, Marszalek M, Zisserman A, & Reid I (2010) High Five: Recognising human interactions in TV shows. In BMVC
  57. Peng X, Bai Q, Xia X, Huang Z, Saenko K, & Wang B (2019) Moment matching for multi-source domain adaptation. In ICCV
  58. Peng X, Usman B, Kaushik N, Hoffman J, Wang D, & Saenko K (2017) VisDA: The visual domain adaptation challenge. arXiv
  59. Perazzi F, Pont-Tuset J, McWilliams B, Gool LV, Gross M, & Sorkine- Hornung A (2016) A benchmark dataset and evaluation method- ology for video object segmentation. In CVPR
  60. Pirsiavash H, & Ramanan D (2012) Detecting activities of daily living in first-person camera views. In CVPR
  61. Planamente M, Plizzari C, Alberti E, & Caputo B (2021) Cross-domain first person audio-visual action recognition through relative norm alignment. arXiv preprint arXiv:2106.01689 · arXiv
  62. Plizzari C, Planamente M, Alberti E, & Caputo B (2021). Polito-iit sub- mission to the epic-kitchens-100 unsupervised domain adaptation challenge for action recognition. arXiv preprint arXiv:2107.00337 · arXiv
  63. Qi F, Yang X, & Xu C (2018) A unified framework for multimodal domain adaptation. In ACM-MM Rasiwasia N, Mahajan D, Mahadevan V, & Aggarwal G (2014) Cluster canonical correlation analysis. In AISTATS
  64. Richard A, Kuehne H, Iqbal A, & Gall J (2018) NeuralNetwork-Viterbi: A framework for weakly supervised video learning. In CVPR
  65. Rohrbach, M., Amin, S., Andriluka, M., & Schiele, B. (2012). A database for fine grained activity detection of cooking activities. In CVPR
  66. Rohrbach A, Rohrbach M, Tandon N, & Schiele B (2015) A dataset for movie description. In CVPR
  67. Roth J, Chaudhuri S, Klejch O, Marvin R, Gallagher A, Kaver L, Ramaswamy S, Stopczynski A, Schmid C, Xi Z, et al. (2019). AVA-ActiveSpeaker: An audio-visual dataset for active speaker detection. arXiv
  68. Saenko K, Kulis B, Fritz M, & Darrell T (2010) Adapting Visual Category Models to New Domains. In European Conference on Computer Vision
  69. Shan D, Geng J, Shu M, & Fouhey DF (2020) Understanding human hands in contact at internet scale. In CVPR
  70. Sigurdsson GA, Gupta A, Schmid C, Farhadi A, & Alahari K (2018) Charades-ego: A large-scale dataset of paired third and first person videos. In ArXiv
  71. Sigurdsson GA, Varol G, Wang X, Farhadi A, Laptev I, & Gupta A (2016) Hollywood in Homes: Crowdsourcing data collection for activity understanding. In ECCV
  72. Soomro K, Zamir AR, & Shah M (2012) UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv
  73. Stein S, & McKenna SJ (2013). Combining Embedded Accelerom- eters with Computer Vision for Recognizing Food Preparation Activities. In International Joint Conference on Pervasive and Ubiquitous Computing
  74. Ueberla JP (1997) Domain adaptation with clustered language mod- els. In International Conference on Acoustics, Speech and Signal Processing
  75. Venkateswara H, Eusebio J, Chakraborty S, & Panchanathan S (2017) Deep hashing network for unsupervised domain adaptation. In CVPR
  76. Vondrick C, Shrivastava A, Fathi A, Guadarrama S, & Murphy K (2018) Tracking emerges by colorizing videos. In ECCV
  77. Wang L, Xiong Y, Lin D, & Van Gool L (2017) Untrimmednets for weakly supervised action recognition and detection. In CVPR
  78. Wang L, Xiong Y, Wang Z, Qiao Y, Lin D, Tang X, & Gool LV (2016) Temporal Segment Networks: Towards good practices for deep action recognition. In ECCV
  79. Weinzaepfel P, Martin X, & Schmid C (2016) Human action localization with sparse spatial supervision. arXiv
  80. Wray M, Larlus D, Csurka G, & Damen D (2019) Fine-grained action retrieval through multiple parts-of-speech embeddings. In ICCV
  81. Wulfmeier M, Bewley A, & Posner I (2018) Incremental Adversarial Domain Adaptation for Continually Changing Environments. In International Conference on Robotics and Automation. 4489-4495
  82. Xu J, Mei T, Yao T, & Rui Y (2016) MSR-VTT: A large video descrip- tion dataset for bridging video and language. In CVPR
  83. Yang L, Huang Y, Sugano Y, & Sato Y (2021) Epic-kitchens-100 unsu- pervised domain adaptation challenge for action recognition 2021: Team m3em technical report. arXiv preprint arXiv:2106.10026 · arXiv
  84. Yeung S, Russakovsky O, Jin N, Andriluka M, Mori G, & Fei-Fei L (2017) Every Moment Counts: Dense detailed labeling of actions in complex videos. IJCV
  85. Yu F, Xian W, Chen Y, Liu F, Liao M, Madhavan V, & Darrell T (2018) BDD100K: A diverse driving video database with scalable anno- tation tooling. arXiv
  86. Zamir AR, Sax A, Shen W, Guibas L, Malik J, & Savarese S (2018) Taskonomy: Disentangling task transfer learning. In CVPR
  87. Zhai X, Puigcerver J, Kolesnikov A, Ruyssen P, Riquelme C, Lucic M, Djolonga J, Pinto AS, Neumann M, Dosovitskiy A, Beyer L, Bachem O, Tschannen M, Michalski M, Bousquet O, Gelly S, & Houlsby N (2019) A large-scale study of representation learning with the visual task adaptation benchmark. arXiv
  88. Zhao H, Yan Z, Torresani L, & Torralba A (2019) HACS: Human action clips and segments dataset for recognition and temporal localiza- tion. In ICCV
  89. Zhou B, Andonian A, Oliva A, & Torralba A (2018) Temporal relational reasoning in videos. ECCV
  90. Zhou L, Kalantidis Y, Chen X, Corso JJ, & Rohrbach M (2019) Grounded video description. In CVPR
  91. Zhou L, Xu C, & Corso JJ (2017) Towards automatic learning of pro- cedures from web instructional videos. In AAAI
  92. Zhou B, Zhao H, Puig X, Fidler S, Barriuso A, & Torralba A (2017) Scene parsing through ADE20K dataset. In CVPR

Cited by

Sorted by citations

Authors
Damen, D., Doughty, H., Farinella, G. M., Furnari, A., Kazakos, E., Ma, J., Moltisanti, D., Munro, J., Perrett, T., Price, W., & Wray, M.
Affiliations
  • University of Bristol (GB)
  • University of Catania (IT)
Corresponding author
Damen, Dima
ORCID
Venue
International Journal of Computer Vision
Publisher
Springer Science and Business Media LLC
Published
2021-10-20
Volume
130
Issue
1
Pages
33-55
ISSN
  • 0920-5691 (ISSN-L)
  • 1573-1405
DOI
10.1007/s11263-021-01531-2
OpenAlex
W3207758636
Cited by
400
References
25
Citations by year
2026: 66 · 2025: 113 · 2024: 102 · 2023: 91 · 2022: 26 · 2021: 2
FWCI
19.87
Type
journal article
Language
en
Open access
Licence
Locations
  • International Journal of Computer Vision (publishedVersion, cc-by)
  • UvA-DARE (University of Amsterdam) (submittedVersion, cc-by)
  • DR-NTU (Nanyang Technological University) (submittedVersion, cc-by)
  • Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
Funders
Engineering and Physical Sciences Research Council (EP/T004991/1, EP/N033779/1)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
  • Physical Sciences › Computer Science › Human-Computer Interaction › Hand Gesture Recognition Systems
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Context-Aware Activity Recognition Systems
Keywords
egocentric vision, EPIC-KITCHENS-100, head-mounted camera, action detection, action prediction, human action recognition, weakly supervised learning, cross-modal retrieval, unsupervised domain adaptation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review