Comprehensive Instructional Video Analysis: The COIN Dataset and Performance Evaluation
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
Thanks to the substantial and explosively increased instructional videos on the Internet, novices are able to acquire knowledge for completing various tasks. Over the past decade, growing efforts have been devoted to investigating the problem on instructional video analysis. However, most existing instructional video datasets have limitations in diversity and scale, which makes them far from many real-world applications where more diverse activities occur. To address this, in this article, we propose a large-scale dataset called "COIN" for COmprehensive INstructional video analysis. Organized with a hierarchical structure, the COIN dataset contains 11,827 videos of 180 tasks in 12 domains (e.g., vehicles, gadgets, etc.) related to our daily life. With a new developed toolbox, all the videos are annotated efficiently with a series of step labels and the corresponding temporal boundaries. In order to provide a benchmark for instructional video analysis, we evaluate plenty of approaches on our COIN dataset under five different settings. Furthermore, we exploit two important characteristics (i.e., task-consistency and ordering-dependency) for localizing important steps in instructional videos. Accordingly, we propose two simple yet effective methods, which can be easily plugged into conventional proposal-based action detection models. We believe the introduction of the COIN dataset will promote the future in-depth research on instructional video analysis for the community. Our dataset, annotation toolbox and source codes are available at http://coin-dataset.github.io.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Action Recognition with Improved Trajectories
Wang, H., & Schmid, C. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.441 · Cited by 3,407
ActivityNet: A large-scale video benchmark for human activity understanding
Heilbron, F. C., Escorcia, V., Ghanem, B., & Niebles, J. C. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298698 · Cited by 2,578
A Duality Based Approach for Realtime TV-L 1 Optical Flow
Zach, C., Pock, T., & Bischof, H. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-74936-3_22 · Cited by 1,658 · cited 1×
“Here SSN−Flow adopted the optical flows calculated by [78], and SSN−Fusion combined the predicted scores of SSN−RGB and SSN−Flow.”
MethodLong-Term Recurrent Convolutional Networks for Visual Recognition and Description
Donahue, J., Hendricks, L. A., Rohrbach, M., Venugopalan, S., Guadarrama, S., Saenko, K., & Darrell, T. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2599174 · Cited by 1,547
Anderson, P., Wu, Q., Teney, D., Bruce, J., Johnson, M., Sunderhauf, N., Reid, I., Gould, S., & van den Hengel, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00387 · Cited by 1,187
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
Miech, A., Zhukov, D., Alayrac, J.-B., Tapaswi, M., Laptev, I., & Sivic, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00272 · Cited by 986 · cited 3×
“More recently, Miech et al. [46] introduced another large-scale dataset HowTo100M, which contained 136 million clips sourced from 1.238 million instructional videos.”
BackgroundMethodInfluentialTemporal Action Detection with Structured Segment Networks
Zhao, Y., Xiong, Y., Wang, L., Wu, Z., Tang, X., & Lin, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.317 · Cited by 892
Koedinger, K. R., Corbett, A. T., & Perfetti, C. (2012) · Cognitive Science · doi:10.1111/j.1551-6709.2012.01245.x · Cited by 772 · cited 1×
“Over the past decades, learning from instruction has become an important topic in various areas such as cognition science [34], educational psychology [54], and the intersection of computer vision, nature language processing and robotics [7], [8], [47].”
BackgroundBMN: Boundary-Matching Network for Temporal Action Proposal Generation
Lin, T., Liu, X., Li, X., Ding, E., & Wen, S. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00399 · Cited by 694 · cited 5×
“Besides, we include the results based on two state-of-the-art methods ( i.e., BMN [39] and BSN [40]) to see if the improvements are subtle or significant.”
BackgroundMethodInfluentialMS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation
Farha, Y. A., & Gall, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00369 · Cited by 673 · cited 1×
“(TDRN) [38], multi-stage temporal convolutional network (MS-TCN) [24], etc.”
The Language of Actions: Recovering the Syntax and Semantics of Goal-Directed Human Activities
Kuehne, H., Arslan, A., & Serre, T. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.105 · Cited by 571 · cited 10×
“For the action segmentation task, Kuehne et al. [36] developed a hierarchical model based on HMMs and a context-free grammar to parse the main steps in the cooking activities.”
BackgroundMethodInfluentialA Multi-stream Bi-directional Recurrent Neural Network for Fine-Grained Action Detection
Singh, B., Marks, T. K., Jones, M., Tuzel, O., & Shao, M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.216 · Cited by 461
Combining embedded accelerometers with computer vision for recognizing food preparation activities
Stein, S., & McKenna, S. J. (2013) · Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing · doi:10.1145/2493432.2493482 · Cited by 448 · cited 1×
“In the meantime, a number of datasets for instructional video analysis [6], [15], [36], [56], [65], [70], [82] have been collected in the community.”
BackgroundGaussian Temporal Awareness Networks for Action Localization
Long, F., Yao, T., Qiu, Z., Tian, X., Luo, J., & Mei, T. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00043 · Cited by 380
Scaling Egocentric Vision: The EPIC-KITCHENS Dataset
Damen, D., Doughty, H., Farinella, G. M., Fidler, S., Furnari, A., Kazakos, E., Moltisanti, D., Munro, J., Perrett, T., Price, W., et al. (2018) · arXiv · doi:10.48550/arxiv.1804.02748 · Cited by 274 · cited 1×
“6h 206 1,703 SA self-collected 7 SD + TB 3 2014 “5 tasks” [6] 5h 150 CT YouTube 7 SD + TB 5 2016 Ikea-FA [70] 3h,50m 101 1,911 AF self-collected 7 SL + TB - 2017 Recipe1M† [57] 432 CA 7 SD - 2017 Recipe1M+† [44] - 13,735,679 CA Google 7 SD - 2018 YouCook2 [82] 176h 2,000 13,829 CA YouTube 7 SD + TB + OBL 89 2018 EPIC-…”
Uncertainty-Aware Score Distribution Learning for Action Quality Assessment
Tang, Y., Ni, Z., Zhou, J., Zhang, D., Lu, J., Wu, Y., & Zhou, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00986 · Cited by 209
Self-citationWeakly-supervised action segmentation with iterative soft boundary assignment
Ding, L., & Xu, C. (2018) · arXiv · doi:10.48550/arxiv.1803.10699 · Cited by 124 · cited 5×
“When taking into account the ordering information, NN-Viterbi [53] and TCFPN-ISBA [19] outperform Action-Sets with a large margin of 16.”
MethodInfluentialA Perceptual Prediction Framework for Self Supervised Event Segmentation
Aakur, S. N., & Sarkar, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00129 · Cited by 79
Optimizing the number of steps in learning tasks for complex skills
Nadolski, R. J., Kirschner, P. A., & Van Merriënboer, J. J. G. (2005) · British Journal of Educational Psychology · doi:10.1348/000709904x22403 · Cited by 70 · cited 2×
“As suggested by the scientists in educational psychology [54], novices often face difficulties in learning from the whole realistic task, and it is necessary to divide the whole task into smaller segments or steps as a form of simplification.”
Background
Research works printed but not linked: 54
- 10.1007/978-3-319-46484-8_2 · DOI · Paper page
- 10.1109/ICCV.2015.510 · DOI · Paper page
- 10.1109/DICTA.2017.8227441 · DOI
- yosinski, 2014, How transferable are features in deep neural networks?, Proc 27th Int Conf Neural Inf Process Syst, 3320
- 10.1109/CVPR.2018.00629 · DOI
- 10.1109/ICCV.2017.617 · DOI · Paper page
- 10.1109/CVPR.2016.571 · DOI · Paper page
- 10.1109/CVPR.2018.00705 · DOI · Paper page
- 10.1007/978-3-319-46478-7_47 · DOI · Paper page
- 10.1109/CVPR.2018.00623 · DOI
- 10.1109/CVPR.2017.116 · DOI
- 10.1109/CVPR.2019.01234 · DOI
- 10.1109/ICCV.2017.83 · DOI · Paper page
- simonyan, 2014, Two-stream convolutional networks for action recognition in videos, Proc 27th Int Conf Neural Inf Process Syst, 568
- simonyan, 2015, Very deep convolutional networks for large-scale image recognition, Proc Int Conf Learn Representations, 1
- song, 2015, TVSum: Summarizing web videos using titles, Proc IEEE Conf Comput Vis Pattern Recognit, 5179
- 10.1109/CVPR.2018.00633 · DOI · Paper page
- gao, 2014, JHU-ISI gesture and skill assessment working set (JIGSAWS): A surgical activity dataset for human motion modeling, Proc Model Monit Comput Assisted Interv Workshop
- 10.1109/ICCV.2019.00756 · DOI · Paper page
- 10.1007/978-3-319-10584-0_33 · DOI · Paper page
- 10.1109/CVPR.2019.00130 · DOI · Paper page
- 10.1109/TIP.2019.2914577 · DOI
- 10.1109/CVPR.2019.00805 · DOI
- doughty, 2018, Who’s better? Who’s best? Pairwise deep ranking for skill determination, Proc IEEE Conf Comput Vis Pattern Recognit, 6057
- 10.1109/TPAMI.2015.2511748 · DOI · Paper page
- 10.7551/mitpress/7287.001.0001 · DOI · Paper page
- 10.1109/CVPR.2018.00560 · DOI · Paper page
- 10.1109/CVPRW.2017.16 · DOI · Paper page
- 10.1109/ICCV.2015.509 · DOI · Paper page
- 10.1109/CVPR.2018.00873 · DOI
- 10.1109/CVPR.2017.327 · DOI · Paper page
- 10.1109/CVPR.2012.6247801 · DOI · Paper page
- 10.3233/FI-2000-411207 · DOI · Paper page
- 10.1109/CVPR.2018.00771 · DOI
- 10.1109/CVPR.2018.00627 · DOI
- 10.1109/CVPR.2017.502 · DOI · Paper page
- chang, 2019, Procedure planning in instructional videos, CoRR, 1
- 10.1007/978-3-030-01225-0_1 · DOI · Paper page
- 10.1109/CVPR.2019.00366 · DOI
- chunhui, 2017, PKU-MMD: A large scale benchmark for continuous multi-modal human action understanding, Proc ACM Multimedia Workshop, 1
- 10.1109/CVPR.2013.340 · DOI · Paper page
- 10.1109/CVPR.2018.00911 · DOI · Paper page
- 10.1016/j.patrec.2010.08.004 · DOI · Paper page
- zhou, 2018, Towards automatic learning of procedures from web instructional videos, Proc AAAI Conf Artif Intell, 7590
- 10.1109/CVPR.2019.00365 · DOI · Paper page
- 10.1109/CVPR.2016.495 · DOI · Paper page
- 10.1109/TPAMI.2017.2749223 · DOI
- 10.1109/TIP.2017.2708902 · DOI
- 10.1214/aoms/1177699147 · DOI · Paper page
- 10.1109/CVPR42600.2020.00990 · DOI · Paper page
- 10.1177/0278364915602060 · DOI · Paper page
- 10.1109/CVPR.2019.00372 · DOI · Paper page
- 10.1109/CVPR.2019.00139 · DOI · Paper page
- 10.1109/TPAMI.2019.2927476 · DOI · Paper page
Cited by
Uncertainty-Aware Score Distribution Learning for Action Quality Assessment
Tang, Y., Ni, Z., Zhou, J., Zhang, D., Lu, J., Wu, Y., & Zhou, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00986 · Cited by 209
Self-citationEgo-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
Grauman, K., Westbury, A., Torresani, L., Kitani, K., Malik, J., Afouras, T., Ashutosh, K., Baiyya, V., Bansal, S., Boote, B., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01834 · Cited by 182
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
Grauman, K., Westbury, A., Torresani, L., Kitani, K., Malik, J., Afouras, T., Ashutosh, K., Baiyya, V., Bansal, S., Boote, B., et al. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02557-6 · Cited by 3
“[10, 126, 145, 148] or third-person [9, 100, 159, 205, 207] videos; however, limited work considers both views together.”
Metadata
- Authors
- Tang, Y., Lu, J., & Zhou, J.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2021-09-01
- Volume
- 43
- Issue
- 9
- Pages
- 3138-3153
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W3011215845
- Cited by
- 42
- References
- 21
- FWCI
- 1.85
- Type
- journal article
- Language
- en
- Open access
- green
- Licence
- Funders
- National Key Research and Development Program of China (2017YFA0700802)
- National Natural Science Foundation of China (61822603, U1813218, U1713214, 61672306)
- Beijing Academy of Artificial Intelligence (BAAI2020ZJ0202)
- Shenzhen Fundamental Research and Discipline Layout project (JCYJ20170412170602564)
- Tsinghua University Initiative Scientific Research Program
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.