Comprehensive Instructional Video Analysis: The COIN Dataset and Performance Evaluation
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
Thanks to the substantial and explosively increased instructional videos on the Internet, novices are able to acquire knowledge for completing various tasks. Over the past decade, growing efforts have been devoted to investigating the problem on instructional video analysis. However, most existing instructional video datasets have limitations in diversity and scale, which makes them far from many real-world applications where more diverse activities occur. To address this, in this article, we propose a large-scale dataset called "COIN" for COmprehensive INstructional video analysis. Organized with a hierarchical structure, the COIN dataset contains 11,827 videos of 180 tasks in 12 domains (e.g., vehicles, gadgets, etc.) related to our daily life. With a new developed toolbox, all the videos are annotated efficiently with a series of step labels and the corresponding temporal boundaries. In order to provide a benchmark for instructional video analysis, we evaluate plenty of approaches on our COIN dataset under five different settings. Furthermore, we exploit two important characteristics (i.e., task-consistency and ordering-dependency) for localizing important steps in instructional videos. Accordingly, we propose two simple yet effective methods, which can be easily plugged into conventional proposal-based action detection models. We believe the introduction of the COIN dataset will promote the future in-depth research on instructional video analysis for the community. Our dataset, annotation toolbox and source codes are available at http://coin-dataset.github.io.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Action Recognition with Improved Trajectories
Wang, H., & Schmid, C. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.441 · Cited by 3,407
ActivityNet: A large-scale video benchmark for human activity understanding
Heilbron, F. C., Escorcia, V., Ghanem, B., & Niebles, J. C. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298698 · Cited by 2,578
A Duality Based Approach for Realtime TV-L 1 Optical Flow
Zach, C., Pock, T., & Bischof, H. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-74936-3_22 · Cited by 1,658 · cited 1×
“Here SSN−Flow adopted the optical flows calculated by [78], and SSN−Fusion combined the predicted scores of SSN−RGB and SSN−Flow.”
MethodLong-Term Recurrent Convolutional Networks for Visual Recognition and Description
Donahue, J., Hendricks, L. A., Rohrbach, M., Venugopalan, S., Guadarrama, S., Saenko, K., & Darrell, T. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2599174 · Cited by 1,547
Anderson, P., Wu, Q., Teney, D., Bruce, J., Johnson, M., Sunderhauf, N., Reid, I., Gould, S., & van den Hengel, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00387 · Cited by 1,187
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
Miech, A., Zhukov, D., Alayrac, J.-B., Tapaswi, M., Laptev, I., & Sivic, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00272 · Cited by 986 · cited 3×
“More recently, Miech et al. [46] introduced another large-scale dataset HowTo100M, which contained 136 million clips sourced from 1.238 million instructional videos.”
BackgroundMethodInfluentialTemporal Action Detection with Structured Segment Networks
Zhao, Y., Xiong, Y., Wang, L., Wu, Z., Tang, X., & Lin, D. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.317 · Cited by 892
Koedinger, K. R., Corbett, A. T., & Perfetti, C. (2012) · Cognitive Science · doi:10.1111/j.1551-6709.2012.01245.x · Cited by 772 · cited 1×
“Over the past decades, learning from instruction has become an important topic in various areas such as cognition science [34], educational psychology [54], and the intersection of computer vision, nature language processing and robotics [7], [8], [47].”
Background
Research works printed but not linked: 54
- 10.1007/978-3-319-46484-8_2 · DOI · Paper page
- 10.1109/ICCV.2015.510 · DOI · Paper page
- 10.1109/DICTA.2017.8227441 · DOI
- yosinski, 2014, How transferable are features in deep neural networks?, Proc 27th Int Conf Neural Inf Process Syst, 3320
- 10.1109/CVPR.2018.00629 · DOI
- 10.1109/ICCV.2017.617 · DOI · Paper page
- 10.1109/CVPR.2016.571 · DOI · Paper page
- 10.1109/CVPR.2018.00705 · DOI · Paper page
- 10.1007/978-3-319-46478-7_47 · DOI · Paper page
- 10.1109/CVPR.2018.00623 · DOI
- 10.1109/CVPR.2017.116 · DOI
- 10.1109/CVPR.2019.01234 · DOI
- 10.1109/ICCV.2017.83 · DOI · Paper page
- simonyan, 2014, Two-stream convolutional networks for action recognition in videos, Proc 27th Int Conf Neural Inf Process Syst, 568
- simonyan, 2015, Very deep convolutional networks for large-scale image recognition, Proc Int Conf Learn Representations, 1
- song, 2015, TVSum: Summarizing web videos using titles, Proc IEEE Conf Comput Vis Pattern Recognit, 5179
- 10.1109/CVPR.2018.00633 · DOI · Paper page
- gao, 2014, JHU-ISI gesture and skill assessment working set (JIGSAWS): A surgical activity dataset for human motion modeling, Proc Model Monit Comput Assisted Interv Workshop
- 10.1109/ICCV.2019.00756 · DOI · Paper page
- 10.1007/978-3-319-10584-0_33 · DOI · Paper page
- 10.1109/CVPR.2019.00130 · DOI · Paper page
- 10.1109/TIP.2019.2914577 · DOI
- 10.1109/CVPR.2019.00805 · DOI
- doughty, 2018, Who’s better? Who’s best? Pairwise deep ranking for skill determination, Proc IEEE Conf Comput Vis Pattern Recognit, 6057
- 10.1109/TPAMI.2015.2511748 · DOI · Paper page
- 10.7551/mitpress/7287.001.0001 · DOI · Paper page
- 10.1109/CVPR.2018.00560 · DOI · Paper page
- 10.1109/CVPRW.2017.16 · DOI · Paper page
- 10.1109/ICCV.2015.509 · DOI · Paper page
- 10.1109/CVPR.2018.00873 · DOI
- 10.1109/CVPR.2017.327 · DOI · Paper page
- 10.1109/CVPR.2012.6247801 · DOI · Paper page
- 10.3233/FI-2000-411207 · DOI · Paper page
- 10.1109/CVPR.2018.00771 · DOI
- 10.1109/CVPR.2018.00627 · DOI
- 10.1109/CVPR.2017.502 · DOI · Paper page
- chang, 2019, Procedure planning in instructional videos, CoRR, 1
- 10.1007/978-3-030-01225-0_1 · DOI · Paper page
- 10.1109/CVPR.2019.00366 · DOI
- chunhui, 2017, PKU-MMD: A large scale benchmark for continuous multi-modal human action understanding, Proc ACM Multimedia Workshop, 1
- 10.1109/CVPR.2013.340 · DOI · Paper page
- 10.1109/CVPR.2018.00911 · DOI · Paper page
- 10.1016/j.patrec.2010.08.004 · DOI · Paper page
- zhou, 2018, Towards automatic learning of procedures from web instructional videos, Proc AAAI Conf Artif Intell, 7590
- 10.1109/CVPR.2019.00365 · DOI · Paper page
- 10.1109/CVPR.2016.495 · DOI · Paper page
- 10.1109/TPAMI.2017.2749223 · DOI
- 10.1109/TIP.2017.2708902 · DOI
- 10.1214/aoms/1177699147 · DOI · Paper page
- 10.1109/CVPR42600.2020.00990 · DOI · Paper page
- 10.1177/0278364915602060 · DOI · Paper page
- 10.1109/CVPR.2019.00372 · DOI · Paper page
- 10.1109/CVPR.2019.00139 · DOI · Paper page
- 10.1109/TPAMI.2019.2927476 · DOI · Paper page
Cited by
Uncertainty-Aware Score Distribution Learning for Action Quality Assessment
Tang, Y., Ni, Z., Zhou, J., Zhang, D., Lu, J., Wu, Y., & Zhou, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00986 · Cited by 209
Self-citationEgo-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
Grauman, K., Westbury, A., Torresani, L., Kitani, K., Malik, J., Afouras, T., Ashutosh, K., Baiyya, V., Bansal, S., Boote, B., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01834 · Cited by 182
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
Grauman, K., Westbury, A., Torresani, L., Kitani, K., Malik, J., Afouras, T., Ashutosh, K., Baiyya, V., Bansal, S., Boote, B., et al. (2025) · International Journal of Computer Vision · doi:10.1007/s11263-025-02557-6 · Cited by 3
“[10, 126, 145, 148] or third-person [9, 100, 159, 205, 207] videos; however, limited work considers both views together.”
Metadata
- Authors
- Tang, Y., Lu, J., & Zhou, J.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2021-09-01
- Volume
- 43
- Issue
- 9
- Pages
- 3138-3153
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W3011215845
- Cited by
- 42
- References
- 21
- FWCI
- 1.85
- Type
- journal article
- Language
- en
- Open access
- green
- Licence
- Funders
- National Key Research and Development Program of China (2017YFA0700802)
- National Natural Science Foundation of China (61822603, U1813218, U1713214, 61672306)
- Beijing Academy of Artificial Intelligence (BAAI2020ZJ0202)
- Shenzhen Fundamental Research and Discipline Layout project (JCYJ20170412170602564)
- Tsinghua University Initiative Scientific Research Program
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Comprehensive Instructional Video Analysis: The COIN Dataset and Performance Evaluation. https://app.cochat.ai/papers/10.1109/tpami.2020.2980824
- The paper
- Tang, Y., Lu, J., & Zhou, J. (2021). Comprehensive Instructional Video Analysis: The COIN Dataset and Performance Evaluation. IEEE Transactions on Pattern Analysis and Machine Intelligence. https://doi.org/10.1109/tpami.2020.2980824
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.