CoChat

Comprehensive Instructional Video Analysis: The COIN Dataset and Performance Evaluation

Open access
Tang, Yansong, Lu, Jiwen, Zhou, Jie · 2021 · IEEE Transactions on Pattern Analysis and Machine Intelligence
Cited by 42

The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.

Abstract

Thanks to the substantial and explosively increased instructional videos on the Internet, novices are able to acquire knowledge for completing various tasks. Over the past decade, growing efforts have been devoted to investigating the problem on instructional video analysis. However, most existing instructional video datasets have limitations in diversity and scale, which makes them far from many real-world applications where more diverse activities occur. To address this, in this article, we propose a large-scale dataset called "COIN" for COmprehensive INstructional video analysis. Organized with a hierarchical structure, the COIN dataset contains 11,827 videos of 180 tasks in 12 domains (e.g., vehicles, gadgets, etc.) related to our daily life. With a new developed toolbox, all the videos are annotated efficiently with a series of step labels and the corresponding temporal boundaries. In order to provide a benchmark for instructional video analysis, we evaluate plenty of approaches on our COIN dataset under five different settings. Furthermore, we exploit two important characteristics (i.e., task-consistency and ordering-dependency) for localizing important steps in instructional videos. Accordingly, we propose two simple yet effective methods, which can be easily plugged into conventional proposal-based action detection models. We believe the introduction of the COIN dataset will promote the future in-depth research on instructional video analysis for the community. Our dataset, annotation toolbox and source codes are available at http://coin-dataset.github.io.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tpami.2020.2980824

Cites

21 of 21 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 54

  1. 10.1007/978-3-319-46484-8_2 · DOI · Paper page
  2. 10.1109/ICCV.2015.510 · DOI · Paper page
  3. 10.1109/DICTA.2017.8227441 · DOI
  4. yosinski, 2014, How transferable are features in deep neural networks?, Proc 27th Int Conf Neural Inf Process Syst, 3320
  5. 10.1109/CVPR.2018.00629 · DOI
  6. 10.1109/ICCV.2017.617 · DOI · Paper page
  7. 10.1109/CVPR.2016.571 · DOI · Paper page
  8. 10.1109/CVPR.2018.00705 · DOI · Paper page
  9. 10.1007/978-3-319-46478-7_47 · DOI · Paper page
  10. 10.1109/CVPR.2018.00623 · DOI
  11. 10.1109/CVPR.2017.116 · DOI
  12. 10.1109/CVPR.2019.01234 · DOI
  13. 10.1109/ICCV.2017.83 · DOI · Paper page
  14. simonyan, 2014, Two-stream convolutional networks for action recognition in videos, Proc 27th Int Conf Neural Inf Process Syst, 568
  15. simonyan, 2015, Very deep convolutional networks for large-scale image recognition, Proc Int Conf Learn Representations, 1
  16. song, 2015, TVSum: Summarizing web videos using titles, Proc IEEE Conf Comput Vis Pattern Recognit, 5179
  17. 10.1109/CVPR.2018.00633 · DOI · Paper page
  18. gao, 2014, JHU-ISI gesture and skill assessment working set (JIGSAWS): A surgical activity dataset for human motion modeling, Proc Model Monit Comput Assisted Interv Workshop
  19. 10.1109/ICCV.2019.00756 · DOI · Paper page
  20. 10.1007/978-3-319-10584-0_33 · DOI · Paper page
  21. 10.1109/CVPR.2019.00130 · DOI · Paper page
  22. 10.1109/TIP.2019.2914577 · DOI
  23. 10.1109/CVPR.2019.00805 · DOI
  24. doughty, 2018, Who’s better? Who’s best? Pairwise deep ranking for skill determination, Proc IEEE Conf Comput Vis Pattern Recognit, 6057
  25. 10.1109/TPAMI.2015.2511748 · DOI · Paper page
  26. 10.7551/mitpress/7287.001.0001 · DOI · Paper page
  27. 10.1109/CVPR.2018.00560 · DOI · Paper page
  28. 10.1109/CVPRW.2017.16 · DOI · Paper page
  29. 10.1109/ICCV.2015.509 · DOI · Paper page
  30. 10.1109/CVPR.2018.00873 · DOI
  31. 10.1109/CVPR.2017.327 · DOI · Paper page
  32. 10.1109/CVPR.2012.6247801 · DOI · Paper page
  33. 10.3233/FI-2000-411207 · DOI · Paper page
  34. 10.1109/CVPR.2018.00771 · DOI
  35. 10.1109/CVPR.2018.00627 · DOI
  36. 10.1109/CVPR.2017.502 · DOI · Paper page
  37. chang, 2019, Procedure planning in instructional videos, CoRR, 1
  38. 10.1007/978-3-030-01225-0_1 · DOI · Paper page
  39. 10.1109/CVPR.2019.00366 · DOI
  40. chunhui, 2017, PKU-MMD: A large scale benchmark for continuous multi-modal human action understanding, Proc ACM Multimedia Workshop, 1
  41. 10.1109/CVPR.2013.340 · DOI · Paper page
  42. 10.1109/CVPR.2018.00911 · DOI · Paper page
  43. 10.1016/j.patrec.2010.08.004 · DOI · Paper page
  44. zhou, 2018, Towards automatic learning of procedures from web instructional videos, Proc AAAI Conf Artif Intell, 7590
  45. 10.1109/CVPR.2019.00365 · DOI · Paper page
  46. 10.1109/CVPR.2016.495 · DOI · Paper page
  47. 10.1109/TPAMI.2017.2749223 · DOI
  48. 10.1109/TIP.2017.2708902 · DOI
  49. 10.1214/aoms/1177699147 · DOI · Paper page
  50. 10.1109/CVPR42600.2020.00990 · DOI · Paper page
  51. 10.1177/0278364915602060 · DOI · Paper page
  52. 10.1109/CVPR.2019.00372 · DOI · Paper page
  53. 10.1109/CVPR.2019.00139 · DOI · Paper page
  54. 10.1109/TPAMI.2019.2927476 · DOI · Paper page

Cited by

See all 42

Metadata

Authors
Tang, Y., Lu, J., & Zhou, J.
Venue
IEEE Transactions on Pattern Analysis and Machine Intelligence
Published
2021-09-01
Volume
43
Issue
9
Pages
3138-3153
ISSN
  • 0162-8828 (ISSN-L)
  • 2160-9292
  • 1939-3539
OpenAlex
W3011215845
Cited by
42
References
21
FWCI
1.85
Type
journal article
Language
en
Open access
green
Funders
  • National Key Research and Development Program of China (2017YFA0700802)
  • National Natural Science Foundation of China (61822603, U1813218, U1713214, 61672306)
  • Beijing Academy of Artificial Intelligence (BAAI2020ZJ0202)
  • Shenzhen Fundamental Research and Discipline Layout project (JCYJ20170412170602564)
  • Tsinghua University Initiative Scientific Research Program

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.