CoChat
Sign Up
Cited by 410Multimodal Machine Learning ApplicationsHuman Pose and Action RecognitionVideo Analysis and Summarization

The DOI matches its Crossref record; 79 of the 79 references it cites were found in a registry or an index.

HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training

Open access
Li, Linjie, Chen, Yen-Chun, Cheng, Yu, Gan, Zhe, Yu, Licheng, Liu, Jingjing·2020·Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
DOI 10.18653/v1/2020.emnlp-main.161
Open this paper in CoChat — free accountOpen PDF

Abstract

We present HERO, a novel framework for large-scale video+language omnirepresentation learning.HERO encodes multimodal inputs in a hierarchical structure, where local context of a video frame is captured by a Cross-modal Transformer via multimodal fusion, and global video context is captured by a Temporal Transformer.In addition to standard Masked Language Modeling (MLM) and Masked Frame Modeling (MFM) objectives, we design two new pre-training tasks: (i) Video-Subtitle Matching (VSM), where the model predicts both global and local temporal alignment; and (ii) Frame Order Modeling (FOM), where the model predicts the right order of shuffled video frames.HERO is jointly trained on HowTo100M and large-scale TV datasets to gain deep understanding of complex social dynamics with multi-character interactions.Comprehensive experiments demonstrate that HERO achieves new state of the art on multiple benchmarks over Text-based Video/Video-moment Retrieval, Video Question Answering (QA), Video-and-language Inference and Video Captioning tasks across different domains.We also introduce two new challenging benchmarks How2QA and How2R for Video QA and Retrieval, collected from diverse video content over multimodalities. 1

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2020.emnlp-main.161

Cites

79 of 79 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 HERO Architecture (best viewed in color), consisting of Cross-Modal Transformer and Temporal Transformer, learned via four pre-training tasks hierarchically. Initial frame features are obtained by SlowFast and ResNet feature extractors, and word embeddings are learned via an embedding layer initialized from RoBERTa.
Figure 1 HERO Architecture (best viewed in color), consisting of Cross-Modal Transformer and Temporal Transformer, learned via four pre-training tasks hierarchically. Initial frame features are obtained by SlowFast and ResNet feature extractors, and word embeddings are learned via an embedding layer initialized from RoBERTa.CC-BY · From the PDF
Image 2
CC-BY · From the PDF
Figure 3 Visualization of the attention maps learned by Cross-modal Transformers of HERO model.
Figure 3 Visualization of the attention maps learned by Cross-modal Transformers of HERO model.CC-BY · From the PDF
Figure 4 HERO model adapted to downstream task: Text-based Video Moment Retrieval.
Figure 4 HERO model adapted to downstream task: Text-based Video Moment Retrieval.CC-BY · From the PDF
Figure 5 HERO model adapted to downstream task: Video Question Answering.
Figure 5 HERO model adapted to downstream task: Video Question Answering.CC-BY · From the PDF
Image 6
CC-BY · From the PDF
Image 7
CC-BY · From the PDF
(b) User interface for question/answer annotation. Each worker is provided with a segmented clip and required to write a question with four answers in the text boxes.
(b) User interface for question/answer annotation. Each worker is provided with a segmented clip and required to write a question with four answers in the text boxes.CC-BY · From the PDF
Figure 7 Distribution of video segment length.
Figure 7 Distribution of video segment length.CC-BY · From the PDF
Image 10
CC-BY · From the PDF
Figure 9 How2QA question length distribution.
Figure 9 How2QA question length distribution.CC-BY · From the PDF
Figure 10 How2QA answer length distribution.
Figure 10 How2QA answer length distribution.CC-BY · From the PDF
Figure 11 Distribution of questions categorized by their leading words in How2QA.
Figure 11 Distribution of questions categorized by their leading words in How2QA.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 4

  1. Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. 2017b. Localizing moments in video with natural language. In CVPR.
  2. Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine trans- lation. arXiv preprint arXiv:1806.00187. · arXiv
  3. Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016b. Msr- vtt: A large video description dataset for bridging video and language. CVPR.
  4. Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, and Tamara L Berg. 2018a. Mat- tnet: Modular attention network for referring expres- sion comprehension. In CVPR.

Cited by

Sorted by citations

See all 410

Related

See all

Metadata

Authors
Li, L., Chen, Y.-C., Cheng, Y., Gan, Z., Yu, L., & Liu, J.
Affiliations
Microsoft (United States) (US)
ORCID
Venue
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Publisher
Association for Computational Linguistics
Published
2020
Pages
2046-2065
DOI
10.18653/v1/2020.emnlp-main.161
OpenAlex
W3105232955
Cited by
408
References
79
Citations by year
2026: 17 · 2025: 49 · 2024: 67 · 2023: 129 · 2022: 87 · 2021: 52 · 2020: 7
FWCI
21.21
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Video Analysis and Summarization
Keywords
multimodal representation learning, hierarchical encoder, cross-modal transformer, temporal transformer, video question answering, video retrieval, video captioning, video moment retrieval

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review