CoChat
Sign Up
Cited by 117Multimodal Machine Learning ApplicationsTopic ModelingAdvanced Image and Video Retrieval Techniques

CLIP-Event: Connecting Text and Images with Event Structures

Open access
Li, Manling, Xu, Ruochen, Wang, Shuohang, Zhou, Luowei, Lin, Xudong, Zhu, Chenguang, Zeng, Michael, Ji, Heng +1 more·2022·2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
DOI 10.1109/cvpr52688.2022.01593
Open this paper in CoChat — free accountOpen PDF

Abstract

Vision-language (V+L) pretraining models have achieved great success in supporting multimedia applications by understanding the alignments between images and text. While existing vision-language pretraining models primarily focus on understanding objects in images or entities in text, they often ignore the alignment at the level of events and their argument structures. In this work, we propose a contrastive learning framework to enforce vision-language pretraining models to comprehend events and associated argument (participant) roles. To achieve this, we take advantage of text information extraction technologies to obtain event structural knowledge, and utilize multiple prompt functions to contrast difficult negative descriptions by manipulating event structures. We also design an event graph alignment loss based on optimal transport to capture event argument structures. In addition, we collect a large event-rich dataset (106,875 images) for pretraining, which provides a more challenging image retrieval benchmark to assess the understanding of complicated lengthy sentences11The data and code are publicly available for research purpose in https://github.com/limanling/clip-event.. Experiments show that our zero-shot CLIP-Event outperforms the state-of-the-art supervised model in argument extraction on Multimedia Event Extraction, achieving more than 5% absolute F-score gain in event extraction, as well as significant improvements on a variety of downstream tasks under zero-shot settings.

Integrity signals

Registry

Crossref · 10.1109/cvpr52688.2022.01593

Cites

43 of 43 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Recent attempts mainly use two straightforward methods to improve robustness for multimodal Transformer models: augmentation and adversarial learning based strategies, and fine-grained loss functions.

    “Recent attempts mainly use two straightforward methods to improve the robustness for multimodal Transformer models: (1) augmentation and adversarial learning based strategies [293], [294], (2) fine-grained loss functions [295].”

    From Multimodal learning with transformers: A survey (2023) · cites this paper as [294]

  • Thanks to large model capacity, Transformer models handle challenging domain gaps and shifts better via effective pretraining on large-scale corpora.

    “Thanks to the large model capacity, Transformer models handle the challenging domain gaps and shifts (e.g., linguistic and visual) better via effective pretraining on large-scale corpora [294].”

    From Multimodal learning with transformers: A survey (2023) · cites this paper as [294]

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Li, M., Xu, R., Wang, S., Zhou, L., Lin, X., Zhu, C., Zeng, M., Ji, H., & Chang, S.-F.
Affiliations
  • University of Illinois Urbana-Champaign (US)
  • Microsoft (United States) (US)
  • Columbia University (US)
ORCID
Venue
2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher
IEEE
Published
2022-06
Pages
16399-16408
ISSN
1063-6919 (ISSN-L)
DOI
10.1109/cvpr52688.2022.01593
OpenAlex
W4312407537
Cited by
117
References
42
Citations by year
2026: 8 · 2025: 24 · 2024: 27 · 2023: 46 · 2022: 12
FWCI
9.88
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Defense Advanced Research Projects Agency (FA8750-19-2-1004, FA8750, FA8750-18-2-0014)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
Keywords
vision-language pre-training, argument structure, contrastive learning, optimal transport, image-text alignment, zero-shot learning, image retrieval

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review