Cited by 188Multimodal Machine Learning ApplicationsAdvanced Image and Video Retrieval TechniquesDomain Adaptation and Few-Shot Learning

Learning Aligned Cross-Modal Representations from Weakly Aligned Data

Castrejon, Lluis, Aytar, Yusuf, Vondrick, Carl, Pirsiavash, Hamed, Torralba, Antonio·2016·2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

Abstract

People can recognize scenes across many different modalities beyond natural images. In this paper, we investigate how to learn cross-modal scene representations that transfer across modalities. To study this problem, we introduce a new cross-modal scene dataset. While convolutional neural networks can categorize cross-modal scenes well, they also learn an intermediate representation not aligned across modalities, which is undesirable for crossmodal transfer applications. We present methods to regularize cross-modal convolutional neural networks so that they have a shared representation that is agnostic of the modality. Our experiments suggest that our scene representation can help transfer representations across modalities for retrieval. Moreover, our visualizations suggest that units emerge in the shared representation that tend to activate on consistent concepts independently of the modality.

Integrity signals

Registry

Crossref · 10.1109/cvpr.2016.321

Cites

48 of 48 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Pre-training with a diverse set of tasks improves the performance on downstream tasks.

    “Instead, pre-training with a diverse set of tasks [11,83] has been observed to improve the performance on downstream tasks [31,80] and potentially learn a better representation.”

    From MultiMAE: Multi-modal Multi-task Masked Autoencoders (2022) · cites this paper as [11]

Cites

Flagged first, then by citations

3 of the 51 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Authors
Castrejon, L., Aytar, Y., Vondrick, C., Pirsiavash, H., & Torralba, A.
Affiliations
  • MIT Computer Science and Artificial Intelligence Laboratory (US)
  • University of Maryland, Baltimore County (US)
Venue
2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher
IEEE
Published
2016-06
Pages
2940-2949
ISSN
1063-6919 (ISSN-L)
OpenAlex
W2474574787
Cited by
188
References
51
Citations by year
2026: 7 · 2025: 5 · 2024: 9 · 2023: 12 · 2022: 22 · 2021: 17 · 2020: 49 · 2019: 25 · 2018: 17 · 2017: 17 · 2016: 7 · 2015: 1
FWCI
7.64
Type
conference paper
Language
en
Open access
closed
Locations
IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
Keywords
cross-modal transfer, shared representations, cross-modal retrieval, scene classification

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review