CoChat

Effective Techniques for Multimodal Data Fusion: A Comparative Analysis

Open access

The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.

Abstract

Data processing in robotics is currently challenged by the effective building of multimodal and common representations. Tremendous volumes of raw data are available and their smart management is the core concept of multimodal learning in a new paradigm for data fusion. Although several techniques for building multimodal representations have been proven successful, they have not yet been analyzed and compared in a given production setting. This paper explored three of the most common techniques, (1) the late fusion, (2) the early fusion, and (3) the sketch, and compared them in classification tasks. Our paper explored different types of data (modalities) that could be gathered by sensors serving a wide range of sensor applications. Our experiments were conducted on Amazon Reviews, MovieLens25M, and Movie-Lens1M datasets. Their outcomes allowed us to confirm that the choice of fusion technique for building multimodal representation is crucial to obtain the highest possible model performance resulting from the proper modality combination. Consequently, we designed criteria for choosing this optimal data fusion technique.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3390/s23052381

Cites

31 of 31 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 The idea of binarizing the sketch. Instead of representing an observation with a subspace ID, it can be represented as a set of binary features. Then, 0 and 1 represent where the point lies concerning a single hyperplane. Such a sketch consumes much less memory and perhaps preserves more information about a single observation.
Figure 1 The idea of binarizing the sketch. Instead of representing an observation with a subspace ID, it can be represented as a set of binary features. Then, 0 and 1 represent where the point lies concerning a single hyperplane. Such a sketch consumes much less memory and perhaps preserves more information about a single observation.CC BY · PubMed Central
Figure 2 Here, we present the architectures used in the Amazon Reviews dataset. They are depicted in their trimodal forms. Bimodal and unimodal architectures look the same. The only difference is that unimodal models do not have a model on top of their outputs in the late fusion approach. Above, the output means the probabilities of observations belonging to respective classes.
Figure 2 Here, we present the architectures used in the Amazon Reviews dataset. They are depicted in their trimodal forms. Bimodal and unimodal architectures look the same. The only difference is that unimodal models do not have a model on top of their outputs in the late fusion approach. Above, the output means the probabilities of observations belonging to respective classes.CC BY · PubMed Central
Figure 3 Architectures used in the MovieLens25M dataset.
Figure 3 Architectures used in the MovieLens25M dataset.CC BY · PubMed Central
Figure 4 The architecture of gender classification. The user is represented as a sum of sketches. Then, a width-wise L2 normalization is performed. Finally, the sketch is flattened and fed into the classifier.
Figure 4 The architecture of gender classification. The user is represented as a sum of sketches. Then, a width-wise L2 normalization is performed. Finally, the sketch is flattened and fed into the classifier.CC BY · PubMed Central
Figure 5 The late fusion models results concerning categories.
Figure 5 The late fusion models results concerning categories.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 3

  1. Varshney, K. (2021, March 23). Trust in Machine Learning, Manning Publications, Shelter Island, Chapter 4 Data Sources and Biases, Section 4.1 Modalities. Available online: https://livebook.manning.com/book/trust-in-machine-learning/chapter-4/v-2/.
  2. Rychalska, B., Basaj, D.B., Dabrowski, J., and Daniluk, M. (2020). I know why you like this movie: Interpretable Efficient Multimodal Recommender. arXiv.
  3. Wirojwatanakul, P., and Wangperawong, A. (2019). Multi-Label Product Categorization Using Multi-Modal Fusion Models. arXiv.

Cited by

See all 220

Metadata

Authors
Pawłowski, M., Wróblewska, A., & Sysko-Romańczuk, S.
Affiliations
Warsaw University of Technology (PL)
Corresponding author
  • Pawłowski, Maciej
  • Wróblewska, Anna
Venue
Sensors
Publisher
MDPI AG
Published
2023-02-21
Volume
23
Issue
5
Pages
2381
ISSN
1424-8220 (ISSN-L)
OpenAlex
W4321497840
Cited by
180
References
30
Citations by year
2026: 50 · 2025: 94 · 2024: 30 · 2023: 6
FWCI
16.31
Type
journal article
Language
en
Open access
Locations
  • Sensors (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University) (cc-by)
Funders
  • Centre for Priority Research Area Artificial Intelligence and Robotics of Warsaw University of Technology within the Excellence Initiative: Research University (IDUB) programme (1820/27/Z01/POB2/2021, POIR.01.01.01-00-0066/22)
  • National Centre for Research and Development in Poland, Smart Growth Operational Program for 2014-2020, Digital Innovations (1820/27/Z01/POB2/2021, POIR.01.01.01-00-0066/22)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
Keywords
multimodal fusion, decision fusion, feature-level fusion, multimodal representation learning, classification, sensor data, recommender systems, Amazon reviews, MovieLens

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.