Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index; 12 of the 23 sources its claims cite were checked for support.

Benchmarking Multimodal Deep Fusion Strategies for Heterogeneous Neuroimaging and Cognitive Data Using a Controlled Sex Classification Task

Open access

Abstract

Background/Objectives: Multimodal data fusion is increasingly applied in neuroinformatics to integrate heterogeneous sources of information. However, the optimal strategies for combining modalities with markedly different dimensionality, scale, and noise characteristics remain unclear. To our knowledge, this is among the first systematic and controlled benchmarks explicitly disentangling the effects of fusion strategy and feature scaling within a unified deep learning framework. Methods: Using data from 747 healthy participants from the Human Connectome Project, we evaluated multiple fusion paradigms—including early fusion, attention-based fusion, subspace-based fusion, and graph-based fusion—within a unified and reproducible framework. Importantly, we assessed how different feature scaling techniques (Standard, Min–Max, and Robust scaling) interact with fusion strategies and influence model performance. Biological sex was used as a controlled benchmark task to focus on methodological insights rather than task-specific optimization. Results: Early feature-level fusion consistently achieved the highest classification performance across all evaluated configurations. In particular, direct concatenation of cognitive and neuroimaging features combined with Standard Scaling yielded the best results (AUC–ROC = 0.96 (0.95–0.96)), outperforming unimodal baselines as well as intermediate and late fusion strategies. Conclusions: This systematic benchmark demonstrates that multimodal deep learning performance in neuroscience is driven primarily by the interaction between fusion strategy and feature scaling rather than by architectural complexity alone. By explicitly disentangling the effects of fusion level and preprocessing within a unified framework, this study provides practical methodological guidance for the design, evaluation, and reproducible deployment of multimodal deep learning models in neuroscience.

Overview

CoChat Summary

TL;DR

This paper benchmarks unimodal and multimodal deep fusion strategies for combining cognitive and neuroimaging features, using biological sex as a controlled classification task in 747 HCP participants. It finds that early feature-level concatenation with Standard Scaling performs best (AUC–ROC = 0.96), outperforming more complex intermediate and late fusion approaches.

Key findings

  • Early feature-level fusion, especially direct concatenation of cognitive and neuroimaging features with Standard Scaling, achieved the highest performance: AUC–ROC = 0.96 (0.95–0.96), outperforming unimodal baselines and intermediate/late fusion strategies.
  • Unimodal neuroimaging features were stronger than cognitive features: neuroimaging-only reached AUC–ROC = 0.93 (0.93–0.94) under Robust Scaling, while cognitive-only reached AUC–ROC = 0.78 (0.77–0.81) under Robust Scaling.
  • Intermediate fusion was heterogeneous: Tabular Crossmodal Multi-Head Attention was stable across scaling (AUC–ROC = 0.92, 0.94, and 0.92 for Standard, Min–Max, and Robust), Edge Correlation GNN was strong under Standard Scaling (0.92 (0.90–0.94)) but declined under Min–Max/Robust, and Attention-Weighted GNN underperformed across pipelines (AUC–ROC ≈ 0.50). Late fusion (Tabular Decision) was stable but not superior (AUC–ROC 0.91–0.93).
  • Standard Scaling was generally the most effective preprocessing strategy, particularly for early fusion and graph-based models; Robust Scaling benefited cognitive features and selected intermediate models but was suboptimal for high-performing early fusion.

Methods

  • Data: WU-Minn HCP Young Adult dataset; after exclusions, final cohort of 747 healthy participants (410 females, 337 males), aged 22–36. Features were 36 cognitive/behavioral measures and 168 FreeSurfer brain morphology metrics, giving 204 features; the task was binary sex classification.
  • Pipeline: three scaling methods (Standard, Min–Max, Robust) were compared across unimodal baselines and multimodal fusion paradigms, including early/operation-based fusion (concat tabular data, concat feature maps, channel-wise multi net), attention-based fusion, subspace-based fusion (MCVAE), graph-based fusion (Edge Correlation GNN, Attention-Weighted GNN), and late fusion (Tabular Decision), implemented in Fusilli with PyTorch/PyTorch Lightning.
  • Evaluation: 3-fold cross-validation; primary metrics were AUC–ROC and accuracy, with balanced accuracy, F1-score, Cohen’s kappa, precision, recall, and average precision also reported; 95% CIs were computed via bootstrap resampling, and overfitting was monitored with learning curves and early stopping where supported.

Limitations

  • The paper states the analysis was restricted to two modalities (cognitive measures and imaging-derived brain morphology) and a single healthy cohort with a narrow age range (22–36), limiting generalizability to clinical populations or broader lifespan contexts.
  • The paper states that complete data availability and synchronized modalities may favor early joint feature learning; real-world missing, heterogeneous, or asynchronously acquired data may favor late fusion. The cohort was also restricted to participants who self-identified as White or Black, which may introduce selection bias and limit generalizability.
  • The paper states that interpretability remains a critical requirement and was not the focus; future work should include explainable AI, additional modalities, disease-specific cohorts, larger multicenter datasets, and fully nested validation. The text also makes evident that the single-dataset, 3-fold cross-validation design may limit robustness, and no explicit feature selection or dimensionality reduction was applied by design.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3390/brainsci16040405

Cites

48 of 48 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Proposed workflow for multimodal sex classification using cognitive and neuroimaging features. The pipeline integrates data from the WU–Minn Human Connectome Project (HCP) and is structured into three main stages: (A) feature scaling, (B) unimodal processing of cognitive and neuroimaging modalities, and (C–F) multimodal data fusion. Four fusion strategies are evaluated: (C) operation-based, (D) attention-based, (E) subspace-based, and (F) graph-based approaches.
Figure 1 Proposed workflow for multimodal sex classification using cognitive and neuroimaging features. The pipeline integrates data from the WU–Minn Human Connectome Project (HCP) and is structured into three main stages: (A) feature scaling, (B) unimodal processing of cognitive and neuroimaging modalities, and (C–F) multimodal data fusion. Four fusion strategies are evaluated: (C) operation-based, (D) attention-based, (E) subspace-based, and (F) graph-based approaches.CC BY · PubMed Central
Figure 2 Performance comparison of unimodal and multimodal fusion models across three feature scaling strategies: (a) Standard Scaling, (b) Min–Max Scaling, and (c) Robust Scaling. For each scaling method, boxplots report the distribution of AUC-ROC (left) and accuracy (right) across 3-fold cross-validation. Models include unimodal baselines (Tabular1, Tabular2), early fusion approaches (e.g., Concatenating Tabular Data, Concatenating Tabular Feature Maps), intermediate fusion models (e.g., Tabular Crossmodal Multi-Head Attention, MCVAE Tabular, Edge Correlation GNN, Attention-Weighted GNN), and decision-level late fusion (Tabular Decision). Pink boxplots represent AUC-ROC, while light blue boxplots represent accuracy.
Figure 2 Performance comparison of unimodal and multimodal fusion models across three feature scaling strategies: (a) Standard Scaling, (b) Min–Max Scaling, and (c) Robust Scaling. For each scaling method, boxplots report the distribution of AUC-ROC (left) and accuracy (right) across 3-fold cross-validation. Models include unimodal baselines (Tabular1, Tabular2), early fusion approaches (e.g., Concatenating Tabular Data, Concatenating Tabular Feature Maps), intermediate fusion models (e.g., Tabular Crossmodal Multi-Head Attention, MCVAE Tabular, Edge Correlation GNN, Attention-Weighted GNN), and decision-level late fusion (Tabular Decision). Pink boxplots represent AUC-ROC, while light blue boxplots represent accuracy.CC BY · PubMed Central
Figure 3 Receiver operating characteristic (ROC) and precision–recall (PR) curves for the best-performing models under Standard Scaling. (a) ROC curves and (b) PR curves are shown for selected unimodal and multimodal fusion models evaluated using 3-fold cross-validation. Curves are computed from aggregated predictions across folds, and AUC–ROC and average precision values are reported in the legend.
Figure 3 Receiver operating characteristic (ROC) and precision–recall (PR) curves for the best-performing models under Standard Scaling. (a) ROC curves and (b) PR curves are shown for selected unimodal and multimodal fusion models evaluated using 3-fold cross-validation. Curves are computed from aggregated predictions across folds, and AUC–ROC and average precision values are reported in the legend.CC BY · PubMed Central
Figure 4 Confusion matrices for the best-performing fusion model (Concatenating Tabular Data) across three preprocessing pipelines: (a) Standard Scaling, (b) Min–Max Scaling, and (c) Robust Scaling. Each matrix shows the distribution of predicted versus true class labels across the test folds. Standard Scaling yields the highest classification performance (AUC-ROC = 0.96 (0.95–0.96)) and the lowest number of misclassifications.
Figure 4 Confusion matrices for the best-performing fusion model (Concatenating Tabular Data) across three preprocessing pipelines: (a) Standard Scaling, (b) Min–Max Scaling, and (c) Robust Scaling. Each matrix shows the distribution of predicted versus true class labels across the test folds. Standard Scaling yields the highest classification performance (AUC-ROC = 0.96 (0.95–0.96)) and the lowest number of misclassifications.CC BY · PubMed Central

Verification

Claims the paper makes

8 Valid · 0 Not valid · 7 Not checked · 78 Own results and interpretations

42 more claims

Create a free account to see all 93 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 17
  • It may state two separate claims as one. 39

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

3 of 3

Citations

Cites

Show all 38 references

Research works printed but not linked: 3

  1. Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., and Ng, A. (2011). Multimodal Deep Learning, Springer.
  2. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems 30 (NIPS 2017), NeurIPS.
  3. Antelmi, L., Ayache, N., Robert, P., and Lorenzi, M. (2019, January 24). Sparse Multi-Channel Variational Autoencoder for the Joint Analysis of Heterogeneous Data. Proceedings of the 36th International Conference on Machine Learning, Long Beach, CA, USA.

Metadata

Authors
Camastra, C., Pelagi, A., Quattrone, A., & Sarica, A.
Affiliations
  • Centre for Addiction and Mental Health (CA)
  • Magna Graecia University (IT)
Corresponding author
Sarica, Alessia
Venue
Brain Sciences
Publisher
MDPI AG
Published
2026-04-10
Volume
16
Issue
4
Pages
405
ISSN
2076-3425 (ISSN-L)
OpenAlex
W7153217618
Cited by
1
References
38
Citations by year
2026: 1
FWCI
4.11
Type
journal article
Language
en
Open access
Locations
  • Brain Sciences (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • National Institutes of Health (1U54‐MH‐091657)
  • McDonnell Center for Systems Neuroscience (1U54MH091657)
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Functional Brain Connectivity Studies
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
Keywords
multimodal fusion, fusion strategies, feature scaling, sex classification, feature-level fusion, attention fusion, Human Connectome Project, benchmarking, reproducibility

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Benchmarking Multimodal Deep Fusion Strategies for Heterogeneous Neuroimaging and Cognitive Data Using a Controlled Sex Classification Task. https://app.cochat.ai/papers/10.3390/brainsci16040405
The paper
Camastra, C., Pelagi, A., Quattrone, A., & Sarica, A. (2026). Benchmarking Multimodal Deep Fusion Strategies for Heterogeneous Neuroimaging and Cognitive Data Using a Controlled Sex Classification Task. Brain Sciences. https://doi.org/10.3390/brainsci16040405

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.