The DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index; 12 of the 23 sources its claims cite were checked for support.
Benchmarking Multimodal Deep Fusion Strategies for Heterogeneous Neuroimaging and Cognitive Data Using a Controlled Sex Classification Task
Open accessAbstract
Background/Objectives: Multimodal data fusion is increasingly applied in neuroinformatics to integrate heterogeneous sources of information. However, the optimal strategies for combining modalities with markedly different dimensionality, scale, and noise characteristics remain unclear. To our knowledge, this is among the first systematic and controlled benchmarks explicitly disentangling the effects of fusion strategy and feature scaling within a unified deep learning framework. Methods: Using data from 747 healthy participants from the Human Connectome Project, we evaluated multiple fusion paradigms—including early fusion, attention-based fusion, subspace-based fusion, and graph-based fusion—within a unified and reproducible framework. Importantly, we assessed how different feature scaling techniques (Standard, Min–Max, and Robust scaling) interact with fusion strategies and influence model performance. Biological sex was used as a controlled benchmark task to focus on methodological insights rather than task-specific optimization. Results: Early feature-level fusion consistently achieved the highest classification performance across all evaluated configurations. In particular, direct concatenation of cognitive and neuroimaging features combined with Standard Scaling yielded the best results (AUC–ROC = 0.96 (0.95–0.96)), outperforming unimodal baselines as well as intermediate and late fusion strategies. Conclusions: This systematic benchmark demonstrates that multimodal deep learning performance in neuroscience is driven primarily by the interaction between fusion strategy and feature scaling rather than by architectural complexity alone. By explicitly disentangling the effects of fusion level and preprocessing within a unified framework, this study provides practical methodological guidance for the design, evaluation, and reproducible deployment of multimodal deep learning models in neuroscience.
Overview
CoChat Summary
TL;DR
This paper benchmarks unimodal and multimodal deep fusion strategies for combining cognitive and neuroimaging features, using biological sex as a controlled classification task in 747 HCP participants. It finds that early feature-level concatenation with Standard Scaling performs best (AUC–ROC = 0.96), outperforming more complex intermediate and late fusion approaches.
Key findings
- Early feature-level fusion, especially direct concatenation of cognitive and neuroimaging features with Standard Scaling, achieved the highest performance: AUC–ROC = 0.96 (0.95–0.96), outperforming unimodal baselines and intermediate/late fusion strategies.
- Unimodal neuroimaging features were stronger than cognitive features: neuroimaging-only reached AUC–ROC = 0.93 (0.93–0.94) under Robust Scaling, while cognitive-only reached AUC–ROC = 0.78 (0.77–0.81) under Robust Scaling.
- Intermediate fusion was heterogeneous: Tabular Crossmodal Multi-Head Attention was stable across scaling (AUC–ROC = 0.92, 0.94, and 0.92 for Standard, Min–Max, and Robust), Edge Correlation GNN was strong under Standard Scaling (0.92 (0.90–0.94)) but declined under Min–Max/Robust, and Attention-Weighted GNN underperformed across pipelines (AUC–ROC ≈ 0.50). Late fusion (Tabular Decision) was stable but not superior (AUC–ROC 0.91–0.93).
- Standard Scaling was generally the most effective preprocessing strategy, particularly for early fusion and graph-based models; Robust Scaling benefited cognitive features and selected intermediate models but was suboptimal for high-performing early fusion.
Methods
- Data: WU-Minn HCP Young Adult dataset; after exclusions, final cohort of 747 healthy participants (410 females, 337 males), aged 22–36. Features were 36 cognitive/behavioral measures and 168 FreeSurfer brain morphology metrics, giving 204 features; the task was binary sex classification.
- Pipeline: three scaling methods (Standard, Min–Max, Robust) were compared across unimodal baselines and multimodal fusion paradigms, including early/operation-based fusion (concat tabular data, concat feature maps, channel-wise multi net), attention-based fusion, subspace-based fusion (MCVAE), graph-based fusion (Edge Correlation GNN, Attention-Weighted GNN), and late fusion (Tabular Decision), implemented in Fusilli with PyTorch/PyTorch Lightning.
- Evaluation: 3-fold cross-validation; primary metrics were AUC–ROC and accuracy, with balanced accuracy, F1-score, Cohen’s kappa, precision, recall, and average precision also reported; 95% CIs were computed via bootstrap resampling, and overfitting was monitored with learning curves and early stopping where supported.
Limitations
- The paper states the analysis was restricted to two modalities (cognitive measures and imaging-derived brain morphology) and a single healthy cohort with a narrow age range (22–36), limiting generalizability to clinical populations or broader lifespan contexts.
- The paper states that complete data availability and synchronized modalities may favor early joint feature learning; real-world missing, heterogeneous, or asynchronously acquired data may favor late fusion. The cohort was also restricted to participants who self-identified as White or Black, which may introduce selection bias and limit generalizability.
- The paper states that interpretability remains a critical requirement and was not the focus; future work should include explainable AI, additional modalities, disease-specific cohorts, larger multicenter datasets, and fully nested validation. The text also makes evident that the single-dataset, 3-fold cross-validation design may limit robustness, and no explicit feature selection or dimensionality reduction was applied by design.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
8 Valid · 0 Not valid · 7 Not checked · 78 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Sex-related differences in brain structure and function have been consistently reported across large-scale neuroimaging studies, involving both global and regional measures such as total brain volume, cortical thickness, white matter microstructure, and functional connectivity [29,30,31,32].”
§ Benchmarking Multimodal Deep Fusion Strategies for Heterogeneous Neuroimaging and Cognitive Data Using a Controlled Sex Classification Task > 1. Introduction
What the sources say
- Partly supportedcites [29] Duerden, 2020, Sex-Based Differences in Cortical and Subcortical Development in 436 Individuals Aged 4–54 Years, Cereb. Cortex sourceExperimental — accuracy not yet measured
The Introduction (page unavailable) reports sex-related differences in total cerebral volume and white-matter volume, and the Conclusions (page unavailable) report differences in regional cortical thickness and surface area. However, this structural MRI study does not report sex differences in white-matter microstructure or functional connectivity. Its Abstract also describes prior cortical-thickness findings as inconsistent, rather than consistently reported. (LLM, conf=0.89)
- Not checked — no full text of the sourcecites [30] Camastra, C., and Sarica, A. (2025). Brain Morphometry Differences Across Sexes Revealed Through Explainable Artificial Intelligence: A Human Connectome Project Young Adult Study. Machine Learning, Optimization, and Data Science, Springer. source
- Supportedcites [31] Ritchie, 2018, Sex Differences in the Adult Human Brain: Evidence from 5216 UK Biobank Participants, Cereb. Cortex source
The Discussion reports sex differences in total brain volume, cortical thickness, white matter microstructure, and functional connectivity in a UK Biobank sample of over 5,000 adults. The Introduction also discusses findings from earlier studies, supporting the sentence’s characterization of these differences as reported across studies. (LLM, conf=0.88)
- Not checked — no full text of the sourcecites [32] Gur, 1999, Sex Differences in Brain Gray and White Matter in Healthy Young Adults: Correlations with Cognitive Performance, J. Neurosci. source
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Late fusion approaches, which combine modality-specific predictions at the decision level, demonstrated stable and competitive performance across all evaluated scaling strategies.”
§ Benchmarking Multimodal Deep Fusion Strategies for Heterogeneous Neuroimaging and Cognitive Data Using a Controlled Sex Classification Task > 3. Results > 3.4. Late Fusion Models
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Graph-based intermediate fusion models showed mixed behavior.”
§ Benchmarking Multimodal Deep Fusion Strategies for Heterogeneous Neuroimaging and Cognitive Data Using a Controlled Sex Classification Task > 3. Results > 3.3. Intermediate Fusion Models
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Within this setting, early fusion strategies based on direct feature concatenation achieved equal or superior performance compared to more complex architectures.”
§ Benchmarking Multimodal Deep Fusion Strategies for Heterogeneous Neuroimaging and Cognitive Data Using a Controlled Sex Classification Task > 4. Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Intermediate fusion models, which integrate modalities after learning modality-specific representations through shared latent spaces or attention mechanisms, exhibited more heterogeneous performance patterns compared to early fusion approaches.”
§ Benchmarking Multimodal Deep Fusion Strategies for Heterogeneous Neuroimaging and Cognitive Data Using a Controlled Sex Classification Task > 3. Results > 3.3. Intermediate Fusion Models
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
42 more claims
Create a free account to see all 93 claims and every source verdict.
- Probably not a substantial claim. 17
- It may state two separate claims as one. 39
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Benchmarking Multimodal Deep Fusion Strategies for Heterogeneous Neuroimaging and Cognitive Data Using a Controlled Sex Classification Task > 2. Materials and Methods > 2.2. Data Fusion Workflow
“Inspired by the population-graph learning approach proposed by Bintsi et al. [27], this model employs a static graph representation.”
§ Benchmarking Multimodal Deep Fusion Strategies for Heterogeneous Neuroimaging and Cognitive Data Using a Controlled Sex Classification Task > References
“27. Calhoun V.D. Sui J. Multimodal Fusion of Brain Imaging Data: A Key to Finding the Missing Link(s) in Complex Mental Illness”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Benchmarking Multimodal Deep Fusion Strategies for Heterogeneous Neuroimaging and Cognitive Data Using a Controlled Sex Classification Task > 1. Introduction
“a unified and reproducible pipeline implemented in the Fusilli Python library (version 3.9).”
§ Benchmarking Multimodal Deep Fusion Strategies for Heterogeneous Neuroimaging and Cognitive Data Using a Controlled Sex Classification Task > 2. Materials and Methods > 2.2. Data Fusion Workflow
“four deep learning-based fusion strategies implemented in the Fusilli v1.2.3 Python library [39].”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Benchmarking Multimodal Deep Fusion Strategies for Heterogeneous Neuroimaging and Cognitive Data Using a Controlled Sex Classification Task > 2. Materials and Methods > 2.2. Data Fusion Workflow
“This model adopts the Multi-channel Variational Autoencoder (MCVAE) framework introduced by Antelmi et al. [26].”
§ Benchmarking Multimodal Deep Fusion Strategies for Heterogeneous Neuroimaging and Cognitive Data Using a Controlled Sex Classification Task > References
“26. Kipf T.N. Welling M. Semi-Supervised Classification with Graph Convolutional Networks”
Citations
Cites
Semi-supervised classification with graph convolutional networks
Kipf, T. N., & Welling, M. (2016) · arXiv · doi:10.48550/arxiv.1609.02907 · Cited by 24,853
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018
The Human Connectome Project: A data acquisition perspective
Van Essen, D. C., Ugurbil, K., Auerbach, E., Barch, D., Behrens, T. E. J., Bucholz, R., Chang, A., Chen, L., Corbetta, M., Curtiss, S. W., et al. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.02.018 · Cited by 2,680
Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects
Lahat, D., Adali, T., & Jutten, C. (2015) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2460697 · Cited by 1,296 · cited 1×
“These modalities provide complementary perspectives on brain structure and function but differ substantially in dimensionality, scale, noise characteristics, and statistical distribution [2,3].”
KNIME - the Konstanz information miner: version 2.0 and beyond
Berthold, M. R., Cebron, N., Dill, F., Gabriel, T. R., Kötter, T., Meinl, T., Ohl, P., Thiel, K., & Wiswedel, B. (2009) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1656274.1656280 · Cited by 1,189
Deep multimodal learning: a survey on recent advances and trends
Ramachandram, D., & Taylor, G. W. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2738401 · Cited by 1,139 · cited 2×
“…deep learning–based multimodal fusion have introduced increasingly expressive architectures—such as attention-based models, variational subspace learning, and graph neural networks (GNN)—that are specifically designed to address modality heterogeneity and complex cross-modal interactions [6,10,11].”
Hierarchical Feature Representation and Multimodal Fusion with Deep Learning for AD/MCI Diagnosis
Suk, H.-I., Lee, S.-W., & Shen, D. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2014.06.077 · Cited by 968
Huang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I., & Lungren, M. P. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00341-z · Cited by 934 · cited 3×
“Recent studies have proposed increasingly sophisticated fusion architectures, including attention-based models, variational autoencoders, and graph neural networks, often reporting improved predictive performance in complex biomedical and neuroimaging tasks [6,9,45–47].”
Gur, R. C., Turetsky, B. I., Matsui, M., Yan, M., Bilker, W., Hughett, P., & Gur, R. E. (1999) · The Journal of Neuroscience · doi:10.1523/jneurosci.19-10-04065.1999 · Cited by 909 · cited 1×
“Sex-related differences in brain structure and function have been consistently reported across large-scale neuroimaging studies, involving both global and regional measures such as total brain volume, cortical thickness, white matter microstructure, and functional connectivity [29–32].”
Research works printed but not linked: 3
- Ngiam, J., Khosla, A., Kim, M., Nam, J., Lee, H., and Ng, A. (2011). Multimodal Deep Learning, Springer.
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., and Polosukhin, I. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems 30 (NIPS 2017), NeurIPS.
- Antelmi, L., Ayache, N., Robert, P., and Lorenzi, M. (2019, January 24). Sparse Multi-Channel Variational Autoencoder for the Joint Analysis of Heterogeneous Data. Proceedings of the 36th International Conference on Machine Learning, Long Beach, CA, USA.
Metadata
- Authors
- Camastra, C., Pelagi, A., Quattrone, A., & Sarica, A.
- Affiliations
- Centre for Addiction and Mental Health (CA)
- Magna Graecia University (IT)
- Corresponding author
- Sarica, Alessia
- Venue
- Brain Sciences
- Publisher
- MDPI AG
- Published
- 2026-04-10
- Volume
- 16
- Issue
- 4
- Pages
- 405
- ISSN
- 2076-3425 (ISSN-L)
- PMC
- PMC13114584
- OpenAlex
- W7153217618
- Cited by
- 1
- References
- 38
- Citations by year
- 2026: 1
- FWCI
- 4.11
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Brain Sciences (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Institutes of Health (1U54‐MH‐091657)
- McDonnell Center for Systems Neuroscience (1U54MH091657)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Functional Brain Connectivity Studies
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Keywords
- multimodal fusion, fusion strategies, feature scaling, sex classification, feature-level fusion, attention fusion, Human Connectome Project, benchmarking, reproducibility
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Benchmarking Multimodal Deep Fusion Strategies for Heterogeneous Neuroimaging and Cognitive Data Using a Controlled Sex Classification Task. https://app.cochat.ai/papers/10.3390/brainsci16040405
- The paper
- Camastra, C., Pelagi, A., Quattrone, A., & Sarica, A. (2026). Benchmarking Multimodal Deep Fusion Strategies for Heterogeneous Neuroimaging and Cognitive Data Using a Controlled Sex Classification Task. Brain Sciences. https://doi.org/10.3390/brainsci16040405
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.