CoChat
Sign Up
PreprintCited by 127Privacy-Preserving Technologies in DataDomain Adaptation and Few-Shot LearningMultimodal Machine Learning Applications

Multimodal Federated Learning: A Survey

Open access
Che, Liwei, Wang, Jiaqi, Zhou, Yao, Ma, Fenglong·2023·Sensors
DOI 10.20944/preprints202307.1420.v1PMC PMC10422520

Preprint — published as 10.3390/s23156986

Open this paper in CoChat — free accountOpen PDF

Abstract

Federated learning (FL) has become a burgeoning and attractive research area, which provides a collaborative training scheme for distributed data sources with privacy concerns. Most existing FL studies focus on taking unimodal data, such as images and text, as the model input and resolving the heterogeneity challenge, i.e., the non-identically distributed (non-IID) challenge caused by data distribution imbalance related to data labels and data amount. In real-world applications, data are usually described by multiple modalities. However, to the best of our knowledge, only a handful of studies have been proposed to improve the system performance by utilizing multimodal data. In this survey paper, we identify the significance of this emerging research topic – multimodal federated learning (MFL) and perform a literature review on the state-of-art MFL methods. Furthermore, we categorize multi-modal federated learning into congruent and incongruent multimodal federated learning based on whether all clients possess the same modal combinations. We investigate the feasible application tasks and related benchmarks for MFL. Lastly, we summarize the promising directions and fundamental challenges in this field for future research.

AI snapshot

Written from the paper’s full text.

TL;DR

This survey defines multimodal federated learning (MFL) as federated systems containing at least two data modalities and reviews state-of-the-art methods, tasks, and benchmarks. It proposes a taxonomy of congruent and incongruent MFL, with incongruent MFL split into vertical, transfer, and hybrid settings, and outlines challenges including modality heterogeneity, missing modalities, data complexity, large-scale pre-trained models, and weak supervision.

Key findings

  • MFL is formally defined for \(K \ge 2\) clients and \(M \ge 2\) modality categories, where each client \(k\) has \(M_k \in [1, M]\); clients are unimodal if \(M_k = 1\) and multimodal if \(M_k > 1\). The taxonomy divides MFL into congruent MFL (mainly horizontal) and incongruent MFL (vertical, multimodal federated transfer, and hybrid).
  • In hybrid MFL, the theoretical number of client types is \(2^M - 1\); a bi-modal system can have three kinds of clients, and a tri-modal system can have seven.
  • The survey reviews representative MFL tasks—vision-language interaction, human activity recognition, emotion recognition, and healthcare—and lists benchmarks such as CUB-200-2011 (11,788 images, 200 subcategories, 10 text descriptions per image), MS COCO (328K images), Flickr30k (31,000 images with 5 reference sentences each; 158,915 captions for 31,783 images), NTU RGB+D 120 (106 subjects, over 114 thousand video samples, 8 million frames, 120 action classes), and FedMultimodal as the first federated learning benchmark for multimodal learning.
  • Key challenges identified are modality heterogeneity at client and system levels, modality missing, data complexity in healthcare, integrating large-scale pre-trained models, and weak-supervised learning. The paper states that existing non-IID and personalized methods are questionable in multimodal federated settings and that a one-fit-all global training strategy may be sub-optimal.

Methods

  • This is a survey/literature review, not an empirical study. It reviews existing MFL methods, extends federated learning categorization to multimodal scenarios, and discusses tasks, benchmarks, challenges, and future directions.
  • It provides formal preliminaries for MFL, including definitions of local multimodal datasets, modality combinations, local model updates via stochastic gradient descent, modality-weighted local loss, and global optimization targets.
  • It categorizes MFL into four settings based on input modalities—horizontal, vertical, multimodal federated transfer, and hybrid—and summarizes them in Table 1, then surveys application tasks and benchmark datasets.

Limitations

  • The paper states that benchmarking of MFL frameworks specifically designed for testing and executing MFL tasks is still in its exploratory stage, and that the discussed challenges are non-exclusive and root in the data modality distribution in the federated learning system. The text also identifies itself as a “Not peer-reviewed version.”
  • Evident from the text: the survey reports no empirical evaluation, no systematic search protocol, and no quantitative comparison of MFL methods; the benchmark section is descriptive rather than experimental.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.20944/preprints202307.1420.v1

Cites

Statistics

Nothing to recompute.

Claims the paper makes

0 of 95 cited sources checked · 0 supported · 0 flagged

  • Premisecitation

    In the healthcare domain, congruent MFL has shown great application value by providing diagnosis assistance with distributed digital health data.

    “In the healthcare domain [13][14][15], the congruent MFL has shown great application value by providing diagnosis assistance with distributed digital health data.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [13]

      Not checked in the free run.

    • cites [14]

      Not checked in the free run.

    • cites [15]

      Not checked in the free run.

  • Interpretationcitation

    Given M modalities in a federated system, the theoretical client types are 2^M - 1, including both unimodal and multimodal clients.

    “Given M modalities in a federated system, the theoretical client types are 2 M -1, including both unimodal and multimodal clients.”
    Quote in the paperNot a substantial claim
  • Premisecitation

    Federated learning provides a paradigm to promote knowledge discovery of multimodal data under distributed scenarios.

    “federated learning [2] provides a paradigm to promote knowledge discovery of multimodal data under distributed scenarios.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [2]

      Not checked in the free run.

  • Premisecitation

    The majority of existing MFL work focuses on the congruent federated setting, where all clients hold the same input modality categories and feature space but differ on the sample space.

    “The majority of existing MFL work [9][10][11][12] also focuses on this federated setting, where all the clients hold the same input modality categories and feature space but differ on the sample space.”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [9]

      Not checked in the free run.

    • cites [10]

      Not checked in the free run.

    • cites [11]

      Not checked in the free run.

  • Premisecitation

    Statistical heterogeneity, also called the non-IID challenge, caused by skew of labels, features, and data quantity among clients, is one of the most critical challenges in federated learning.

    “Among those studies, statistical heterogeneity [3], a.k.a. the non-IID challenge, caused by the skew of label, features, and data quantity among clients, is one of the most critical challenges that attracts much attention [4][5][6][7][8].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [3]

      Not checked in the free run.

    • cites [4]

      Not checked in the free run.

    • cites [5]

      Not checked in the free run.

  • Classified as probably not a substantial claim. 87
  • May combine two claims. 46

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    A great amount of medical-related MFL work is emerging.

    “A great amount of medical-related MFL work is emerging.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In [10], the authors propose a multimodal federated learning framework for multimodal activity recognition with an early fusion approach via local co-attention.

    “In [10], the authors propose a multimodal federated learning framework for multimodal activity recognition with an early fusion approach via local co-attention.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [10]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation

    Federated systems trained with multimodal data are intuitively more powerful and insightful compared to unimodal ones.

    “The federated systems trained with multimodal data are intuitively more powerful and insightful compared to unimodal ones [1].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [1]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    They proposed FedMSplit for multi-task learning in the hybrid-MFL setting, with a graph-based attention mechanism to extract the client relationship for aggregation.

    “They proposed FedMSplit for multi-task learning in the hybrid-MFL setting, with a graph-based attention mechanism to extract the client relationship for aggregation.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [18]

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Multimodal federated learning introduces modality heterogeneity, leading to significant differences in model structures, local tasks, and parameter spaces among clients, thereby exposing substantial limitations of traditional unimodal algorithms.

    “However, multimodal federated learning further introduces modality heterogeneity, which leads to significant differences in model structures, local tasks, and parameter spaces among clients, thereby exposing substantial limitations of traditional uni-modal algorithms.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    If two clients hold the same or similar modality combinations, they will have a smaller semantic gap and task gap.

    “If two clients hold the same or similar modality combinations (e.g., both image and text data), they will have a smaller semantic gap and task gap.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    FedProx [20] modifies the local objective of each client by adding a proximal term.

    “FedProx [20] modifies the local objective of each client, adding a proximal term to it.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [20]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    In [19], the authors discover that non-IID data will cause client drift in the uploading process and affect system convergence rates.

    “In [19], the authors discover that the non-IID data will cause the problem of client drift in the uploading process and affect the system convergence rates.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [19]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The more congruent modality combinations the clients hold, the less heterogeneous the modality distribution of the system is.

    “In other words, the more congruent modality combinations the clients hold, the less heterogeneous the modality distribution of the system is.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Zhao et al. in [4] investigate the under-performance problem of FedAvg [2] under the non-IID setting and indicate that weight divergence causes the performance reduction issue.

    “From the convergence optimization aspect, Zhao et al. in [4] investigate the under-performance problem of FedAvg [2] under the non-IID setting.They define and indicate that weight divergence causes the performance reduction issue and give a further analysis of that.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [4]

      Not checked in the free run.

    • cites [2]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In [21], the authors propose SCAFFOLD to mitigate client drifts between local models and the global model.

    “In [21], the authors propose SCAFFOLD to mitigate the client drifts between the local models and the global model.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [21]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In [12], the authors provide a detailed analysis of the convergence problem of MFL with late fusion methods under the non-IID setting.

    “The authors in [12] provide a detailed analysis of the convergence problem of MFL with late fusion methods under the non-IID setting.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [12]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisereasoning

    Existing unimodal methods are mainly proposed from three aspects: federated convergence optimization, personalized federated learning, and federated multi-task learning.

    “To mitigate the optimization divergence caused by heterogeneity challenges and increase the system robustness under the non-IID setting, existing unimodal methods are mainly proposed from three aspects: federated convergence optimization, personalized federated learning, and federated multi-task learning.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    [18] states a significant challenge for hybrid-MFL, which is modality incongruity, where the unique modality combination among clients enlarges heterogeneity.

    “[18] states a significant challenge for hybrid-MFL, which is modality incongruity, where the unique modality combination among the clients enlarges the heterogeneity.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [18]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    In [17], the authors propose a fine-grained representation block named aimNet and evaluate their methods on different FL settings, including the transfer setting between two different vision-language tasks.

    “For instance, in [17], the authors propose a fine-grained representation block named aimNet.They evaluate their methods on different FL settings, including the transfer setting between two different vision-language tasks.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [17]

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Federated learning allows distributed clients to collaboratively train a better-performing global model without sharing their local data.

    “This paradigm allows the distributed clients collaboratively train a better-performed global model without sharing their local data.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [2]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    Hybrid-MFL is a more challenging setting, where the data relationships among clients cannot be appropriately described by any of the above three settings alone.

    “Hybrid-MFL is a more challenging setting, where the data relationships among the clients can not be appropriately described by any of the above three settings alone.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    In [16], the authors assume that each client only contains one specific modality and propose FDARN, a five-module framework, for cross-modal federated human activity recognition.

    “In [16], the authors assume that each client only contains one specific modality and, correspondingly, propose FDARN, a five-module framework, for cross-modal federated human activity recognition (CMF-HAR).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [16]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In [9], the authors use an ensemble of local and global models to reduce both data variance and device variance in the federated system.

    “In [9], the authors use an ensemble of local and global models to reduce both data variance and device variance in the federated system.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [9]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In [22], FedBVA decomposes the aggregation error into bias and variance for collaborative adversarial training in order to improve convergence and performance.

    “In [22], FedBVA decomposes the aggregation error into bias and variance for collaborative adversarial training in order to improve convergence and performance.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [22]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Marfoq et al. in [31] use a federated expectation-maximization algorithm to solve the multi-task problem with a mixture of unknown underlying distributions.

    “Marfoq et al. in [31] use federated expectation-maximization algorithm to solve the multi-task problem with a mixture of unknown underlying distributions.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [31]

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    In the multimodal setting, federated clients could have different local model structures due to modality heterogeneity, which makes the general solution more unfeasible.

    “Especially in the multimodal setting, federated clients could have different local model structures due to the modality heterogeneity, which makes the general solution more unfeasible.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The authors also validate the effectiveness of their method in the modality missing challenge, where some clients only have part of shared data modalities in the horizontal federation.

    “The authors also validate the effectiveness of their method in the modality missing challenge, where some clients only have part of shared data modalities in the horizontal federation.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [33]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Ruan and Joe-Wong in [25] propose FedSoft, which reduces clients' workload using proximal updates and brings both personalized local models and global cluster models.

    “Ruan and Joe-Wong in [25] propose FedSoft, which reduces the clients' workload using proximal updates and brings both personalized local models and global cluster models.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [25]

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    The introduction of modality heterogeneity among clients brings unique challenges and makes the existing challenge even more demanding.

    “Instead, the introduction of modality heterogeneity among the clients brings unique challenges and makes the existing challenge even more demanding.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given

    Multimodal federated learning faces limitations such as communication costs, data heterogeneity, and hardware disparities compared to centralized multimodal learning.

    “However, it also faces limitations such as communication costs, data heterogeneity, and hardware disparities compared to centralized multimodal learning.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationcitation

    It is not illustrative enough to describe the data distribution relationship in such a way, especially when clients contain different combinations of data modalities, i.e., the modality incongruity challenge proposed by [18].

    “It is not illustrative enough to describe the data distribution relationship in such a way, especially when the clients contain different combinations of data modalities, i.e., modality incongruity challenge proposed by [18].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [18]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given

    Visual and language data are widely present in data centers and edge devices, making visual-language interaction an important task in MFL.

    “Visual and language data are widely present in data centers and edge devices, making visuallanguage interaction an important task in MFL.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    In addition to the unique challenges of modal heterogeneity, the original multimodal learning tasks become more challenging when performed within a federated learning framework.

    “Therefore, in addition to the unique challenges of modal heterogeneity, the original multimodal learning tasks become more challenging when performed within a federated learning framework.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    Multiple modalities can make different contributions to the final loss affected by the problem context, data quality, and downstream tasks.

    “Multiple modalities can make different contributions to the final loss affected by the problem context, data quality, and downstream tasks.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    Existing methods have shown great achievement in alleviating the statistical heterogeneity challenge via global optimization or learning personalized local models.

    “The existing methods have shown great achievement in alleviating the statistical heterogeneity challenge via global optimization or learning personalized local models.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given

    The previously proposed three-way division can clearly identify different categories of federated learning settings in unimodal scenarios.

    “This previously proposed three-way division can clearly identify different categories of federated learning settings in unimodal scenarios.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Each client holds a single modality data set that may differ from group to group in their experiments.

    “Notably, each client holds a single modality data set that may differ from group to group in their experiments.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [16]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Interpretationcitation

    This indicates the recent research trend to design more general-purpose multimodal models and reveals the importance of exploring hybrid MFL, the most challenging and complex multimodal federated scenario.

    “This indicates the recent research trend to design more general-purpose multimodal models and reveals the importance of exploring hybrid MFL, the most challenging and complex multimodal federated scenario.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    With five adversarial training modules, the proposed method captures both modality-agnostic features and modality-specific discriminative characteristics of each client to achieve better performance than existing personalized federated learning methods.

    “With five adversarial training modules, the proposed method captures both the modality-agnostic features and modality-specific discriminative characteristics of each client to achieve a better performance than existing personalized federated learning methods.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [16]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    They validate their methods in horizontal, vertical, and transfer multimodal federated learning settings to show their superiority.

    “They validate their methods in horizontal, vertical, and transfer multimodal federated learning settings to show their superiority.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [17]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    These task gaps and semantic gaps can negatively impact global aggregation on the server side, posing challenges for achieving convergent global optimization.

    “These task gaps and semantic gaps can negatively impact global aggregation on the server side, posing challenges for achieving convergent global optimization.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    In [32], federated learning is categorized into horizontal federated learning, vertical federated learning, and federated transfer learning based on data distribution characteristics.

    “In [32], federated learning is categorized into horizontal federated learning, vertical federated learning, and federated transfer learning based on the data distribution characteristics.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [32]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In [28], the authors reassemble models and select the most fitted personalized models for clients by calculating similarity.

    “In [28], the authors reassemble models and select the most fitted personalized models for clients by calculating the similarity.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [28]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Federated multi-task learning [29] aims to find personalized models for each client by utilizing similarity among them.

    “Instead of seeking a general solution, federated multi-task learning [29] aims to find personalized models for each client by utilizing the similarity among them.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [29]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In [27], the authors add Moreau envelopes as a regularization term in local loss functions to help achieve personalized model optimization.

    “In [27], the authors add Moreau envelopes as a regularization term in the local loss functions to help achieve personalized model optimization.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [27]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    The relationships among clients in the above-mentioned case could be decomposed into modality-wise levels.

    “In the above-mentioned case, the relationships among clients could be decomposed into modality-wise levels.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    MOCHA [29] is the first proposed federated multi-task learning framework for convex models.

    “MOCHA [29] is the first proposed federated multi-task learning framework for convex models.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [29]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In [16], the authors propose the feature-disentangled activity recognition network (FDARN) for the cross-modal federated human activity recognition task.

    “In [16], the authors propose the feature-disentangled activity recognition network (FDARN) for the cross-modal federated human activity recognition task.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [16]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given

    Multimodal federated learning offers many advantages, such as privacy preservation and addressing the data silo problem.

    “Multimodal federated learning (MFL) offers many advantages, such as privacy preservation and addressing the data silo problem.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Liu et al. propose aimNet to generate fine-grained image representations and improve performance on various vision-and-language grounding problems under federated settings.

    “Liu et al. propose aimNet to generate fine-grained image representations and improve the performance on various vision-and-language grounding problems under federated settings.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [17]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Heterogeneity in data, particularly in terms of label and modality, often leads to differences in model architectures and tasks among clients.

    “On the other hand, heterogeneity in data, particularly in terms of label and modality, often leads to differences in model architectures and tasks among clients.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In [23], the authors introduce reinforcement learning into the federated global update stage, where the server dynamically selects a subset of clients with the highest rewards to mitigate heterogeneity.

    “In [23], the authors introduce reinforcement learning into the federated global update stage, where the server will dynamically select a subset of clients with the highest rewards to mitigate the heterogeneity challenge.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [23]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Corinzia et al. in [30] propose VIRTUAL, which uses approximated variational inference and simulates the federated system with a star-shaped Bayesian network.

    “Corinzia et al. in [30] propose VIRTUAL, which uses approximated variational inference and simulates the federated system with a star-shaped Bayesian network.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [30]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Personalized federated learning [24] is a research topic proposed to handle statistical heterogeneity from another aspect, where many existing methods aim to provide each client with personalized adaptation instead of a one-fit-all global solution.

    “Personalized federated learning [24] is a research topic proposed to handle the statistical heterogeneity challenge (i.e., the non-IID setting) from another aspect, where many existing methods aim to provide each client with personalized adaptation instead of a one-fit-all global solution.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [24]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisereasoning

    The MFL framework offers the capability of efficient collaborative training while ensuring privacy protection.

    “In this context, the MFL framework offers the capability of efficient collaborative training while ensuring privacy protection.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    The modality distribution or data partition way could be more diverse according to hardware deployment and environmental factors.

    “However, the modality distribution or data partition way could be more diverse according to the hardware deployment and environmental factors.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    The wireless distributed sensor system, such as the IoT system, is a significant application scenario for MFL, where multiple sensors provide consistent observations of the same object or event.

    “The wireless distributed sensor system, such as the IoT system, is a signification application scenario for MFL, where multiple sensors provide consistent observations of the same object or event.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The proposed hierarchical aggregation method aggregates the encoders based on the modality type held by the clients and utilizes an attention-based method to align the decoder weights regardless of the data modality.

    “The proposed hierarchical aggregation method aggregates the encoders based on the modality type held by the clients and utilizes an attention-based method to align the decoder weights regardless of the data modality.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [37]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Oxford 102 Flower collects 102 categories of flowers that commonly occur in the United Kingdom, each category contains 40 to 258 images, and there are 10 text descriptions for each image.

    “Oxford 102 Flower [44] is a fine-grained classification dataset, which collects 102 categories of flowers that commonly occur in the United Kingdom.Each category contains 40 to 258 images.There are 10 text descriptions for each image.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [44]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisereasoning

    Human activity recognition is one of the most representative tasks in this setting, due to the privacy preservation requirement.

    “Human activity recognition (HAR) is one of the most representative tasks in this setting, due to the privacy preservation requirement.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    CreamFL [38] utilizes contrastive learning for ensembling uploaded heterogeneous local models based on their output representations.

    “CreamFL [38] utilizes contrastive learning for ensembling the uploaded heterogeneous local models based on their output representations.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [38]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    This presents a series of crucial challenges within the realm of multimodal federated learning, encompassing tasks such as AI-assisted diagnosis, medical image analysis, and laboratory report generation.

    “This presents a series of crucial challenges within the realm of multimodal federated learning, encompassing tasks such as AI-assisted diagnosis, medical image analysis, and laboratory report generation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    CUB-200-2011 contains 11,788 images of 200 subcategories belonging to birds.

    “CUB-200-2011 [42] is one of the most widely used fine-grained categorization datasets.It contains 11,788 images of 200 subcategories belonging to birds.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [42]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Flickr30k comprises 31,000 images sourced from Flickr, accompanied by 5 reference sentences per image generated by human annotators.

    “The Flickr30k dataset [47] comprises 31,000 images sourced from Flickr, accompanied by 5 reference sentences per image generated by human annotators.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [47]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    UPMC Food-101 contains both images and paired captions of 101 food categories, with 750 training and 250 testing images per category, totaling 101,000 images each paired with one caption.

    “The Food-101 [45] is a noisy multimodal classification dataset that contains both images and paired captions of 101 food categories.Each category has 750 training and 250 testing images.There are a total of 101,000 images, each paired with one caption.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [45]

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Agbley et al. in [14] apply federated learning on the prediction of melanoma and obtained performance that is on-par with the centralized training result.

    “Agbley et al. in [14] apply federated learning on the prediction of melanoma and obtained performance that is on-par with the centralized training result.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [14]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Multimodal data generated during the use of mobile phones often provide valuable insights into identifying users who may have underlying mental health issues.

    “The multimodal data generated during the use of mobile phones often provide valuable insights into identifying users who may have underlying mental health issues.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning

    As a result, healthcare institutions have formed isolated data islands, impeding direct collaboration in cooperative training and data sharing through open databases.

    “As a result, these healthcare institutions have formed isolated data islands, impeding direct collaboration in terms of cooperative training and data sharing through open databases.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    MS COCO comprises a total of 328K images and provides detailed annotations for object detection, captioning, keypoint detection, stuff image segmentation, panoptic segmentation, and dense pose estimation.

    “The MS COCO dataset [46] is a comprehensive dataset used for various tasks such as object detection, segmentation, key-point detection, captioning, stuff image segmentation, panoptic segmentation, and dense pose estimation.It comprises a total of 328K images.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [46]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    EPIC-KITCHENS-100 features multi-faceted audio-visual recordings of individuals' daily kitchen activities captured in their homes using head-mounted cameras, comprising 45 kitchens across four cities, 100 hours of Full HD footage and 20 million frames.

    “EPIC-KITCHENS-100 [49] is a large-scale dataset focusing on first-person (egocentric) vision.It features multi-faceted audio-visual recordings of individuals' daily kitchen activities captured in their homes using head-mounted cameras.The dataset, comprising 45 kitchens across four cities, offers diverse environmental contexts.With 100 hours of Full HD footage and 20 million frames”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [49]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Each cluster contains healthcare entities that contain the same modality, such as X-ray and Ultrasound.

    “Each cluster contains the healthcare entities that contain the same modality, such as X-ray and Ultrasound.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [13]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    NTU RGB+D 120 comprises over 114 thousand video samples and 8 million frames from 106 unique subjects, encompassing 120 distinct action classes.

    “The NTU RGB+D 120 dataset [48] is a comprehensive collection specifically designed for RGB+D human action recognition.It comprises a vast amount of data sourced from 106 unique subjects, encompassing over 114 thousand video samples and 8 million frames.This dataset encompasses a wide range of 120 distinct action classes”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [48]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    The benchmarking of MFL frameworks specifically designed for testing and executing MFL tasks is still in its exploratory stage.

    “However, the benchmarking of MFL frameworks specifically designed for testing and executing MFL tasks is still in its exploratory stage.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Multimedia data associated with user emotions is highly privacy-sensitive, including chat records and personal photos.

    “However, multimedia data associated with user emotions is highly privacy-sensitive, including chat records and personal photos.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The local network is divided into five modules for either modality-wise aggregation for clients with the same modality or general aggregation for all clients.

    “The local network is divided into five modules for either modality-wise aggregation for clients with the same modality or general aggregation for all clients.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [33]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    FedMSplit [18] utilizes a dynamic and multi-view graph structure to flexibly capture the correlations among client models in a multimodal setting.

    “The FedMSplit approach [18] utilizes a dynamic and multi-view graph structure to flexibly capture the correlations among client models in a multimodal setting.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [18]

      Not checked in the free run.

  • May combine two claims.

    Interpretationnone given

    In the multimodal federated learning setting, the modality heterogeneity challenge exists at both the client level and the system level.

    “In the MFL setting, the modality heterogeneity challenge exists at both the client level and the system level.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    In [37], each client in the system contains only one modality and the uni-modal encoders are trained on the local side.

    “In [37], each client in the system contains only one modality and the uni-modal encoders are trained on the local side.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [37]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Reed et al. expand the CUB-200-2011 dataset by providing ten fine-grained text description sentences for each image.

    “Reed et al. expand the dataset by providing ten fine-grained text description sentences for each image [43].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [43]

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    MMFed [10] equally divided the multimodal data for each client, and the local co-attention mechanism then performs multimodal fusion.

    “The former one is represented by MMFed [10], which equally divided the multimodal data for each client.The local co-attention mechanism then performs multimodal fusion.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [10]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Liang et al. in [41] propose a decentralized privacy-preserving representation learning method that uses multimodal behavior markers to predict users' daily moods and identify early risks of suicide.

    “Liang et al. in [41] propose a decentralized privacy-preserving representation learning method that uses multimodal behavior markers to predict users' daily moods and identify the early risks of suicide.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [41]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given

    These multimodal data are typically tightly linked to patient identifiers and require stringent privacy protection measures.

    “These multimodal data are typically tightly linked to patient identifiers and require stringent privacy protection measures.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    FedNorm [15] performs modality-based normalization techniques to enhance liver segmentation and trains over uni-modal clients holding CT and MRI data separately.

    “FedNorm [15] performs modality-based normalization techniques to enhance liver segmentation and trains over uni-modal clients holding CT and MRI data separately.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [15]

      Not checked in the free run.

  • May combine two claims.

    Premisenone given

    Some works in healthcare have explored multimodal federated learning, often assuming that all institutions have the same set of modalities (horizontal MFL) or that each institution possesses only a single modality (vertical MFL).

    “Some works in the field of healthcare have explored multimodal federated learning, often assuming that all institutions have the same set of modalities, referred to as horizontal MFL, or that each institution possesses only a single modality, known as vertical MFL.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    The data partition method for the HAR task in existing MFL works has two types: client-as-device and client-as-sensor.

    “The data partition method for the HAR task in existing MFL works has two types, client-as-device and client-as-sensor.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Zhao et al. conduct their experiment by giving each client only a single type of modality [33].

    “Zhao et al. conduct their experiment by giving each client only a single type of modality [33].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [33]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Numerous healthcare centers and hospitals have accumulated vast amounts of multimodal data during patient consultations and treatments, including X-ray images, CT scans, physician diagnoses, and physiological measurements of patients.

    “Numerous healthcare centers and hospitals have accumulated vast amounts of multimodal data during patient consultations and treatments, including X-ray images, CT scans, physician diagnoses, and physiological measurements of patients.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Qayyum et al. utilize cluster federated learning for automatic diagnosis of COVID-19 [13].

    “Qayyum et al. utilize the cluster federated learning for automatic diagnosis of COVID-19 [13].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [13]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    FedCMR [11] explores the federated cross-modal retrieval task and mitigates the representation space gap via weighted aggregation based on local data amount and category number.

    “FedCMR [11] explores the federated cross-modal retrieval task and mitigates the representation space gap via weighted aggregation based on local data amount and category number.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [11]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    FedNorm explores medical image segmentation in the FL setting.

    “FedNorm [15] explores the medical image segmentation in the FL setting.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [15]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Cobbinah et al. provide an FL-based method for the prediction of the phase of Alzheimer's disease utilizing MRI data from multi-center.

    “Cobbinah et al. in [69] provide an FL-based method for the prediction of the phase of Alzheimer's disease utilizing the MRI data from multi-center.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [69]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisenone given

    FedPCL uses class prototypes and contrastive learning to share class-relevant knowledge among clients, demonstrating improved personalization and knowledge integration capabilities with a pre-trained backbone model.

    “Tan et al. proposed Federated Prototype-wise Contrastive Learning (FedPCL) algorithm uses class prototypes and contrastive learning to share class-relevant knowledge among clients, demonstrating improved personalization and knowledge integration capabilities with a pre-trained backbone model.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation

    The one-fit-all global training strategy for MFL might not work because optimizing all clients jointly can result in sub-optimal performance.

    “the one-fit-all global training strategy for MFL might not work since optimizing all the clients jointly can result in sub-optimal [63].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [63]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The head and mapping layers of BERT are distributed to clients for local training and then aggregated on the server side.

    “The head and mapping layers of the BERT are distributed to clients for local training and then aggregated on the server side.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [73]

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    Due to modality heterogeneity and different local model tasks, the common subspaces differ from each other, resulting in task gaps that are difficult to bridge.

    “Due to the modality heterogeneity and the different local model tasks, those common subspaces differs from each other, resulting the task gaps that are difficult to bridge.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning

    MFL enables exploration of information within complex healthcare data, facilitating collaborative training across healthcare and medical data silos and delivering improved AI-assisted diagnosis, medical image analysis, report generation, and other related services.

    “MFL enables the exploration of information within such complex data, facilitating collaborative training across healthcare and medical data silos and thereby delivering improved AI-assisted diagnosis, medical image analysis, report generation, and other related services.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Healthcare data such as electronic health records, X-ray, and CT scan images are stored and managed in a privacy-sensitive manner.
    • Rests on an assumption: Healthcare data are more complex and diverse in both format and granularity than data in the IoT or multimedia domain.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    The heterogeneity of multimodal data will further challenge the robustness of the system with limited and noisy supervision signals.

    “However, the heterogeneity of multimodal data will further challenge the robustness of the system with limited and noisy supervision signals.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: In realistic application scenarios, collected multimodal data in data silos usually contain limited supervision signals such as labels or matching relationships among the modalities.

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    In centralized multimodal learning, representation learning usually transforms the different modalities into a common representation space via embedding operation.

    “In centralized multimodal learning, representation learning usually transforms the different modalities into a common representation space via embedding operation.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    MFL divides the common space of the centralized scenario into N common subspaces, making the unifying of the embedding operation among all clients difficult.

    “Comparatively, MFL divides the common space of the centralized scenario into N common subspaces, making the unifying of the embedding operation among all the clients difficult.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Unimodal clients and multimodal clients could hold totally different parameter spaces and work on different feature spaces.

    “For example, unimodal clients and multimodal clients could hold totally different parameter spaces and work on different feature spaces.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    The training data for pre-training large models are often massive and diverse, making it challenging for clients in a federated learning system to collect and store such data.

    “This is due to the fact that the training data for pre-training large models are often massive and diverse, making it challenging for clients in a federated learning system to collect and store such data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    One of the most popular solutions for multimodal representation learning is to map the different modalities of data into a mutual latent space.

    “One of the most popular solutions for multimodal representation learning is to map the different modalities of data into a mutual latent space.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [61]

      Not checked in the free run.

    • cites [62]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    There still exist two main challenges known to block the universal deployment of large-scale pre-trained models in federated learning systems.

    “However, there still exist two main challenges known to block the universal deployment of those large-scale pre-trained models in federated learning systems.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The cost of building and training large-scale pre-trained models is extremely expensive and not affordable for most computing devices and data centers.
    • Rests on an assumption: Massive training data is required to gather for effective training of large foundation models.
    • Rests on an assumption: Clients in federated learning systems usually have limited hardware resources and communication bandwidth.

  • May combine two claims.

    Interpretationcitation

    The heterogeneity of data is further amplified among healthcare institutions due to differences in medical equipment, diagnostic methods, and data management practices, making federated collaboration more difficult.

    “The heterogeneity of data is further amplified among healthcare institutions due to differences in medical equipment, diagnostic methods, and data management practices, making federated collaboration [68] more difficult.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [68]

      Not checked in the free run.

  • May combine two claims.

    Interpretationnone given

    Another significant challenge for MFL is modality missing, pointing to some clients suffering data quantity imbalance among different modalities of their local data sets.

    “Another significant challenge for MFL is modality missing, pointing to some clients suffering data quantity imbalance among different modalities of their local data sets.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Modality missing happens frequently in realistic scenarios caused by hardware limitations, collection errors, and storage issues.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    In [13,14], the authors use MFL for AI-assisted disease diagnosis and achieve satisfying performance.

    “In [13,14], the authors use MFL for AI-assisted disease diagnosis and achieve satisfying performance.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [13]

      Not checked in the free run.

    • cites [14]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    Improving the capability of integrating large-scale pre-trained models will greatly enhance the performance of MFL systems.

    “Improving the capability of integrating large-scale pre-trained models will greatly enhance the performance of the MFL systems.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationnone given

    Modality heterogeneity could result in more divergent gradients and even heterogeneous local model architectures.

    “The modality heterogeneity could result in more divergent gradients and even heterogeneous local model architectures.”
    Quote in the paperSubstantial claim
104 of 104

  • The paper relies on this without support.

    Most existing federated learning studies focus on taking unimodal data, such as images and text, as the model input and resolving the non-IID challenge caused by data distribution imbalance related to data labels and data amount.

    “Most existing FL studies focus on taking unimodal data, such as images and text, as the model input and resolving the heterogeneity challenge, i.e., the non-identically distributed (non-IID) challenge caused by data distribution imbalance related to data labels and data amount.”

  • The paper relies on this without support.

    In real-world applications, data are usually described by multiple modalities.

    “In real-world applications, data are usually described by multiple modalities.”

  • The paper relies on this without support.

    Only a handful of studies have been proposed to improve system performance by utilizing multimodal data.

    “However, to the best of our knowledge, only a handful of studies have been proposed to improve the system performance by utilizing multimodal data.”

  • The paper relies on this without support.

    The majority of previous work focuses on the unimodal setting where all clients in the federated system hold the same data modality.

    “However, the majority of the previous work focuses on the unimodal setting where all the clients in the federated system hold the same data modality, as shown in Figure 1.”

  • The paper relies on this without support.

    The heterogeneity problem in unimodal FL is usually caused by the imbalance of data quantity and data label skews.

    “The heterogeneity problem in unimodal FL is usually caused by the imbalance of data quantity and data label skews.”

  • The paper relies on this without support.

    Modality missing happens frequently in realistic scenarios caused by hardware limitations, collection errors, and storage issues.

    “The modality missing happens frequently in realistic scenarios caused by hardware limitations, collection errors, and storage issues.”

  • The paper relies on this without support.

    Healthcare data such as electronic health records, X-ray, and CT scan images are stored and managed in a privacy-sensitive manner.

    “those healthcare data, such as electronic health records (EHR), X-ray, and CT scan images, are stored and managed in a privacy-sensitive manner.”

  • The paper relies on this without support.

    Healthcare data are more complex and diverse in both format and granularity than data in the IoT or multimedia domain.

    “Different from the application in IoT or multimedia domain, the healthcare data are more complex and diverse in both format and granularity.”

  • The paper relies on this without support.

    The cost of building and training large-scale pre-trained models is extremely expensive and not affordable for most computing devices and data centers.

    “the cost of building and training those large-scale pre-trained models could be extremely expensive and not affordable for most computing devices and data centers.”

  • The paper relies on this without support.

    Massive training data is required to gather for effective training of large foundation models.

    “massive training data is required to gather for effective training of large foundation models.”

  • The paper relies on this without support.

    Clients in federated learning systems usually have limited hardware resources and communication bandwidth.

    “In a federated learning system, the clients usually have limited hardware resources and communication bandwidth”

  • The paper relies on this without support.

    In realistic application scenarios, collected multimodal data in data silos usually contain limited supervision signals such as labels or matching relationships among the modalities.

    “In realistic application scenarios, the collected multimodal data in data silos usually contain limited supervision signals such as labels or matching relationships among the modalities.”
12 of 12

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Hybrid example is described as including a unimodal client

    Minor

    The Figure 4 description says the illustrated system includes unimodal clients, but the specified three clients each hold screen-time data plus one other modality. All three are therefore multimodal under the paper's definition.

  • FedNorm is classified as congruent despite clients having different modalities

    Minor

    The introduction lists FedNorm [15] under congruent MFL, which it describes as involving clients with the same or similar modality combinations. The healthcare section instead says FedNorm trains separate unimodal CT and MRI clients.

2 of 2

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 92 · Cited by 127

Research works printed but not linked: 10

  1. Liang, P.P.; Liu, T.; Ziyin, L.; Allen, N.B.; Auerbach, R.P.; Brent, D.; Salakhutdinov, R.; Morency, L.P. Think locally, act globally: Federated learning with local and global representations. arXiv preprint arXiv:2001.01523 2020. · arXiv
  2. Zong, L.; Xie, Q.; Zhou, J.; Wu, P.; Zhang, X.; Xu, B. FedCMR: Federated Cross-Modal Retrieval. Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2021, pp. 1672-1676.
  3. Tan, A.Z.; Yu, H.; Cui, L.; Yang, Q. Towards personalized federated learning. IEEE Transactions on Neural Networks and Learning Systems 2022.
  4. T Dinh, C.; Tran, N.; Nguyen, J. Personalized federated learning with moreau envelopes. Advances in Neural Information Processing Systems 2020, 33, 21394-21405.
  5. Banos, O.; Garcia, R.; Saez, A. MHEALTH Dataset. UCI Machine Learning Repository, 2014. DOI: https://doi.org/10.24432/C5TW22. · DOI
  6. Goldberger, A.L.; Amaral, L.A.; Glass, L.; Hausdorff, J.M.; Ivanov, P.C.; Mark, R.; Mietus, J.E.; Moody, G.B.; Peng, C.K.; Stanley, H.E. PhysioBank, PhysioToolkit, and PhysioNet: Components of a new research resource for complex physiologic signals. Circulation 2000, 101, e215-e220.
  7. Ma, M.; Ren, J.; Zhao, L.; Tulyakov, S.; Wu, C.; Peng, X. Smil: Multimodal learning with severely missing modality. Proceedings of the AAAI Conference on Artificial Intelligence, 2021, Vol. 35, pp. 2302-2310.
  8. Rieke, N.; Hancox, J.; Li, W.; Milletari, F.; Roth, H.R.; Albarqouni, S.; Bakas, S.; Galtier, M.N.; Landman, B.A.; Maier-Hein, K.; others. The future of digital health with federated learning. NPJ digital medicine 2020, 3, 119.
  9. Tian, Y.; Wan, Y.; Lyu, L.; Yao, D.; Jin, H.; Sun, L. FedBERT: when federated learning meets pre-training. ACM Transactions on Intelligent Systems and Technology (TIST) 2022, 13, 1-26.
  10. Jeong, W.; Yoon, J.; Yang, E.; Hwang, S.J. Federated semi-supervised learning with inter-client consistency & disjoint learning. arXiv preprint arXiv:2006.12097 2020. · arXiv

Authors
Che, L., Wang, J., Zhou, Y., & Ma, F.
Affiliations
  • Pennsylvania State University (US)
  • Instacart (United States) (US)
Corresponding author
Ma, Fenglong
ORCID
Venue
Sensors
Publisher
MDPI AG
Published
2023-07-20
Volume
23
Issue
15
Pages
6986-6986
ISSN
1424-8220 (ISSN-L)
DOI
10.20944/preprints202307.1420.v1
PMC
PMC10422520
OpenAlex
W4385615086
Cited by
127
References
92
Citations by year
2026: 39 · 2025: 48 · 2024: 38 · 2023: 2
FWCI
2.65
Type
journal article
Language
en
Open access
Licence
Locations
  • Sensors (publishedVersion, cc-by)
  • Preprints.org (acceptedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Privacy-Preserving Technologies in Data
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
Keywords
multimodal federated learning, federated learning, multimodal data, non-IID data, data heterogeneity

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-08.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-08.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review