The DOI matches its Crossref record; 65 of the 65 references it cites were found in a registry or an index.
Semi-Supervised Multimodal Representation Learning Through a Global Workspace
Open accessAbstract
Recent deep learning models can efficiently combine inputs from different modalities (e.g., images and text) and learn to align their latent representations or to translate signals from one domain to another (as in image captioning or text-to-image generation). However, current approaches mainly rely on brute-force supervised training over large multimodal datasets. In contrast, humans (and other animals) can learn useful multimodal representations from only sparse experience with matched cross-modal data. Here, we evaluate the capabilities of a neural network architecture inspired by the cognitive notion of a "global workspace" (GW): a shared representation for two (or more) input modalities. Each modality is processed by a specialized system (pretrained on unimodal data and subsequently frozen). The corresponding latent representations are then encoded to and decoded from a single shared workspace. Importantly, this architecture is amenable to self-supervised training via cycle-consistency: encoding-decoding sequences should approximate the identity function. For various pairings of vision-language modalities and across two datasets of varying complexity, we show that such an architecture can be trained to align and translate between two modalities with very little need for matched data (from four to seven times less than a fully supervised approach). The GW representation can be used advantageously for downstream classification and cross-modal retrieval tasks and for robust transfer learning. Ablation studies reveal that both the shared workspace and the self-supervised cycle-consistency training are critical to the system's performance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, recent studies have used AI to test predictions of AST [ 75,76] and GWT [ 77,78].”
From Identifying indicators of consciousness in AI systems (2026) · cites this paper as [77]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“We use a ResNet50 [50] pretrained on ImageNet [51] for our visual domain encoder and Beijing Academy of Artificial Intelligence general embed-dings (BGEs) [52] for the text encoder.”
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1×
“We use a ResNet50 [50] pretrained on ImageNet [51] for our visual domain encoder and Beijing Academy of Artificial Intelligence general embed-dings (BGEs) [52] for the text encoder.”
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Unpaired image-to-image translation using cycle-consistent adversarial networks
Zhu, J.-Y., Park, T., Isola, P., & Efros, A. A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.244 · Cited by 23,602
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302 · cited 1×
“…objectives for multimodal learning in the literature; for example, text-to-image generation systems (such as DALL · E 2 [10] or Stable Diffusion [42]) and image captioning models (such as CoCa [9]) are based on translation objectives, while CLIP [2] relies on contrastive learning for…”
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 1×
“For instance, Lample et al. [23] used back-translations to optimize a sequence-to-sequence model with attention [24] so that it could translate between two languages without ever having access to aligned multilingual corpora during training.”
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Deep Visual-Semantic Alignments for Generating Image DescriptionsPreprint
Karpathy, A., & Fei-Fei, L. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.48550/arxiv.1412.2306 · Cited by 6,303 · cited 2×
“We use the popular Karpathy split [47] with 5000 images for both validation and testing and use the remaining 30 504 images for additional training examples (“restval”).”
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902
Research works printed but not linked: 2
- Jaegle, 2021, Perceiver: General Perception With Iterative Attention
- 2023, Open-Source Mobile Robot Simulation Software
Cited by
Identifying indicators of consciousness in AI systems
Butlin, P., Long, R., Bayne, T., Bengio, Y., Birch, J., Chalmers, D., Constant, A., Deane, G., Elmoznino, E., Fleming, S. M., et al. (2026) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2025.10.011 · Cited by 56
From a Social Cognitive Perspective: Context-Aware Visual Social Relationship Recognition
Wu, S., Zhang, C., Chen, J., Xu, T., Wu, L. J., Wang, Q., Gao, Y. F., Hu, Y. B., & Chen, E. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3623197 · Cited by 4
“Similar to the principle [61], we further regularize MSAT with shared parameters across visual and linguistic branches, which tightens the coupling between modalities and improves generalization.”
Meng, L., Song, Z., & Lu, J. (2025) · Brain Informatics · doi:10.1186/s40708-025-00282-x · Cited by 1
Foundation model-based generative AI for smart manufacturing: A paradigm shift
Zheng, H., Geng, Z., & Xu, X. W. (2026) · ENGINEERING Management · doi:10.1007/s42524-026-5359-0 · Cited by 6
“This requires fundamentally new representation learning and tokenization strategies (Dai et al., 2023; Mai et al., 2023; Sun et al., 2023; Devillers et al., 2025; Zhao et al., 2024a) to prevent the loss of signal fidelity when translating physical dynamics into semantic descriptions, a hurdle not…”
Wang, E., & Khan, F. A. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.2596 · Cited by 1
“Devillers, Maytié & VanRullen (2024) proposed a novel multimodal dataset for investigating the emotional connotations of colors as mediated by lines, strokes, textures, shapes, and language.”
Metadata
- Authors
- Devillers, B., Maytié, L., & VanRullen, R.
- Affiliations
- Centre National de la Recherche Scientifique (FR)
- CerCo, CNRS UMR 5549, Université de Toulouse, Toulouse, France
- Venue
- IEEE Transactions on Neural Networks and Learning Systems
- Published
- 2025-05
- Volume
- 36
- Issue
- 5
- Pages
- 7843-7857
- ISSN
- 2162-237X (ISSN-L)
- 2162-2388
- OpenAlex
- W4400229207
- Semantic Scholar
- 8a1130871ea53b76ce6e1b5093983bf37a157fb0
- Cited by
- 5
- References
- 56
- Citations by year
- 2026: 1 · 2025: 3
- FWCI
- 0.70
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- IEEE Transactions on Neural Networks and Learning Systems (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- HAL (submittedVersion)
- arXiv (Cornell University) (submittedVersion)
- HAL (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- Funders
- High Performance Computing (HPC) Resources from CALMIP (2020-p20032)
- Agence Nationale de la Recherche (ANR) grant Natural Language Programming for Conversational Cobots (ANR-21-FAI2-0005)
- Region Occitanie grant Cobots Conversationnels pour Processus Industriels et Logistiques
- “Défi-clé Robotique centrée sur l’humain” Ph.D.-thesis grant
- Artificial and Natural Intelligence Toulouse Institute (ANITI) Chair (ANR-19-PI3A-004)
- European Research Council (ERC) Advanced Grant Global Latent Workspace (101096017)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- global workspace, multimodal representation learning, cross-modal alignment, cross-modal translation, self-supervised learning, cycle consistency, vision-language models, transfer learning, cross-modal retrieval, shared latent representation, semi-supervised learning
- MeSH terms
- Supervised Machine Learning, Deep Learning, Algorithms, Animals, Humans, Databases, Factual, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Semi-Supervised Multimodal Representation Learning Through a Global Workspace. https://app.cochat.ai/papers/10.1109/tnnls.2024.3416701
- The paper
- Devillers, B., Maytié, L., & VanRullen, R. (2025). Semi-Supervised Multimodal Representation Learning Through a Global Workspace. IEEE Transactions on Neural Networks and Learning Systems. https://doi.org/10.1109/tnnls.2024.3416701
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.