Semi-Supervised Multimodal Representation Learning Through a Global Workspace
Open accessThe DOI matches its Crossref record; 65 of the 65 references it cites were found in a registry or an index.
Abstract
Recent deep learning models can efficiently combine inputs from different modalities (e.g., images and text) and learn to align their latent representations or to translate signals from one domain to another (as in image captioning or text-to-image generation). However, current approaches mainly rely on brute-force supervised training over large multimodal datasets. In contrast, humans (and other animals) can learn useful multimodal representations from only sparse experience with matched cross-modal data. Here, we evaluate the capabilities of a neural network architecture inspired by the cognitive notion of a "global workspace" (GW): a shared representation for two (or more) input modalities. Each modality is processed by a specialized system (pretrained on unimodal data and subsequently frozen). The corresponding latent representations are then encoded to and decoded from a single shared workspace. Importantly, this architecture is amenable to self-supervised training via cycle-consistency: encoding-decoding sequences should approximate the identity function. For various pairings of vision-language modalities and across two datasets of varying complexity, we show that such an architecture can be trained to align and translate between two modalities with very little need for matched data (from four to seven times less than a fully supervised approach). The GW representation can be used advantageously for downstream classification and cross-modal retrieval tasks and for robust transfer learning. Ablation studies reveal that both the shared workspace and the self-supervised cycle-consistency training are critical to the system's performance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, recent studies have used AI to test predictions of AST [ 75,76] and GWT [ 77,78].”
From Identifying indicators of consciousness in AI systems (2026) · cites this paper as [77]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 1×
“We use a ResNet50 [50] pretrained on ImageNet [51] for our visual domain encoder and Beijing Academy of Artificial Intelligence general embed-dings (BGEs) [52] for the text encoder.”
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1×
“We use a ResNet50 [50] pretrained on ImageNet [51] for our visual domain encoder and Beijing Academy of Artificial Intelligence general embed-dings (BGEs) [52] for the text encoder.”
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Unpaired image-to-image translation using cycle-consistent adversarial networks
Zhu, J.-Y., Park, T., Isola, P., & Efros, A. A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.244 · Cited by 23,602
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302 · cited 1×
“…objectives for multimodal learning in the literature; for example, text-to-image generation systems (such as DALL · E 2 [10] or Stable Diffusion [42]) and image captioning models (such as CoCa [9]) are based on translation objectives, while CLIP [2] relies on contrastive learning for…”
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · cited 1×
“For instance, Lample et al. [23] used back-translations to optimize a sequence-to-sequence model with attention [24] so that it could translate between two languages without ever having access to aligned multilingual corpora during training.”
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902
Harnad, S. (1990) · Physica D: Nonlinear Phenomena · doi:10.1016/0167-2789(90)90087-6 · Cited by 4,235 · cited 1×
“Thanks to this multimodal convergence, human language is grounded in the sensory environment, and conversely, sensory perception is semantically grounded by its relation to language [1].”
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1512.03385 · Cited by 4,196 · cited 1×
“We use a ResNet50 [50] pretrained on ImageNet [51] for our visual domain encoder and Beijing Academy of Artificial Intelligence general embed-dings (BGEs) [52] for the text encoder.”
(1988) · Cited by 3,656
(2017) · International Conference on Learning Representations · Cited by 3,248
DeViSE: A Deep Visual-Semantic Embedding Model
Frome, A., Corrado, G. S., Shlens, J., Bengio, S., Dean, J., Ranzato, M., & Mikolov, T. (2013) · neural information processing systems · Cited by 3,134
Hierarchical Text-Conditional Image Generation with CLIP Latents
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., & Chen, M. (2022) · arXiv · doi:10.48550/arxiv.2204.06125 · Cited by 2,511 · cited 2×
“(b) Illustration of the primary and secondary desirable properties for multimodal systems.”
Fast Kd-Trees for the Kullback-Leibler Divergence and Other Decomposable Bregman Divergences
Pham, T., & Wagner, H. (2025) · Leibniz international proceedings in informatics · doi:10.4230/lipics.wads.2025.45 · Cited by 2,354
DualGAN: Unsupervised Dual Learning for Image-to-Image Translation
Yi, Z., Zhang, H., Tan, P., & Gong, M. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.310 · Cited by 2,134 · cited 1×
“Basedon this logic, cycle-consistency was also used to synchronize multiple visual domains, i.e., unsupervised image-to-image translation [25], [26], [27].”
TSM: Temporal Shift Module for Efficient Video Understanding
Lin, J., Gan, C., & Han, S. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00718 · Cited by 2,055 · cited 1×
“Yet, most recent multimodal systems used either contrastive learning (e.g., CLIP [2] and temporal shift module (TSM) [16]), which forces alignment of the modalities without preserving modality-specific information or translation objectives (e.g., visual representations from textual annotations…”
Zero-Shot Learning—A Comprehensive Evaluation of the Good, the Bad and the Ugly
Xian, Y., Lampert, C. H., Schiele, B., & Akata, Z. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2857768 · Cited by 1,678 · cited 1×
“…can be leveraged as natural and readily available annotations [2]; they can be used as semantic constraints to train zero-shot learning models [3], [4] and more capable and general systems [5] or to infuse each modality with additional semantic knowledge (i.e., multimodal grounding) [6], [7], [8].”
Microsoft COCO Captions: Data Collection and Evaluation Server
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., & Zitnick, C. L. (2015) · arXiv · doi:10.48550/arxiv.1504.00325 · Cited by 1,614 · cited 1×
“Finally, to replicate our findings on a dataset with naturalistic images and human-written captions, we will evaluate COCO Captions [46].”
Hybrid computing using a neural network with dynamic external memory
Graves, A., Wayne, G., Reynolds, M., Harley, T., Danihelka, I., Grabska-Barwińska, A., Colmenarejo, S. G., Grefenstette, E., Ramalho, T., Agapiou, J., et al. (2016) · Nature · doi:10.1038/nature20101 · Cited by 1,412 · cited 1×
“Moreover, more advanced modules could be envisioned in this dynamic context, such as a memory module [54], [55] or a world model [56].”
A neuronal model of a global workspace in effortful cognitive tasks
Dehaene, S., Kerszberg, M., & Changeux, J.-P. (1998) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.95.24.14529 · Cited by 1,347
Emergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264 · cited 1×
“This trend of brute-force supervised training on ever larger datasets has led to an impressive boost in the models’ performance and to the emergence of new abilities in recent large AI models [11].”
Boehning, D., & Snyder, S. H. (2003) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.26.041002.131047 · Cited by 1,225
Graves, A., Wayne, G., & Danihelka, I. (2014) · arXiv · doi:10.48550/arxiv.1410.5401 · Cited by 1,058
Global workspace theory of consciousness: toward a cognitive neuroscience of human experience
Baars, B. J. (2005) · Progress in Brain Research · doi:10.1016/s0079-6123(05)50004-9 · Cited by 888 · cited 3×
“The GW theory offers an account of multimodal integration in the human brain [14], [15], [32].”
ImageBind One Embedding Space to Bind Them All
Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K. V., Joulin, A., & Misra, I. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.01457 · Cited by 863 · cited 1×
“Such an extension would resemble the recently proposed extension of the CLIP vision-language alignment model [2] to the new “ImageBind” model, aligning several distinct modalities to a visual representation space [53].”
Forward-Backward Error: Automatic Detection of Tracking Failures
Kalal, Z., Mikolajczyk, K., & Matas, J. (2010) · 2010 20th International Conference on Pattern Recognition · doi:10.1109/icpr.2010.675 · Cited by 818 · cited 1×
“Kalal et al. [19] presented a forward–backward error to solve a visual point-tracking task.”
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2021) · arXiv · doi:10.48550/arxiv.2112.10752 · Cited by 737 · cited 1×
“…objectives for multimodal learning in the literature; for example, text-to-image generation systems (such as DALL · E 2 [10] or Stable Diffusion [42]) and image captioning models (such as CoCa [9]) are based on translation objectives, while CLIP [2] relies on contrastive learning for…”
MirrorGAN: Learning Text-To-Image Generation by Redescription
Qiao, T., Zhang, J., Xu, D., & Tao, D. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00160 · Cited by 644 · cited 1×
“From this point, it was not long until the technique was applied to multimodal use cases, such as text-to-image [28], [29], [30] or touch-to-image translation [31].”
Dual Learning for Machine Translation
Xia, Y., He, D., Qin, T., Wang, L., Yu, N., Liu, T., & Ma, W. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.00179 · Cited by 636 · cited 1×
“…principle was applied in natural language processing (NLP) for unsupervised neural translation: language alignment is successful when the successive translation from language A to language B, and then, back-translation from B to A returns the original sentence [20], [21], [22], [23].”
Unsupervised Neural Machine Translation
Artetxe, M., Labaka, G., Agirre, E., & Cho, K. (2017) · arXiv · doi:10.48550/arxiv.1710.11041 · Cited by 632 · cited 1×
“…principle was applied in natural language processing (NLP) for unsupervised neural translation: language alignment is successful when the successive translation from language A to language B, and then, back-translation from B to A returns the original sentence [20], [21], [22], [23].”
Phrase-Based & Neural Unsupervised Machine Translation
Lample, G., Ott, M., Conneau, A., Denoyer, L., & Ranzato, M. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1549 · Cited by 615 · cited 1×
“We define the demi-cycle-consistency, where and the (full) cycle-consistency property, where These two properties can be seen as unsupervised versions of the primary properties, inspired by the unsupervised language translation literature [23], [43].”
Unsupervised Machine Translation Using Monolingual Corpora Only
Lample, G., Conneau, A., Denoyer, L., & Ranzato, M. (2017) · arXiv · doi:10.48550/arxiv.1711.00043 · Cited by 584
C-Pack: Packed Resources For General Chinese Embeddings
Xiao, S., Liu, Z., Zhang, P., Muennighoff, N., Lian, D., & Nie, J.-Y. (2024) · Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3626772.3657878 · Cited by 523
C-Pack: Packed Resources For General Chinese Embeddings
Xiao, S., Liu, Z., Zhang, P., Muennighoff, N., Lian, D., & Nie, J.-Y. (2023) · arXiv · doi:10.48550/arxiv.2309.07597 · Cited by 514
Pham, H., Liang, P. P., Manzini, T., Morency, L.-P., & Póczos, B. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016892 · Cited by 512
CoCa: Contrastive Captioners are Image-Text Foundation Models
Yu, J., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., & Wu, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.01917 · Cited by 503 · cited 3×
“Recently, the CoCa model [9] confirmed that using both translation and contrastive objectives for supervision could lead to significantly better performance.”
Unsupervised Image-to-Image Translation Networks
Liu, M.-Y., Breuel, T. M., & Kautz, J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.00848 · Cited by 477 · cited 1×
“Basedon this logic, cycle-consistency was also used to synchronize multiple visual domains, i.e., unsupervised image-to-image translation [25], [26], [27].”
LiT: Zero-Shot Transfer with Locked-image text Tuning
Zhai, X., Wang, X., Mustafa, B., Steiner, A., Keysers, D., Kolesnikov, A., & Beyer, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01759 · Cited by 402 · cited 1×
“In addition, recent studies have shown that current multimodal networks sometimes fail to improve upon unimodal networks, i.e., they do not always achieve proper multimodal grounding [12], [13].”
VirTex: Learning Visual Representations from Textual Annotations
Desai, K., & Johnson, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01101 · Cited by 342 · cited 1×
“…of the modalities without preserving modality-specific information or translation objectives (e.g., visual representations from textual annotations (VirTex) [17] and image-conditioned masked language modeling (ICMLM) [18]), which do not provide a joint multimodal space for downstream tasks.”
(2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1 · Cited by 235
Phrase-Based & Neural Unsupervised Machine TranslationPreprint
Lample, G., Ott, M., Conneau, A., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.07755 · Cited by 226 · cited 1×
“We define the demi-cycle-consistency, where and the (full) cycle-consistency property, where These two properties can be seen as unsupervised versions of the primary properties, inspired by the unsupervised language translation literature [23], [43].”
Word Translation Without Parallel Data
Conneau, A., Lample, G., Ranzato, M., Denoyer, L., & Jeǵou, H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.04087 · Cited by 219 · cited 1×
“…principle was applied in natural language processing (NLP) for unsupervised neural translation: language alignment is successful when the successive translation from language A to language B, and then, back-translation from B to A returns the original sentence [20], [21], [22], [23].”
Learning Visual Representations with Caption Annotations
Sarıyıldız, M. B., Pérez, J., & Larlus, D. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58598-3_10 · Cited by 191
Self-Supervised MultiModal Versatile Networks
Alayrac, J.-B., Recasens, A., Schneider, R. G., Arandjelović, R., Ramapuram, J., Fauw, J. D., Smaira, L., Dieleman, S., & Zisserman, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.16228 · Cited by 189 · cited 1×
“…CLIP [2] and temporal shift module (TSM) [16]), which forces alignment of the modalities without preserving modality-specific information or translation objectives (e.g., visual representations from textual annotations (VirTex) [17] and image-conditioned masked language modeling (ICMLM)…”
Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the UglyPreprint
Xian, Y., Lampert, C. H., Schiele, B., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.00600 · Cited by 171 · cited 1×
“…can be leveraged as natural and readily available annotations [2]; they can be used as semantic constraints to train zero-shot learning models [3], [4] and more capable and general systems [5] or to infuse each modality with additional semantic knowledge (i.e., multimodal grounding) [6], [7], [8].”
Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks
Lu, J., Clark, C., Zellers, R., Mottaghi, R., & Kembhavi, A. (2022) · arXiv · doi:10.48550/arxiv.2206.08916 · Cited by 145 · cited 1×
“…can be leveraged as natural and readily available annotations [2]; they can be used as semantic constraints to train zero-shot learning models [3], [4] and more capable and general systems [5] or to infuse each modality with additional semantic knowledge (i.e., multimodal grounding) [6], [7], [8].”
Scalable Deep Multimodal Learning for Cross-Modal Retrieval
Hu, P., Zhen, L., Peng, D., & Liu, P. (2019) · Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3331184.3331213 · Cited by 136 · cited 1×
“The prevalent form is image–text retrieval, which consists of either retrieving the caption of an image or the image that matches a specific description.”
Connecting Touch and Vision via Cross-Modal Prediction
Li, Y., Zhu, J.-Y., Tedrake, R., & Torralba, A. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01086 · Cited by 131 · cited 1×
“From this point, it was not long until the technique was applied to multimodal use cases, such as text-to-image [28], [29], [30] or touch-to-image translation [31].”
Deep learning and the global workspace theory
VanRullen, R., & Kanai, R. (2021) · Trends in Neurosciences · doi:10.1016/j.tins.2021.04.005 · Cited by 128
Mastering Diverse Domains through World Models
Hafner, D., Pasukonis, J., Ba, J., & Lillicrap, T. (2023) · arXiv · doi:10.48550/arxiv.2301.04104 · Cited by 124
Deep Visual-Semantic Alignments for Generating Image Descriptions
Karpathy, A., & Fei-Fei, L. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.2306 · Cited by 114 · cited 2×
“We use the popular Karpathy split [47] with 5000 images for both validation and testing and use the remaining 30 504 images for additional training examples (“restval”).”
Deep Multimodal Transfer Learning for Cross-Modal Retrieval
Zhen, L., Hu, P., Peng, X., Goh, R. S. M., & Zhou, J. T. (2020) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3029181 · Cited by 108 · cited 1×
“Zhen et al. [39] also trained their model with limited pairs of image/text.”
Grounded Models of Semantic Representation
Silberer, C., & Lapata, M. (2012) · Repositori digital de la UPF (Universitat Pompeu Fabra) · Cited by 97
Multi- and Cross-Modal Semantics Beyond Vision: Grounding in Auditory Perception
Kiela, D., & Clark, S. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1293 · Cited by 89 · cited 1×
“…can be leveraged as natural and readily available annotations [2]; they can be used as semantic constraints to train zero-shot learning models [3], [4] and more capable and general systems [5] or to infuse each modality with additional semantic knowledge (i.e., multimodal grounding) [6], [7], [8].”
Pham, H., Liang, P. P., Manzini, T., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.07809 · Cited by 38 · cited 13×
“We then follow the “zero-shot classification” procedure of CLIP [2] to match each image input to the most similar prototype.”
InfluentialThe Perceiver Architecture is a Functional Global Workspace
Cognitive Science Society 2022, & Juliani, A. (2022) · eScholarship (California Digital Library) · doi:10.48448/phj9-7y34 · Cited by 34
C4Synth: Cross-Caption Cycle-Consistent Text-to-Image Synthesis
Joseph, K. J., Pal, A., Rajanala, S., & Balasubramanian, V. N. (2019) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2019.00044 · Cited by 28
Does language help generalization in vision models?
Devillers, B., Choksi, B., Bielawski, R., & VanRullen, R. (2021) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2021.conll-1.13 · Cited by 28
Self-citationMultimodal Disentanglement Variational AutoEncoders for Zero-Shot Cross-Modal Retrieval
Tian, J., Wang, K., Xu, X., Cao, Z., Shen, F., & Shen, H. T. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3532028 · Cited by 20 · cited 1×
“Tian et al. [38] use a combination of coupled discrete variational autoencoders (DVAEs) and a fusion-exchange variational autoencoder (VAE).”
Playing Games in the Dark: An Approach for Cross-Modality Transfer in Reinforcement Learning
Silva, R., Vasco, M., Melo, F. S., Paiva, A., & Veloso, M. (2020) · Adaptive Agents and Multi-Agents Systems · doi:10.65109/znur7157 · Cited by 20
Associate Latent Encodings in Learning from Demonstrations
Yin, H., Melo, F. S., Billard, A., & Paiva, A. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11040 · Cited by 19 · cited 1×
“They also use Kullback–Leibler (KL) divergence to align feature representations of the image and text encoders, as done in other works [40], [41].”
ImageBind: One Embedding Space To Bind Them AllPreprint
Girdhar, R., El-Nouby, A., Liu, Z., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.05665 · Cited by 14 · cited 1×
“Such an extension would resemble the recently proposed extension of the CLIP vision-language alignment model [2] to the new “ImageBind” model, aligning several distinct modalities to a visual representation space [53].”
Text to image generative model using constrained embedding space mapping
Chaudhury, S., Dasgupta, S., Munawar, A., Khan, M. A., & Tachibana, R. (2017) · Machine learning for signal processing ... · doi:10.1109/mlsp.2017.8168111 · Cited by 6 · cited 1×
“From this point, it was not long until the technique was applied to multimodal use cases, such as text-to-image [28], [29], [30] or touch-to-image translation [31].”
Research works printed but not linked: 2
- Jaegle, 2021, Perceiver: General Perception With Iterative Attention
- 2023, Open-Source Mobile Robot Simulation Software
Cited by
Identifying indicators of consciousness in AI systems
Butlin, P., Long, R., Bayne, T., Bengio, Y., Birch, J., Chalmers, D., Constant, A., Deane, G., Elmoznino, E., Fleming, S. M., et al. (2026) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2025.10.011 · Cited by 56
From a Social Cognitive Perspective: Context-Aware Visual Social Relationship Recognition
Wu, S., Zhang, C., Chen, J., Xu, T., Wu, L. J., Wang, Q., Gao, Y. F., Hu, Y. B., & Chen, E. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3623197 · Cited by 4
“Similar to the principle [61], we further regularize MSAT with shared parameters across visual and linguistic branches, which tightens the coupling between modalities and improves generalization.”
Meng, L., Song, Z., & Lu, J. (2025) · Brain Informatics · doi:10.1186/s40708-025-00282-x · Cited by 1
Foundation model-based generative AI for smart manufacturing: A paradigm shift
Zheng, H., Geng, Z., & Xu, X. W. (2026) · ENGINEERING Management · doi:10.1007/s42524-026-5359-0 · Cited by 6
“This requires fundamentally new representation learning and tokenization strategies (Dai et al., 2023; Mai et al., 2023; Sun et al., 2023; Devillers et al., 2025; Zhao et al., 2024a) to prevent the loss of signal fidelity when translating physical dynamics into semantic descriptions, a hurdle not…”
Wang, E., & Khan, F. A. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.2596 · Cited by 1
“Devillers, Maytié & VanRullen (2024) proposed a novel multimodal dataset for investigating the emotional connotations of colors as mediated by lines, strokes, textures, shapes, and language.”
Metadata
- Authors
- Devillers, B., Maytié, L., & VanRullen, R.
- Affiliations
- Centre National de la Recherche Scientifique (FR)
- CerCo, CNRS UMR 5549, Université de Toulouse, Toulouse, France
- Venue
- IEEE Transactions on Neural Networks and Learning Systems
- Published
- 2025-05
- Volume
- 36
- Issue
- 5
- Pages
- 7843-7857
- ISSN
- 2162-237X (ISSN-L)
- 2162-2388
- OpenAlex
- W4400229207
- Semantic Scholar
- 8a1130871ea53b76ce6e1b5093983bf37a157fb0
- Cited by
- 5
- References
- 56
- Citations by year
- 2026: 1 · 2025: 3
- FWCI
- 0.70
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- IEEE Transactions on Neural Networks and Learning Systems (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- HAL (submittedVersion)
- arXiv (Cornell University) (submittedVersion)
- HAL (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- Funders
- High Performance Computing (HPC) Resources from CALMIP (2020-p20032)
- Agence Nationale de la Recherche (ANR) grant Natural Language Programming for Conversational Cobots (ANR-21-FAI2-0005)
- Region Occitanie grant Cobots Conversationnels pour Processus Industriels et Logistiques
- “Défi-clé Robotique centrée sur l’humain” Ph.D.-thesis grant
- Artificial and Natural Intelligence Toulouse Institute (ANITI) Chair (ANR-19-PI3A-004)
- European Research Council (ERC) Advanced Grant Global Latent Workspace (101096017)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- global workspace, multimodal representation learning, cross-modal alignment, cross-modal translation, self-supervised learning, cycle consistency, vision-language models, transfer learning, cross-modal retrieval, shared latent representation, semi-supervised learning
- MeSH terms
- Supervised Machine Learning, Deep Learning, Algorithms, Animals, Humans, Databases, Factual, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.