Multimodal Data Fusion based on the Global Workspace Theory
Open accessAbstract
We propose a novel neural network architecture, named the Global Workspace Network (GWN), which addresses the challenge of dynamic and unspecified uncertainties in multimodal data fusion. Our GWN is a model of attention across modalities and evolving through time, and is inspired by the well-established Global Workspace Theory from the field of cognitive science. The GWN achieved average F1 score of 0.92 for discrimination between pain patients and healthy participants and average F1 score = 0.75 for further classification of three pain levels for a patient, both based on the multimodal EmoPain dataset captured from people with chronic pain and healthy people performing different types of exercise movements in unconstrained settings. In these tasks, the GWN significantly outperforms the typical fusion approach of merging by concatenation. We further provide extensive analysis of the behaviour of the GWN and its ability to address uncertainties (hidden noise) in multimodal data.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Compared to other scientific theories of consciousness, many more proposals have been made for the implementation of GWT in artificial systems.
“Compared to other scientific theories of consciousness, many more proposals have been made for the implementation of GWT in artificial systems (e.g.Franklin & Graesser 1999, Shanahan 2006, Bao et al. 2020).”
From Consciousness in Artificial Intelligence: Insights from the Science of Consciousness (2023) · cites this paper as Bao et al. 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336 · see citing works
Individual Comparisons by Ranking Methods
Wilcoxon, F. (1945) · Biometrics Bulletin · doi:10.2307/3001968 · Cited by 15,354 · see citing works
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883 · see citing works
Layer NormalizationPreprint
Ba, J., Kiros, J., & Hinton, G. E. (2016) · arXiv.org · doi:10.48550/arxiv.1607.06450 · Cited by 13,762 · see citing works
Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
Shi, X., Chen, Z., Wang, H., Yeung, D.-Y., Wong, W.-K., & Woo, W.-C. (2015) · arXiv · doi:10.48550/arxiv.1506.04214 · Cited by 7,709 · see citing works
Extracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436 · see citing works
Proceedings of the 25th international conference on Machine learning - ICML '08
(2008) · doi:10.1145/1390156 · Cited by 6,865 · see citing works
Comparison of the predicted and observed secondary structure of T4 phage lysozyme
Matthews, B. W. (1975) · Biochimica et Biophysica Acta (BBA) - Protein Structure · doi:10.1016/0005-2795(75)90109-9 · Cited by 5,399 · cited 3× · see citing works
“…of the GWN can be seen in Table 1 comparison with the CON-CATN baseline model, based on accuracy (ACC), Matthews Correlation Coefficient (MCC) (Matthews, 1975 Favours-Itself-Always (FIA) The given modality always pays attention to itself and never switches attention to the other modality.”
MethodMultimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018 · see citing works
Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms
Dietterich, T. G. (1998) · Neural Computation · doi:10.1162/089976698300017197 · Cited by 3,840 · see citing works
(1988) · Cited by 3,656 · see citing works
A Critical Review of Recurrent Neural Networks for Sequence LearningPreprint
Lipton, Z. C., Berkowitz, J. A., & Elkan, C. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.00019 · Cited by 2,711 · see citing works
Learning to forget: continual prediction with LSTM
Gers, F. A. (1999) · doi:10.1049/cp:19991218 · Cited by 2,436 · see citing works
Hierarchical Question-Image Co-Attention for Visual Question AnsweringPreprint
Lu, J., Yang, J., Batra, D., & Parikh, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.00061 · Cited by 1,303 · see citing works
Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects
Lahat, D., Adali, T., & Jutten, C. (2015) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2460697 · Cited by 1,296 · see citing works
Multimodal fusion for multimedia analysis: a survey
Atrey, P. K., Hossain, M. A., El Saddik, A., & Kankanhalli, M. S. (2010) · Multimedia Systems · doi:10.1007/s00530-010-0182-0 · Cited by 1,279 · see citing works
A Deep Reinforced Model for Abstractive Summarization
Paulus, R., Xiong, C., & Socher, R. (2017) · arXiv · doi:10.48550/arxiv.1705.04304 · Cited by 1,182 · see citing works
(2016) · doi:10.18653/v1/d16-1147 · Cited by 1,035 · see citing works
The conscious access hypothesis: origins and recent evidence
Baars, B. J. (2002) · Trends in Cognitive Sciences · doi:10.1016/s1364-6613(00)01819-2 · Cited by 994 · see citing works
In the Theater of Consciousness
Baars, B. J. (1997) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195102659.001.1 · Cited by 878 · see citing works
Handling imbalanced datasets: A review
Kotsiantis, S. B., Kanellopoulos, D. N., & Pintelas, P. (2006) · Cited by 875 · see citing works
Scalable tensor factorizations for incomplete data
Acar, E., Dunlavy, D., Kolda, T. G., & Mørup, M. (2010) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/j.chemolab.2010.08.004 · Cited by 604 · see citing works
Calhoun, V. D., & Jing Sui （Beijing Normal University）， my correct affiliation is beijing normal university, N. Q. U. O. S. A. T. (2016) · Biological Psychiatry Cognitive Neuroscience and Neuroimaging · doi:10.1016/j.bpsc.2015.12.005 · Cited by 451 · see citing works
Attention-Based Multimodal Fusion for Video Description
Hori, C., Hori, T., Lee, T.-Y., Zhang, Z., Harsham, B., Hershey, J. R., Marks, T. K., & Sumi, K. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.450 · Cited by 401 · see citing works
Visual Attention Model for Name Tagging in Multimodal Social Media
Lu, D., Neves, L., Carvalho, V., Zhang, N., & Ji, H. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1185 · Cited by 324 · see citing works
Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering
Fan, C., Zhang, X., Zhang, S., Wang, W., Zhang, C., & Huang, H. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00210 · Cited by 293 · see citing works
An Introductory Survey on Attention Mechanisms in NLP Problems
Hu, D. (2019) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-29513-4_31 · Cited by 291 · see citing works
Aung, M. H., Kaltwang, S., Romera‐Paredes, B., Martínez, B., Singh, A., Cella, M., Valstar, M., Meng, H., Kemp, A. H., Shafizadeh, M., et al. (2015) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2015.2462830 · Cited by 219 · see citing works
Self-citationA cognitive architecture that combines internal simulation with a global workspace
Shanahan, M. (2005) · Consciousness and Cognition · doi:10.1016/j.concog.2005.11.005 · Cited by 195 · see citing works
Effects of padding on LSTMs and CNNsPreprint
Dwarampudi, M., & Reddy, N. V. S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.07288 · Cited by 98 · see citing works
2023 31st European Signal Processing Conference (EUSIPCO)
(2023) · doi:10.23919/eusipco58844.2023 · Cited by 95 · see citing works
Weakly Paired Multimodal Fusion for Object Recognition
Liu, H., Wu, Y., Sun, F., Fang, B., & Guo, D. (2017) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2017.2692271 · Cited by 82 · see citing works
Multimodal Attention for Neural Machine TranslationPreprint
Çağlayan, O., Barrault, L., & Bougares, F. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.03976 · Cited by 78 · see citing works
Multi-Modal Sequence Fusion via Recursive Attention for Emotion Recognition
Beard, R., Das, R., Ng, R. W. M., Gopalakrishnan, P. G. K., Eerens, L., Świętojański, P., & Mikšík, O. (2018) · Conference on Computational Natural Language Learning · doi:10.18653/v1/k18-1025 · Cited by 77 · see citing works
A Method for Judicious Fusion of Inconsistent Multiple Sensor Data
Kumar, M., Garg, D. P., & Zachery, R. A. (2007) · IEEE Sensors Journal · doi:10.1109/jsen.2007.894905 · Cited by 75 · see citing works
Weakly Supervised Memory Networks.Preprint
Sukhbaatar, S., Szlam, A. D., Weston, J., & Fergus, R. (2015) · arXiv (Cornell University) · Cited by 74 · see citing works
Multi-Level Multimodal Common Semantic Space for Image-Phrase Grounding
Akbari, H., Karaman, S., Bhargava, S., Chen, B., Vondrick, C., & Chang, S. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01276 · Cited by 73 · see citing works
A spiking neuron model of cortical broadcast and competition
Shanahan, M. (2007) · Consciousness and Cognition · doi:10.1016/j.concog.2006.12.005 · Cited by 64 · see citing works
Liberman, Y., Samuels, R., Alpert, P., & Messer, H. (2014) · Atmospheric measurement techniques · doi:10.5194/amt-7-3549-2014 · Cited by 45 · see citing works
How Can Affect Be Detected and Represented in Technological Support for Physical Rehabilitation?
Olugbade, T. A., Singh, A., Bianchi-Berthouze, N., Marquardt, N., Aung, M., & Williams, A. C. D. C. (2019) · ACM Trans. Comput. Hum. Interact. · doi:10.1145/3299095 · Cited by 43 · see citing works
Self-citationSimultaneous analysis of coupled data matrices subject to different amounts of noise
Wilderjans, T. F., Ceulemans, E., Mechelen, I. V., & van den Berg, R. A. (2010) · British Journal of Mathematical and Statistical Psychology · doi:10.1348/000711010x513263 · Cited by 32 · see citing works
A Genre-Aware Attention Model to Improve the Likability Prediction of Books
Maharjan, S., Montes, M., Gonzalez, F. A., & Solorio, T. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1375 · Cited by 22 · see citing works
Attention in Multimodal Neural Networks for Person Re-identification
Lejbølle, A. R., Krogh, B., Nasrollahi, K., & Moeslund, T. B. (2018) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2018.00055 · Cited by 21 · see citing works
A neuronal global workspace for human-like control of a computer game character
Fountas, Z., Gamez, D., & Fidjeland, A. K. (2011) · IEEE Conference on Computational Intelligence and Games (CIG) · doi:10.1109/cig.2011.6032027 · Cited by 8 · see citing works
Self-citationChen, Z., Zhao, L., Li, Q., Song, X., & Zhang, J. (2022) · Lecture notes in networks and systems · doi:10.1007/978-3-030-87049-2_3 · Cited by 8 · see citing works
Flagged first, then by citations
Research works printed but not linked: 16
- Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 79, 2015, Conference Track Proceedings , 2015. URL http://arxiv.org/abs/1409.0473 . · arXiv
- Baltrusaitis, T., Ahuja, C., and Morency, L. Multimodal machine learning: A survey and taxonomy. CoRR , abs/1705.09406, 2017. URL http://arxiv.org/ abs/1705.09406 . · arXiv
- Bollegala, D. and Bao, C. Learning word meta-embeddings by autoencoding. In Proceedings of the 27th International Conference on Computational Linguistics , pp. 1650-1661, Santa Fe, New Mexico, USA, August 2018. Association for Computational Linguistics. URL https: //www.aclweb.org/anthology/C18-1140 .
- Cheng, J., Dong, L., and Lapata, M. Long shortterm memory-networks for machine reading. CoRR , abs/1601.06733, 2016. URL http://arxiv.org/ abs/1601.06733 . · arXiv
- Cho, K., van Merrienboer, B., Bahdanau, D., and Bengio, Y. On the properties of neural machine translation: Encoderdecoder approaches. CoRR , abs/1409.1259, 2014. URL http://arxiv.org/abs/1409.1259 . · arXiv
- He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. CoRR , abs/1512.03385, 2015. URL http://arxiv.org/abs/1512.03385 . · arXiv
- Jensen, M. P. and Karoly, P. Self-report scales and procedures for assessing pain in adults. Handbook of pain assessment , pp. 135-151, 1992.
- Lee, M. A., Zhu, Y., Srinivasan, K., Shah, P., Savarese, S., Fei-Fei, L., Garg, A., and Bohg, J. Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks. CoRR , abs/1810.10191, 2018. URL http://arxiv.org/ abs/1810.10191 . · arXiv
- Miller, A. H., Fisch, A., Dodge, J., Karimi, A., Bordes, A., and Weston, J. Key-value memory networks for directly reading documents. CoRR , abs/1606.03126, 2016. URL http://arxiv.org/abs/1606.03126 . · arXiv
- Nair, V. and Hinton, G. E. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on International Conference on Machine Learning , ICML'10, pp. 807814, USA, 2010. Omnipress. ISBN 978-1-60558-9077. URL http://dl.acm.org/citation.cfm? id=3104322.3104425 .
- Olugbade, T. A., Newbold, J. W., Johnson, R. M. G., Volta, E., Alborno, P., Niewiadomski, R., Dillon, M., Volpe, G., and Bianchi-Berthouze, N. Automatic detection of reflective thinking in mathematical problem solving based on unconstrained bodily exploration. CoRR , abs/1812.07941, 2018. URL http://arxiv.org/ abs/1812.07941 . · arXiv
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. In Advances in neural information processing systems , pp. 5998-6008, 2017. URL https://papers.nips.cc/paper/ 7181-attention-is-all-you-need.pdf .
- Wang, C., Olugbade, T. A., Mathur, A., de C. Williams, A. C., Lane, N. D., and Bianchi-Berthouze, N. Automatic detection of protective behavior in chronic pain physical rehabilitation: A recurrent neural network approach. CoRR , abs/1902.08990, 2019. URL http: //arxiv.org/abs/1902.08990 . · arXiv
- Weston, J., Chopra, S., and Bordes, A. Memory networks. CoRR , abs/1410.3916, 2015. · arXiv
- Wu, M. and Goodman, N. Multimodal generative models for scalable weakly-supervised learning. In 32nd Conference on Neural Information Processing Systems , pp. 5575-5585, 2018.
- Zadeh, A., Liang, P. P., Mazumder, N., Poria, S., Cambria, E., and Morency, L.-P. Memory fusion network for multi-view sequential learning. In Thirty-Second AAAI Conference on Artificial Intelligence , 2018. imekli, U., Ermi, B., Cemgil, A. T., and Acar, E. Optimal weight learning for coupled tensor factorization with mixed divergences. In 21st European Signal Processing Conference (EUSIPCO 2013) , pp. 1-5, Sep. 2013.
Cited by
Sorted by citations
A Review on Explainability in Multimodal Deep Neural Nets
Joshi, G., Walambe, R., & Kotecha, K. (2021) · IEEE Access · doi:10.1109/access.2021.3070212 · Cited by 204 · see citing works
Consciousness in Artificial Intelligence: Insights from the Science of Consciousness
Butlin, P., Long, R., Elmoznino, E., Bengio, Y., Birch, J., Constant, A., Deane, G., Fleming, S. M., Frith, C., Ji, X., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.08708 · Cited by 196 · see citing works
Deep learning and the global workspace theory
VanRullen, R., & Kanai, R. (2021) · Trends in Neurosciences · doi:10.1016/j.tins.2021.04.005 · Cited by 128 · see citing works
Sun, H., Chen, Y., & Lin, L. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3233070 · Cited by 77 · see citing works
Paul, S. (2023) · Journal of Defense Analytics and Logistics · doi:10.1108/jdal-11-2022-0010 · Cited by 12 · see citing works
Cognitive Robots and the Conscious Mind: A Review of the Global Workspace Theory
Signa, A., Chella, A., & Gentile, M. (2021) · Current Robotics Reports · doi:10.1007/s43154-021-00044-7 · Cited by 12 · see citing works
Exploring Multimodal Fusion for Continuous Protective Behavior Detection
Cen, G., Wang, C., Olugbade, T., Williams, A. C. D. C., & Bianchi‐Berthouze, N. (2022) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii55700.2022.9953851 · Cited by 9 · see citing works
Self-citationDeep Learning and the Global Workspace TheoryPreprint
Rufin Vanrullen, A., & Ryota Kanai, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.10390 · Cited by 4 · see citing works
Radouane, K., Tchechmedjiev, A., Xu, B., & Harispe, S. (2021) · International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW) · doi:10.1109/aciiw52867.2021.9666417 · Cited by 3 · see citing works
AI4Pain2026 Challenge on Automatic Pain Assessment from Body Movement
Azeez, F. O., Wang, J., Wei, X., Olugbade, T., & Bianchi‐Berthouze, N. (2026) · International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW) · doi:10.1109/aciiw71585.2026.11712104
Self-citation
- Authors
- Bao, C., Fountas, Z., Olugbade, T., & Bianchi-Berthouze, N.
- Affiliations
- University College London (GB)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- Pages
- 414-422
- arXiv
- 2001.09485
- OpenAlex
- W3093683378
- Cited by
- 9
- References
- 46
- Citations by year
- 2026: 1 · 2023: 1 · 2022: 2 · 2021: 4 · 2020: 1
- FWCI
- 0.62
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- UCL Discovery (University College London) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Horizon 2020 Framework Programme (824160)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Emotion and Mood Recognition
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Keywords
- multimodal fusion, global workspace theory, chronic pain, neural network architecture
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
