Fusion of tactile and visual information in deep learning models for object recognition

Deep learning

LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works

ImageNet Large Scale Visual Recognition Challenge

Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works

Fully convolutional networks for semantic segmentation

Shelhamer, E., Long, J., & Darrell, T. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2572683 · Cited by 11,898 · see citing works

LSTM: A Search Space Odyssey

Greff, K., Srivastava, R. K., Koutnik, J., Steunebrink, B. R., & Schmidhuber, J. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2016.2582924 · Cited by 6,955 · see citing works

Multimodal Machine Learning: A Survey and Taxonomy

Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018 · see citing works

Humans integrate visual and haptic information in a statistically optimal fashion

Ernst, M. O., & Banks, M. S. (2002) · Nature · doi:10.1038/415429a · Cited by 4,844 · see citing works

VQA: Visual Question Answering

Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487 · see citing works

AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks

Xu, T., Zhang, P., Huang, Q., Zhang, H., Gan, Z., Huang, X., & He, X. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00143 · Cited by 1,923 · see citing works

Learning the signatures of the human grasp using a scalable tactile glove

Sundaram, S., Kellnhofer, P., Li, Y., Zhu, J.-Y., Torralba, A., & Matusik, W. (2019) · Nature · doi:10.1038/s41586-019-1234-z · Cited by 1,162 · see citing works

Deep Visual-Semantic Alignments for Generating Image Descriptions

Karpathy, A., & Fei-Fei, L. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2598339 · Cited by 1,048 · see citing works

A survey on deep learning for multimodal data fusion

Gao, J., Li, P., Chen, Z., & Zhang, J. (2020) · Neural Computation · doi:10.1162/neco_a_01273 · Cited by 821 · see citing works

Deep Audio-Visual Speech Recognition

Afouras, T., Chung, J. S., Senior, A., Vinyals, O., & Zisserman, A. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2889052 · Cited by 801 · see citing works

From Eliza to XiaoIce: Challenges and opportunities with social chatbots

Shum, H.-Y., He, X.-D., & Li, D. (2018) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.1700826 · Cited by 749 · see citing works

Attribute2Image: Conditional Image Generation from Visual Attributes

Yan, X., Yang, J., Sohn, K., & Lee, H. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46493-0_47 · Cited by 688 · see citing works

Regularizing Neural Networks by Penalizing Confident Output Distributions

Pereyra, G., Tucker, G., Chorowski, J., Kaiser, Ł., & Hinton, G. (2017) · arXiv · doi:10.48550/arxiv.1701.06548 · Cited by 655 · see citing works

Deep Multimodal Representation Learning: A Survey

Guo, W., Wang, J., & Wang, S. (2019) · IEEE Access · doi:10.1109/access.2019.2916887 · Cited by 535 · see citing works

Multimodal intelligence: Representation learning, information fusion, and applications

Zhang, C., Yang, Z., He, X., & Deng, L. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.2987728 · Cited by 511 · see citing works

Involvement of visual cortex in tactile discrimination of orientation

Zangaladze, A., Epstein, C. M., Grafton, S. T., & Sathian, K. (1999) · Nature · doi:10.1038/44139 · Cited by 508 · see citing works

Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks

Lee, M. A., Zhu, Y., Zachares, P., Tan, M., Srinivasan, K., Savarese, S., Fei-Fei, L., Garg, A., & Bohg, J. (2020) · IEEE Transactions on Robotics · doi:10.1109/tro.2019.2959445 · Cited by 241 · see citing works

Neural Synergy Between Kinetic Vision and Touch

Blake, R., Sobel, K. V., & James, T. W. (2004) · Psychological Science · doi:10.1111/j.0956-7976.2004.00691.x · Cited by 225 · see citing works

Neural Basis of Touch and Proprioception in Primate Cortex

Delhaye, B. P., Long, K. H., & Bensmaia, S. J. (2018) · Comprehensive Physiology · doi:10.1002/cphy.c170033 · Cited by 214 · see citing works

Robotic learning of haptic adjectives through physical interaction

Chu, V. C.-M., McMahon, I. T., Riano, L., McDonald, C. G., He, Q. L., Perez-Tejada, J. M., Arrigo, M., Darrell, T., & Kuchenbecker, K. J. (2014) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2014.09.021 · Cited by 142 · see citing works

Deep Learning for Surface Material Classification Using Haptic and Visual Information

Zheng, H., Fang, L., Ji, M., Strese, M., Özer, Y., & Steinbach, E. G. (2016) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2016.2598140 · Cited by 128 · see citing works

Multi-sensor fusion for body sensor network in medical human–robot interaction scenario

Lin, K., Li, Y., Sun, J., Zhou, D., & Zhang, Q. (2019) · Information Fusion · doi:10.1016/j.inffus.2019.11.001 · Cited by 125 · see citing works

Time Domain Audio Visual Speech Separation

Wu, J., Xu, Y., Zhang, S., Chen, L., Yu, M., Xie, L., & Yu, D. (2019) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru46091.2019.9003983 · Cited by 116 · see citing works

The Linear Separability Problem: Some Testing Methods

Elizondo, D. A. (2006) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2005.860871 · Cited by 113 · see citing works

From Eliza to XiaoIce: Challenges and Opportunities with Social ChatbotsPreprint

Shum, H., He, X., & Li, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.01957 · Cited by 66 · see citing works

A closed-loop neurobotic system for fine touch sensing

Bologna, L. L., Pinoteau, J., Passot, J.-B., Garrido, J. A., Vogel, J., Vidal, E. R., & Arleo, A. (2013) · Journal of Neural Engineering · doi:10.1088/1741-2560/10/4/046019 · Cited by 43 · see citing works

Deeply Supervised Subspace Learning for Cross-Modal Material Perception of Known and Unknown Objects

Xiong, P., Liao, J., Zhou, M., Song, A., & Liu, P. X. (2022) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2022.3195171 · Cited by 41 · see citing works

Partial Visual-Tactile Fused Learning for Robotic Object Recognition

Zhang, T., Cong, Y., Dong, J., & Hou, D. (2021) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2021.3096235 · Cited by 40 · see citing works

Cross-Modal Zero-Shot-Learning for Tactile Object Recognition

Liu, H., Sun, F., Fang, B., & Guo, D. (2018) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2018.2818184 · Cited by 37 · see citing works

Robot Intelligent Grasp of Unknown Objects Based on Multi-Sensor Information

Ji, S.-Q., Huang, M.-B., & Huang, H. (2019) · Sensors · doi:10.3390/s19071595 · Cited by 35 · see citing works

Knowledge-based multimodal information fusion for role recognition and situation assessment by using mobile robot

Yang, C., Wang, D., Zeng, Y., Yue, Y., & Siritanawan, P. (2018) · Information Fusion · doi:10.1016/j.inffus.2018.10.007 · Cited by 31 · see citing works

A Deep Learning Framework for Tactile Recognition of Known as Well as Novel Objects

Abderrahmane, Z., Ganesh, G., Crosnier, A., & Cherubini, A. (2019) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2019.2898264 · Cited by 29 · see citing works

Visual–Tactile Fused Graph Learning for Object Clustering

Zhang, T., Cong, Y., Sun, G., & Dong, J. (2021) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2021.3080321 · Cited by 27 · see citing works

Human-Exploratory-Procedure-Based Hybrid Measurement Fusion for Material Recognition

Xiong, P., He, K., Wu, E. Q., Zhu, L., Song, A., & Liu, P. X. (2021) · IEEE/ASME Transactions on Mechatronics · doi:10.1109/tmech.2021.3080378 · Cited by 26 · see citing works

Transfer of Learning from Vision to Touch: A Hybrid Deep Convolutional Neural Network for Visuo-Tactile 3D Object Recognition

Rouhafzay, G., Creţu, A.-M., & Payeur, P. (2020) · Sensors · doi:10.3390/s21010113 · Cited by 22 · see citing works

Robotic Multifinger Grasping State Recognition Based on Adaptive Multikernel Dictionary Learning

Xiong, P., Tong, X., Song, A., et al. (2022) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2022.3178500 · Cited by 22 · see citing works

Active Visual-Tactile Cross-Modal Matching

Liu, H., Wang, F., Sun, F., et al. (2018) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2018.2819826 · Cited by 18 · see citing works

A glove-based system for object recognition via visual-tactile fusion

Fang, B., Sun, F., Liu, H., et al. (2019) · Science China Information Sciences · doi:10.1007/s11432-018-9606-6 · Cited by 14 · see citing works

Complementing visual tracking of moving targets by fusion of tactile sensing

Alkkiomäki, O., Kyrki, V., Kalviainen, H. A., et al. (2009) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2009.07.001 · Cited by 13 · see citing works

Vision is superior to touch in shape perception even with equivalent peripheral input

Cho, Y., Craig, J. C., Hsiao, S. S., & Bensmaı̈a, S. J. (2015) · Journal of Neurophysiology · doi:10.1152/jn.00654.2015 · Cited by 11 · see citing works

Task-specific grasping of simiiar objects by probabiiistic fusion of vision and tactiie measurements

Nikandrova, E., & Kyrki, V. (2015) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/humanoids.2015.7363431 · Cited by 7 · see citing works

Multimodal fusion and vision–language models: A survey for robot vision

Han, X., Chen, S., Fu, Z., Feng, Z., Fan, L., An, D., Wang, C., Guo, L., Meng, W., Zhang, X., et al. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103652 · Cited by 41 · see citing works

Robust multimodal federated learning for incomplete modalities

Yu, S., Wang, J., Hussein, W., & Hung, P. C. K. (2023) · Computer Communications · doi:10.1016/j.comcom.2023.12.003 · Cited by 24 · see citing works

Found in Translation: Learning Robust Joint Representations by Cyclic Translations between Modalities

Pham, H., Liang, P. P., Manzini, T., Morency, L.-P., & Póczos, B. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016892 · Cited by 512 · see citing works

Modality Influence in Multimodal Machine LearningPreprint

Haouhat, A., Bellaouar, S., Nehar, A., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.06476

Multimodal Conversational AI: A Survey of Datasets and Approaches

Sundar, A., & Heck, L. P. (2022) · Workshop on NLP for Conversational AI (NLP4ConvAI) · doi:10.18653/v1/2022.nlp4convai-1.12 · Cited by 37 · see citing works

Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between ModalitiesPreprint

Pham, H., Liang, P. P., Manzini, T., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.07809 · Cited by 38 · see citing works