Detect, Reject, Correct: Crossmodal Compensation of Corrupted Sensors
Open accessAbstract
Using sensor data from multiple modalities presents an opportunity to encode redundant and complementary features that can be useful when one modality is corrupted or noisy. Humans do this everyday, relying on touch and proprioceptive feedback in visually-challenging environments. However, robots might not always know when their sensors are corrupted, as even broken sensors can return valid values. In this work, we introduce the Crossmodal Compensation Model (CCM), which can detect corrupted sensor modalities and compensate for them. CMM is a representation model learned with self-supervision that leverages unimodal reconstruction loss for corruption detection. CCM then discards the corrupted modality and compensates for it with information from the remaining sensors. We show that CCM learns rich state representations that can be used for manipulation policies learned with reinforcement learning, even when input modalities are corrupted during policy rollout in ways not seen during training.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Approaches for improving robustness modify the objective function to account for robustness to noisy or missing modalities.
“These approaches modify the objective function to account for robustness to noisy [101] or missing [89, 111, 123] modalities.”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [89]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Soft actor-critic Off-policy maximum entropy deep reinforcement learning with a stochastic actor
Haarnoja, T., Zhou, A., Abbeel, P., & Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1801.01290 · Cited by 5,594 · see citing works
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018 · see citing works
Humans integrate visual and haptic information in a statistically optimal fashion
Ernst, M. O., & Banks, M. S. (2002) · Nature · doi:10.1038/415429a · Cited by 4,844 · see citing works
Coding and use of tactile signals from the fingertips in object manipulation tasks
Johansson, R. S., & Flanagan, J. R. (2009) · Nature Reviews Neuroscience · doi:10.1038/nrn2621 · Cited by 2,063 · see citing works
The Ventriloquist Effect Results from Near-Optimal Bimodal Integration
Alais, D., & Burr, D. (2004) · Current Biology · doi:10.1016/j.cub.2004.01.029 · Cited by 1,936 · see citing works
Deep Learning for Anomaly Detection: A Survey
Chalapathy, R., & Chawla, S. (2019) · arXiv · doi:10.48550/arxiv.1901.03407 · Cited by 1,240 · see citing works
Cross-modal plasticity in specific auditory cortices underlies visual compensations in the deaf
Lomber, S. G., Meredith, M. A., & Kral, A. (2010) · Nature Neuroscience · doi:10.1038/nn.2653 · Cited by 521 · see citing works
Pham, H., Liang, P. P., Manzini, T., Morency, L.-P., & Póczos, B. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016892 · Cited by 512 · see citing works
More Than a Feeling: Learning to Grasp and Regrasp Using Vision and Touch
Calandra, R., Owens, A., Jayaraman, D., Lin, J., Yuan, W., Malik, J., Adelson, E. H., & Levine, S. (2018) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2018.2852779 · Cited by 365 · see citing works
Lee, M. A., Zhu, Y., Srinivasan, K., Shah, P., Savarese, S., Fei-Fei, L., Garg, A., & Bohg, J. (2019) · 2019 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2019.8793485 · Cited by 349 · see citing works
Self-citationLearning Factorized Multimodal Representations
Tsai, Y.-H. H., Liang, P. P., Zadeh, A., Morency, L.-P., & Salakhutdinov, R. (2018) · arXiv · doi:10.48550/arxiv.1806.06176 · Cited by 323 · see citing works
State representation learning for control: An overview
Lesort, T., Díaz-Rodríguez, N., Goudou, J.-F., & Filliat, D. (2018) · Neural Networks · doi:10.1016/j.neunet.2018.07.006 · Cited by 302 · see citing works
Visual and Somatosensory Information about Object Shape Control Manipulative Fingertip Forces
Jenmalm, P., & Johansson, R. S. (1997) · Journal of Neuroscience · doi:10.1523/jneurosci.17-11-04486.1997 · Cited by 291 · see citing works
Deep learning for tactile understanding from visual and haptic data
Gao, Y., Hendricks, L. A., Kuchenbecker, K. J., & Darrell, T. (2016) · 2016 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2016.7487176 · Cited by 264 · see citing works
Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks
Lee, M. A., Zhu, Y., Zachares, P., Tan, M., Srinivasan, K., Savarese, S., Fei-Fei, L., Garg, A., & Bohg, J. (2020) · IEEE Transactions on Robotics · doi:10.1109/tro.2019.2959445 · Cited by 241 · see citing works
Self-citationUncertainty-Aware Reinforcement Learning for Collision AvoidancePreprint
Kahn, G., Villaflor, A., Pong, V. H., Abbeel, P., & Levine, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.01182 · Cited by 228 · see citing works
Multimodal Generative Models for Scalable Weakly-Supervised Learning
Wu, M., & Goodman, N. (2018) · arXiv · doi:10.48550/arxiv.1802.05335 · Cited by 200 · see citing works
Martín-Martín, R., Lee, M. A., Gardner, R., Savarese, S., Bohg, J., & Garg, A. (2019) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros40897.2019.8968201 · Cited by 186 · see citing works
Self-citationSafe Visual Navigation via Deep Learning and Novelty Detection
Richter, C., & Roy, N. (2017) · Robotics: Science and Systems · doi:10.15607/rss.2017.xiii.064 · Cited by 160 · see citing works
Connecting Touch and Vision via Cross-Modal Prediction
Li, Y., Zhu, J.-Y., Tedrake, R., & Torralba, A. B. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01086 · Cited by 132 · see citing works
3D Shape Perception from Monocular Vision, Touch, and Shape Priors
Wang, S., Wu, J., Sun, X., Yuan, W., Freeman, W. T., Tenenbaum, J. B., & Adelson, E. H. (2018) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8593430 · Cited by 116 · see citing works
Deep Multimodal Representation Learning from Temporal Data
Yang, X., Ramesh, P. S., Chitta, R., Madhvanath, S., Bernal, E. A., & Luo, J. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.538 · Cited by 104 · see citing works
Can Autonomous Vehicles Identify, Recover From, and Adapt to Distribution Shifts?Preprint
Filos, A., Tigas, P., McAllister, R., Rhinehart, N., Levine, S., & Gal, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.14911 · Cited by 103 · see citing works
Deep Visuo-Tactile Learning: Estimation of Tactile Properties from Images
Takahashi, K., & Tan, J. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8794285 · Cited by 75 · see citing works
Learning End-to-end Multimodal Sensor Policies for Autonomous NavigationPreprint
Liu, G.-H., Siravuru, A., Prabhakar, S., Veloso, M., & Kantor, G. A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.10422 · Cited by 58 · see citing works
Learning relational object categories using behavioral exploration and multimodal perception
Sinapov, J., Schenck, C., & Stoytchev, A. (2014) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2014.6907696 · Cited by 56 · see citing works
Multimodal Sensor Fusion with Differentiable Filters
Lee, M. A., Yi, B., Martin-Martin, R., Savarese, S., & Bohg, J. (2020) · 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9341579 · Cited by 46 · cited 1× · see citing works
“While differentiable filters such as [20] can learn how to fuse multiple modalities, they were not demonstrated to recover from corrupted sensor inputs unseen during train time.”
BackgroundSelf-citationLearning tactile characterizations of object- and pose-specific grasps
Bekiroglu, Y., Detry, R. J., & Kragić, D. (2011) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2011.6094878 · Cited by 43 · see citing works
Robustness to Out-of-Distribution Inputs via Task-Aware Generative Uncertainty
McAllister, R., Kahn, G., Clune, J., & Levine, S. (2018) · IEEE International Conference on Robotics and Automation · doi:10.1109/icra.2019.8793552 · Cited by 41 · see citing works
Multimodal representation models for prediction and control from partial information
Zambelli, M., Cully, A., & Demiris, Y. (2019) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2019.103312 · Cited by 33 · see citing works
Realtime State Estimation with Tactile and Visual Sensing for Inserting a Suction-held Object
Yu, K., & Rodríguez, A. (2018) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8594077 · Cited by 29 · see citing works
Joint Inference of Kinematic and Force Trajectories with Visuo-Tactile Sensing
Lambert, A. S., Mukadam, M., Sundaralingam, B., Ratliff, N., Boots, B., & Fox, D. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8794048 · Cited by 23 · see citing works
Factorized Inference in Deep Markov Models for Incomplete Multimodal Time Series
Zhi‐Xuan, T., Soh, H., & Ong, D. C. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i06.6597 · Cited by 23 · see citing works
Strategies for multi-modal scene exploration
Bohg, J., Johnson‐Roberson, M., Björkman, M., & Kragić, D. (2010) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2010.5652967 · Cited by 21 · see citing works
Self-citationMartín-Martín, R., & Brock, O. (2017) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8206165 · Cited by 11 · see citing works
Martín-Martín, R., & Brock, O. (2017) · National Conference on Artificial Intelligence · Cited by 9 · see citing works
Multi-Modal Scene Understanding for Robotic Grasping
Bohg, J. (2011) · Cited by 7 · see citing works
Self-citationCross-modal noise compensation in audiovisual words
Baart, M., Armstrong, B. C., Martin, C. D., Frost, R., & Carreiras, M. (2017) · Scientific Reports · doi:10.1038/srep42055 · Cited by 4 · see citing works
Flagged first, then by citations
2 of the 41 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
Self-citationXue, Z., & Marculescu, R. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00256 · Cited by 134 · see citing works
Semantic anomaly detection with large language models
Elhafsi, A., Sinha, R., Agia, C., Schmerling, E., Nesnas, I. A. D., & Pavone, M. (2023) · Autonomous Robots · doi:10.1007/s10514-023-10132-6 · Cited by 101 · see citing works
Out-of-Distribution Detection for Automotive Perception
Nitsch, J., Itkina, M., Senanayake, R., Nieto, J. I., Schmidt, M. T., Siegwart, R., Kochenderfer, M. J., & Cadena, C. (2021) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc48978.2021.9564545 · Cited by 64 · see citing works
EMP: Emotion-guided Multi-modal Fusion and Contrastive Learning for Personality Traits Recognition
Wang, Y., Li, D., Funakoshi, K., & Okumura, M. (2023) · International Conference on Multimedia Retrieval · doi:10.1145/3591106.3592243 · Cited by 30 · see citing works
Non-contact multimodal indoor human monitoring systems: A survey
Nguyen, L. N., Susarla, P., Mukherjee, A., Cañellas, M. L., Casado, C. Á., Wu, X., Sílven, O., Jayagopi, D. B., & López, M. B. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102457 · Cited by 29 · see citing works
Intra- and Inter-Modal Curriculum for Multimodal Learning
Zhou, Y., Wang, X. Y., Chen, H., Duan, X., & Zhu, W. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612468 · Cited by 28 · see citing works
MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
Liang, P. P., Lyu, Y., Fan, X., Wu, Z., Cheng, Y., Wu, J., Chen, L., Wu, P., Lee, M. A., Zhu, Y., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.07502 · Cited by 85 · see citing works
“Improving robustness: These approaches modify the objective function to account for robustness to noisy [101] or missing [89, 111, 123] modalities.”
BackgroundSelf-citationResilient Legged Local Navigation: Learning to Traverse with Compromised Perception End-to-End
Zhang, C., Jin, J., Frey, J., Rudin, N., Mattamala, M., Cadena, C., & Hutter, M. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10611254 · Cited by 23 · see citing works
PoseFusion: Robust Object-in-Hand Pose Estimation with SelectLSTM
Tu, Y., Jiang, J., Li, S., Hendrich, N., Li, M., & Zhang, J. (2023) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10341688 · Cited by 14 · see citing works
Multi-Modal Perception for Soft Robotic Interactions Using Generative Models
Donato, E., Falotico, E., & Thuruthel, T. G. (2024) · IEEE International Conference on Soft Robotics (RoboSoft) · doi:10.1109/robosoft60065.2024.10522007 · Cited by 10 · see citing works
Noh, S., Lee, W., & Myung, H. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107202 · Cited by 9 · see citing works
Ichiwara, H., Ito, H., Yamamoto, K., Hiroki, M. O. R. I., & Ogata, T. (2023) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2023.3327654 · Cited by 8 · see citing works
Real- Time Grocery Packing by Integrating Vision, Tactile Sensing, and Soft Fingers
Chen, V. K., Chin, L., Choi, J., Zhang, A., & Rus, D. L. (2024) · IEEE International Conference on Soft Robotics (RoboSoft) · doi:10.1109/robosoft60065.2024.10521917 · Cited by 6 · see citing works
Versatile graceful degradation framework for bio-inspired proprioception with redundant soft sensors
Sugiyama, T., Kutsuzawa, K., Owaki, D., Almanzor, E., Fumiya Iida, A., & Hayashibe, M. (2025) · Frontiers in Robotics and AI · doi:10.3389/frobt.2024.1504651 · Cited by 4 · see citing works
Multiscale Sensor Fusion and Continuous Control with Neural CDEs
Singh, S., Ramirez, F. M., Varley, J., Zeng, A., & Sindhwani, V. (2022) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros47612.2022.9982210 · Cited by 3 · see citing works
Robot imitation from multimodal observation with unsupervised cross-modal representation
Xu, X., You, M., Zhou, H., & He, B. (2024) · Robotica · doi:10.1017/s0263574724000626 · Cited by 1 · see citing works
AugInsert: Learning Robust Visual-Force Policies via Data Augmentation for Object Assembly Tasks
Diaz, R., Imdieke, A., Veeriah, V., & Desingh, K. (2025) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros60139.2025.11247210 · Cited by 1 · see citing works
Masked Imitation Learning: Discovering Environment-Invariant Modalities in Multimodal Demonstrations
Hao, Y., Wang, R., Cao, Z., Wang, Z., Cui, Y., & Sadigh, D. (2023) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10341728 · Cited by 1 · see citing works
Precision Peg-In-Hole Assembly Based on Multiple Sensations and Cross-Modal Prediction
Liu, R., Li, A., Yang, X., & Lou, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-13841-6_49 · Cited by 1 · see citing works
Cross-Modal Environment Self-Adaptation During Object Recognition in Artificial Cognitive SystemsPreprint
Miralles, D., Garrofé, G., Parés, C., González, A., Serra, G., Soto, A., de Beeck, H. O., Masson, H. L., & Sevillano, X. (2021) · Research Square · doi:10.21203/rs.3.rs-754574/v1
Multi-modal self-adaptation during object recognition in an artificial cognitive system
Miralles, D., Garrofé, G., Parés, C., González, A., Serra, G., Soto, A., Sevillano, X., de Beeck, H. O., & Masson, H. L. (2022) · Scientific Reports · doi:10.1038/s41598-022-07424-9
Zhao, Z., Zhao, C., Chen, K., Hu, X., Du, Y., & Ji, Y. (2025) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc60802.2025.11423007
F usion S ense : Tri-Stage Near-Sensor Learning for Runtime-Adaptive Multimodal Edge Intelligence
Yun, S., Masukawa, R., Na, M., Oh, H., Yamaguchi, Y., Huang, W. M., Jeong, S., & Imani, M. (2026) · International Symposium on Low Power Electronics and Design (ISLPED) · doi:10.1145/3816440.3818531
- Authors
- Lee, M. A., Tan, M., Zhu, Y., & Bohg, J.
- Affiliations
- Stanford University (US)
- The University of Texas at Austin (US)
- Venue
- 2021 IEEE International Conference on Robotics and Automation (ICRA)
- Publisher
- IEEE
- Published
- 2021-05-30
- Pages
- 909-916
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W3107906842
- Cited by
- 24
- References
- 41
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 5 · 2023: 7 · 2022: 3 · 2021: 4
- FWCI
- 2.19
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- National Science Foundation (1955523)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Keywords
- multimodal sensor fusion, self-supervised learning, robotic manipulation, robust reinforcement learning, state representation learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
