Interactive Perception: Leveraging Action in Perception and Perception in Action
Open accessThe DOI matches its Crossref record; 119 of the 119 references it cites were found in a registry or an index.
Abstract
Recent approaches in robot perception follow the insight that perception is facilitated by interaction with the environment. These approaches are subsumed under the term Interactive Perception (IP). This view of perception provides the following benefits. First, interaction with the environment creates a rich sensory signal that would otherwise not be present. Second, knowledge of the regularity in the combined space of sensory data and action parameters facilitates the prediction and interpretation of the sensory signal. In this survey, we postulate this as a principle for robot perception and collect evidence in its support by analyzing and categorizing existing work in this area. We also provide an overview of the most important applications of IP. We close this survey by discussing remaining open questions. With this survey, we hope to help define the field of Interactive Perception and to provide a valuable resource for future research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Such structure is called Action Perception Regularity [1], and is believed to be the key for IP to reveal richer signals, which is impossible for passive and one-shot perception.”
From Interactive Perception for Deformable Object Manipulation (2024) · cites this paper as [1]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Interactive Perception (IP) exploits various types of forceful interactions with the environment to facilitate perception [1].”
From Interactive Perception for Deformable Object Manipulation (2024) · cites this paper as [1]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472 · cited 1×
“t given only a single input image. A method has to cope with occlusions, different lighting conditions, scale of the images, just to name a few. State-of-the-art approaches in Computer Vision as e.g. [107, 108] require enormous amounts of training data to handle these variations. Interactive Perception approaches allo…”
MethodMicrosoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 1×
“These three examples illustrate that biological perception and perceptually-guided behavior intrinsically rely on active exploration and knowledge of the relation between action and sensory response.”
BackgroundFully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
The pascal visual object classes (VOC) challenge
Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2010) · International Journal of Computer Vision · doi:10.1007/s11263-009-0275-4 · Cited by 19,599 · cited 2×
“The most successful approaches learn models from data sets that contain hundreds of thousands of semantically annotated static images, such as Pascal VOC [6], ImageNet [7] or Microsoft COCO [8].”
BackgroundContour Detection and Hierarchical Image Segmentation
Arbeláez, P., Maire, M., Fowlkes, C., & Malik, J. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.161 · Cited by 5,454 · cited 1×
“This emphasizes the blurred boundary which is traditionally drawn between manipulation and perception.”
BackgroundLaValle, S. M. (2006) · Cambridge University Press eBooks · doi:10.1017/cbo9780511546877 · Cited by 5,211
Computational modelling of visual attention
Itti, L., & Koch, C. (2001) · Nature Reviews Neuroscience · doi:10.1038/35058500 · Cited by 4,733 · cited 1×
“of freedom [17]–[19] and models of visual attention [20], [21] that these active vision systems could use for guiding their gaze.”
BackgroundReinforcement learning in robotics: A survey
Kober, J., Bagnell, J. A., & Peters, J. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913495721 · Cited by 3,160 · cited 2×
“Reinforcement learning [126] is also a common choice to learn a policy for action selection under these complex conditions.”
Background
Research works printed but not linked: 14
- allen, 1985, Two sensors are better than one: Example of integration of vision and touch, Proc Int Symp Robot Res
- goldberg, 1984, Active touch and robot perception, Cognition Brain Theory, 7, 199
- bajscy, 1988, Shape from touch, Advances in Automation and Robotics
- sandini, 1993, Vision During Action
- natale, 0, Learning haptic representation of objects, Proc Int Conf Intell Manipulation Grasping
- noë, 2004, Action in Perception
- gibson, 1966, The Senses Considered as Perceptual Systems
- katz, 0, Interactive segmentation of articulated objects in 3D, Mobile Manipulation Workshop Proc Int Conf Robot Automat
- singh, 2004, Predictive state representations: A new theory for modeling dynamical systems, Proceedings of the 20th conference on Uncertainty in artificial intelligence, 512
- gibson, 1979, The Ecological Approach to Visual Perception
- katz, 2014, Interactive Perception of Articulated Objects, 79, 301
- natale, 0, A sensitive approach to grasping, Proc 6th Int Workshop Epigenetic Robot, 87
- pahlavan, 1993, Active vision as a methodology, Active Perception, 19
- torres-jara, 2005, Tapping into touch, Proc 5th Int Workshop on Epigenetic Robotics, 79
Cited by
Trends and challenges in robot manipulation
Billard, A., & Kragic, D. (2019) · Science · doi:10.1126/science.aat8414 · Cited by 993
“Today, it is also recognized that perception and control are tightly coupled, and the field of interactive perception (63) regardsmanipulation as ameans to perceive and perception as a means to achieve better manipulation.”
BackgroundDeep Reinforcement Learning: An Overview
Li, Y. (2017) · arXiv · doi:10.48550/arxiv.1701.07274 · Cited by 762
Neural Fields in Visual Computing and Beyond
Xie, Y., Takikawa, T., Saito, S., Litany, O., Yan, S., Khan, N., Tombari, F., Tompkin, J., sitzmann, V., & Sridhar, S. (2022) · Computer Graphics Forum · doi:10.1111/cgf.14505 · Cited by 484
Lee, M. A., Zhu, Y., Srinivasan, K., Shah, P., Savarese, S., Fei-Fei, L., Garg, A., & Bohg, J. (2019) · 2019 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2019.8793485 · Cited by 349
“There is compelling evidence that the interdependence and concurrency of different sensory streams aid perception and manipulation [9, 18, 31].”
BackgroundSelf-citationState representation learning for control: An overview
Lesort, T., Díaz-Rodríguez, N., Goudou, J.-F., & Filliat, D. (2018) · Neural Networks · doi:10.1016/j.neunet.2018.07.006 · Cited by 301
“This is for instance the case of representation learning from different temporal signals in [Duan, 2017, Bohg et al., 2017].”
BackgroundChallenges and Outlook in Robotic Manipulation of Deformable Objects
Zhu, J., Cherubini, A., Dune, C., Navarro-Alarcon, D., Alambeigi, F., Berenson, D., Ficuciello, F., Harada, K., Kober, J., Li, X., et al. (2022) · IEEE Robotics & Automation Magazine · doi:10.1109/mra.2022.3147415 · Cited by 265
“A promising direction to deal with occlusion and noise is using active/interactive perception [38], [39].”
BackgroundMaking Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks
Lee, M. A., Zhu, Y., Zachares, P., Tan, M., Srinivasan, K., Savarese, S., Fei-Fei, L., Garg, A., & Bohg, J. (2020) · IEEE Transactions on Robotics · doi:10.1109/tro.2019.2959445 · Cited by 238
Self-citationLi, C. (2019) · Reinforcement Learning for Cyber-Physical Systems · doi:10.1201/9781351006620-6 · Cited by 188
SE3-nets: Learning rigid body motion using deep neural networks
Byravan, A., & Fox, D. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989023 · Cited by 236
“interact with objects for manipulation or object exploration tasks, as summarized in a recent survey [12].”
BackgroundAdaptive Rotated Convolution for Rotated Object Detection
Pu, Y., Wang, Y., Xia, Z., Han, Y., Wang, Y., Gan, W., Wang, Z., Song, S., & Huang, G. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00606 · Cited by 195
“In particular, in Embodied AI [11, 77, 50, 80, 13] tasks, as the agents need to explore and interact with the environment [4, 3, 42], the captured images may contain an object from arbitrary view-points [15, 59].”
Background(2021) · Journal of Machine Learning Research · Cited by 224
“In interactive perception [33], the robot physically in-”
BackgroundObject Handovers: A Review for Robotics
Ortenzi, V., Cosgun, A., Pardi, T., Chan, W. P., Croft, E., & Kulic, D. (2021) · IEEE Transactions on Robotics · doi:10.1109/tro.2021.3075365 · Cited by 198
Self-supervised 6D Object Pose Estimation for Robot Manipulation
Deng, X., Xiang, Y., Mousavian, A., Eppner, C., Bretl, T., & Fox, D. (2020) · 2020 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra40945.2020.9196714 · Cited by 207
“By incorporating these techniques into the interactive robot perception framework [31], we can obtain accurate 6D poses of objects to improve the system.”
MethodA survey of robot manipulation in contact
Suomalainen, M., Karayiannidis, Y., & Kyrki, V. (2022) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2022.104224 · Cited by 165
“Finally, when considering vision in the loop, there is a survey of the interplay between vision and touch [16]; for this reason, this survey will not consider vision-based manipulation either, and we will focus on works where the sole feedback, if any, is force.”
BackgroundProximity Perception in Human-centered Robotics: A Survey on Sensing Systems and Applications
Navarro, S. E., Muhlbacher-Karrer, S., Alagi, H., Zangl, H., Koyama, K., Hein, B., Duriez, C., & Smith, J. R. (2022) · IEEE Transactions on Robotics · doi:10.1109/tro.2021.3111786 · Cited by 162
“The active perception principle has been studied in many robotics research papers, as discussed in a survey by Bohg et al. [139].”
BackgroundLessons from the Amazon Picking Challenge: Four Aspects of Building Robotic Systems
Eppner, C., Höfer, S., Jonschkowski, R., Martín-Martín, R., Sieverling, A., Wall, V., & Brock, O. (2016) · Robotics: Science and Systems XII · doi:10.15607/rss.2016.xii.036 · Cited by 124
Self-citationInteractive, Collaborative Robots: Challenges and Opportunities
Kragić, D., Gustafson, J., Karaoğuz, H., Jensfelt, P., & Krug, R. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/3 · Cited by 126
3D Shape Perception from Monocular Vision, Touch, and Shape Priors
Wang, S., Wu, J., Sun, X., Yuan, W., Freeman, W. T., Tenenbaum, J. B., & Adelson, E. H. (2018) · 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8593430 · Cited by 116
Trends of Human-Robot Collaboration in Industry Contexts: Handover, Learning, and Metrics
Castro, A., da Silva, F. M. T. P., & Santos, V. M. F. (2021) · Sensors · doi:10.3390/s21124113 · Cited by 133
“Therefore, the deployment of robotic systems that can decompose the main task into primitive elements and learn to plan complex movement sequences would be a valuable contribution.”
BackgroundEmbodied Intelligence: A Synergy of Morphology, Action, Perception and Learning
Liu, H., Guo, D., & Cangelosi, A. (2025) · ACM Computing Surveys · doi:10.1145/3717059 · Cited by 200
“Various perception modalities, such as vision, hearing, and touch, are also employed in active perception tasks [17, 112].”
Mechanical Search: Multi-Step Retrieval of a Target Object Occluded by Clutter
Danielczuk, M., Kurenkov, A., Balakrishna, A., Matl, M., Wang, D., Martin-Martin, R., Garg, A., Savarese, S., & Goldberg, K. (2019) · 2019 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2019.8794143 · Cited by 102
“There has also been work on optimizing camera positioning for improving visual recognition (i.e., active perception [1, 2]) and embodied interactions to explore (i.e., interactive perception [4, 14]).”
BackgroundZhao, Z., Liu, H., Chen, H., Hu, J., & Guo, H. (2019) · Mechanical Systems and Signal Processing · doi:10.1016/j.ymssp.2019.01.005 · Cited by 103
Deep Object-Centric Representations for Generalizable Robot Learning
Devin, C., Abbeel, P., Darrell, T., & Levine, S. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8461196 · Cited by 86
GapFlyt: Active Vision Based Minimalist Structure-Less Gap Detection For Quadrotor Flight
Sanket, N. J., Singh, C. D., Ganguly, K., Fermuller, C. M., & Aloimonos, Y. (2018) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2018.2843445 · Cited by 105
Advances in Inference and Representation for Simultaneous Localization and Mapping
Rosen, D. M., Doherty, K., Espinoza, A. T., & Leonard, J. J. (2021) · Annual Review of Control Robotics and Autonomous Systems · doi:10.1146/annurev-control-072720-082553 · Cited by 106
“Increasingly often, these tasks include active or interactive perception (83), intelligent exploration, manipulation, and learning as a task in itself.”
BackgroundDitto: Building Digital Twins of Articulated Objects from Interaction
Jiang, Z., Hsu, C.-C., & Zhu, Y. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00553 · Cited by 92
Learning Affordance Grounding from Exocentric Images
Luo, H., Zhai, W., Zhang, J., Cao, Y. H., & Tao, D. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00229 · Cited by 154
“Perceiving and reasoning about possible interactions in local regions of objects is the key to the shift from passive perception systems to embodied intelligence systems that actively interact with and perceive their environment [1, 33, 34, 38].”
BackgroundFully body visual self-modeling of robot morphologies
Chen, B., Kwiatkowski, R., Vondrick, C., & Lipson, H. (2022) · Science Robotics · doi:10.1126/scirobotics.abn1944 · Cited by 101
From screens to scenes: A survey of embodied AI in healthcare
Liu, Y., Cao, X., Chen, T., Jiang, Y., You, J., Wu, M., Wang, X., Feng, M., Jin, Y., & Chen, J. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103033 · Cited by 92
Learning to Singulate Objects Using a Push Proposal Network
Eitel, A., Hauff, N., & Burgard, W. (2019) · Springer proceedings in advanced robotics · doi:10.1007/978-3-030-28619-4_32 · Cited by 77
Bhardwaj, M., Sundaralingam, B., Mousavian, A., Ratliff, N., Fox, D., Ramos, F. T., & Boots, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.13542 · Cited by 72
“Real-world robot manipulation can greatly benefit from realtime perception-driven feedback control [4, 21].”
BackgroundSalientDSO: Bringing Attention to Direct Sparse Odometry
Liang, H.-J., Sanket, N. J., Fermuller, C. M., & Aloimonos, Y. (2019) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2019.2900980 · Cited by 75
ScrewNet: Category-Independent Articulation Model Estimation From Depth Images Using Screw Theory
Jain, A., Lioutikov, R., Chuck, C., & Niekum, S. (2021) · 2021 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48506.2021.9561132 · Cited by 61
“Interactive perception (IP): IP approaches leverage the robot’s interaction with the objects for generating a rich perceptual signal for robust articulation model estimation [18–20].”
BackgroundVAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated ObjectsPreprint
Wu, R., Zhao, Y. L., Mo, K., Guo, Z., Wang, Y., Wu, T., Fan, Q., Chen, X., Guibas, L. J., & Dong, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.14440 · Cited by 104
“112 Learning Perception from Interaction augments the tremendously successful learning paradigm 113 using offline curated datasets [48, 50, 21, 22] by allowing learning agents to collect online active data 114 samples, which are more task-aware and learning-efficient, during navigation [73, 16], recognition [74, 115 7…”
MethodObject Finding in Cluttered Scenes Using Interactive Perception
Novkovic, T., Pautrat, R., Furrer, F., Breyer, M., Siegwart, R., & Nieto, J. (2020) · 2020 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra40945.2020.9197101 · Cited by 68
“actions and their sensory response [9].”
A novel method of urban landscape perception based on biological vision process
Cao, Y., Yang, P., Xu, M., Li, M., Li, Y., & Guo, R. (2024) · Landscape and Urban Planning · doi:10.1016/j.landurbplan.2024.105246 · Cited by 64
Object Discovery in Videos as Foreground Motion Clustering
Xie, C., Yu, X., Harchaoui, Z., & Fox, D. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01023 · Cited by 77
“In the interactive perception setting [8], the robot can even interact with the environment to discover objects by touching or pushing objects.”
BackgroundScrewNet: Category-Independent Articulation Model Estimation From Depth Images Using Screw Theory
Jain, A., Lioutikov, R., Chuck, C., & Niekum, S. (2020) · arXiv · doi:10.48550/arxiv.2008.10518 · Cited by 63
“Interactive perception (IP): IP approaches leverage the robot’s interaction with the objects for generating a rich perceptual signal for robust articulation model estimation [18–20].”
BackgroundLessons from the Amazon Picking Challenge: Four Aspects of Building Robotic Systems
Eppner, C., Höfer, S., Jonschkowski, R., Martín-Martín, R., Sieverling, A., Wall, V., & Brock, O. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/676 · Cited by 62
“led to advances in robot perception [6, 27, 34].”
Self-citationAffordances in Robotic Tasks -- A SurveyPreprint
Ardón, P., Pairet, È., Lohan, K. S., Ramamoorthy, S., & Petrick, R. P. A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.07400 · Cited by 57
“In this category, the relation is built from object target features rather than object categories [13, 67, 88].”
BackgroundSrivastava, S., Li, C., Lingelbach, M., Martín-Martín, R., Xia, F., Vainio, K., Lian, Z., Gokmen, C., Buch, S., Liu, C. K., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.03332 · Cited by 55
“…synchronized ground-truth information of the object being observed, its state and full shape. we expect the dataset to be useful to study and predict human gaze attention, and to develop new embodied AI algorithms for active [80, 81] and interactive perception [82]. and RL with motion primitives.”
Murali, P. K., Dutta, A., Gentner, M., Burdet, E., Dahiya, R., & Kaboli, M. (2022) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2022.3150045 · Cited by 56
A survey of visual navigation: From geometry to embodied AI
Zhang, T., Hu, X., Xiao, J., & Zhang, G. (2022) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2022.105036 · Cited by 68
Intelligent Robotic Perception Systems
Premebida, C., Ambrus, R., & Marton, Z.-C. (2019) · Applications of Mobile Robots · doi:10.5772/intechopen.79742 · Cited by 52
“The argument for embodied learning and grounding of new information evolved, considering the works of Steels and Brooks [ 38 ] and Vernon [ 39 ], and more recently in [ 40 ], robot perception involves planning and interactive segmentation.”
Human-centered AI and robotics
Doncieux, S., Chatila, R., Straube, S., & Kirchner, F. (2022) · AI Perspectives · doi:10.1186/s42467-021-00014-x · Cited by 46
“Pushing an object and observing what has moved clearly shows object boundaries without the need to have a large database of similar objects, this is called interactive perception [32].”
Unfolding the Literature: A Review of Robotic Cloth Manipulation
Longhini, A., Wang, Y., Garcia-Camacho, I., Blanco–Mulero, D., Moletta, M., Welle, M. C., Alenyà, G., Yin, H., Erickson, Z., Held, D., et al. (2024) · Annual Review of Control Robotics and Autonomous Systems · doi:10.1146/annurev-control-022723-033252 · Cited by 73
Wang, G., Zhang, C., Liu, S., Zhang, Y., Zhang, Y., & Wang, L. (2025) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2025.06.014 · Cited by 49
Robotic Grasping of Fully-Occluded Objects using RF Perception
Boroushaki, T., Leng, J., Clester, I., Rodríguez, A., & Adib, F. (2021) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48506.2021.9560956 · Cited by 50
Toward Grounded Commonsense Reasoning
Kwon, M., Hu, H., Myers, V., Karamcheti, S., Dragan, A. D., & Sadigh, D. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10611218 · Cited by 53
“In this work, we do not focus on particular human preferences, though the normative responses of LLMs could be fine-tuned for particular applications. c) Active Perception: When robots must use common-sense reasoning like humans, active information gathering may be important [32].”
BackgroundAutonomous functional movements in a tendon-driven limb via limited experience
Marjaninejad, A., Urbina-Meléndez, D., Cohn, B. A., & Valero‐Cuevas, F. J. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0029-0 · Cited by 42
What foundation models can bring for robot learning in manipulation: A survey
Li, D., Jin, Y., Sun, Y.-H., Yong, A., Yu, H., Shi, J., Hao, X., Peng, H., Liu, H., Li, X., et al. (2025) · The International Journal of Robotics Research · doi:10.1177/02783649251390579 · Cited by 51
“The primary methods for physical parameter identification include estimation from interaction (Seker and Kroemer (2024); Bohg et al. (2017); Xu et al. (2019)), estimation from demonstrations (Torne et al. (2024)), and estimation from observations using foundation models (Gao et al. (2023)).”
InfluentialMotion-Based Object Segmentation Based on Dense RGB-D Scene Flow
Shao, L., Shah, P., Dwaracherla, V., & Bohg, J. (2018) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2018.2856525 · Cited by 44
“Methods that exploit physical interaction to ease perception are often referred to as performing Interactive Perception (IP) [1].”
BackgroundSelf-citationOcclusion-Aware Search for Object Retrieval in Clutter
Bejjani, W., Agboh, W. C., Doğar, M. R., & Leonetti, M. (2021) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9636230 · Cited by 50
Differentiable Physics Simulation of Dynamics-Augmented Neural Objects
Cleac’h, S. L., Yu, H.-X., Guo, M., Howell, T. A., Gao, R., Wu, J., Manchester, Z. R., & Schwager, M. (2023) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2023.3257707 · Cited by 39
“In simulation, we leverage interactive perception [38] to identify the mass scaling factor α from (2), Fig.”
MethodCoupled recursive estimation for online interactive perception of articulated objects
Martín-Martín, R., & Brock, O. (2019) · The International Journal of Robotics Research · doi:10.1177/0278364919848850 · Cited by 51
“Perception that requires or exploits interaction is called interactive perception (IP) (Bohg et al. 2017).”
BackgroundInfluentialSelf-citationVisuomotor Mechanical Search: Learning to Retrieve Target Objects in Clutter
Kurenkov, A., Taglic, J., Kulkarni, R., Dominguez-Kuhne, M., Garg, A., Martín-Martín, R., & Savarese, S. (2020) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9341545 · Cited by 51
“Interactive perception approaches [10] have considered the perceptual problem of segmenting images of a pile of objects into coherent components that move together [11–14].”
BackgroundIn-Hand Object-Dynamics Inference Using Tactile Fingertips
Sundaralingam, B., & Hermans, T. (2021) · IEEE Transactions on Robotics · doi:10.1109/tro.2020.3043675 · Cited by 37
Manipulation for self-Identification, and self-Identification for better manipulation
Hang, K., Bircher, W. G., Morgan, A. S., & Dollar, A. M. (2021) · Science Robotics · doi:10.1126/scirobotics.abe1321 · Cited by 41
“Being a core component of such procedures, interactive perception is often required for the robot to perceive and analyze itself and the world resulted from its own actions (39, 40).”
BackgroundAct the Part: Learning Interaction Strategies for Articulated Object Part Discovery
Gadre, S. Y., Ehsani, K., & Song, S. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01546 · Cited by 65
“Our approach builds on existing work in interactive perception [6], where visual tasks are solved using agent intervention.”
Four aspects of building robotic systems: lessons from the Amazon Picking Challenge 2015
Eppner, C., Höfer, S., Jonschkowski, R., Martín-Martín, R., Sieverling, A., Wall, V., & Brock, O. (2018) · Autonomous Robots · doi:10.1007/s10514-018-9761-2 · Cited by 39
“More recently, advances in robot perception were achieved by combining interaction and perception (Bohg et al. 2017; Katz and Brock 2011; Martín-Martín and Brock 2014), and by integrating interdependent perceptual tasks that were traditionally tackled in isolation (Martín-Martín et al. 2016).”
BackgroundSelf-citationActuated Palms for Soft Robotic Hands: Review and Perspectives
Pozzi, M., Malvezzi, M., Prattichizzo, D., & Salvietti, G. (2023) · IEEE/ASME Transactions on Mechatronics · doi:10.1109/tmech.2023.3328944 · Cited by 43
“Asafuture perspective,theactivecontrolofadditionalDoFsinthepalm, combinedwithanincreasedsensingcapability,couldenable interactiveperceptionstrategies[82]inwhich,forexample, thegripperactivelycollectstactileinformationontheshapeof objectsandexploitsittowraparoundtheminacontrolledand intentionalway.”
Learning 3D Dynamic Scene Representations for Robot ManipulationPreprint
Xu, Z., He, Z., Wu, J., & Song, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.01968 · Cited by 57
“Interactive perception is about perception facilitated by interaction with the environment [29].”
An Interactive Perception Method for Warehouse Automation in Smart Cities
Liu, H., Deng, Y., Guo, D., Fang, B., Sun, F., & Yang, W. (2020) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2020.2969680 · Cited by 34
“This method attempts to conduct physical interactions to produce new sensory information to change the environment and, therefore, simplifies the grasp tasks [19].”
MethodPrecise Multi-Modal In-Hand Pose Estimation using Low-Precision Sensors for Robotic Assembly
von Drigalski, F., Hayashi, K., Huang, Y., Yonetani, R., Hamaya, M., Tanaka, K., & Ijiri, Y. (2021) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48506.2021.9561222 · Cited by 39
“These approaches are also known as active sensing [32] instead of ”in-hand pose estimation”.”
BackgroundRoboNinja: Learning an Adaptive Cutting Policy for Multi-Material Objects
Xu, Z., Xian, Z., Lin, X., Chi, C., Huang, Z., Gan, C., & Song, S. (2023) · Robotics: Science and Systems · doi:10.15607/rss.2023.xix.046 · Cited by 45
Visual Identification of Articulated Object Parts
Zeng, V., Lee, T. E., Liang, J., & Kroemer, O. (2021) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9636054 · Cited by 54
“Interactive Perception (IP) is a well-known approach to this problem [7].”
BackgroundHurtado, E. G. (2019) · IntechOpen eBooks · doi:10.5772/intechopen.74181 · Cited by 38
Development and Evaluation of a Learning-Based Model for Real-Time Haptic Texture Rendering
Heravi, N., Culbertson, H., Okamura, A. M., & Bohg, J. (2024) · IEEE Transactions on Haptics · doi:10.1109/toh.2024.3382258 · Cited by 35
“Current methodologies for haptic texture rendering exist, but they usually develop one model per texture, resulting in low scalability.”
Self-citationHand-Eye Calibration of Surgical Instrument for Robotic Surgery Using Interactive Manipulation
Zhong, F., Wang, Z., Chen, W., He, K., Wang, Y., & Liu, Y. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.2967685 · Cited by 37
“additional sensory feedback for task-relevant input, which is otherwise not available, is referred to as the interactive perception [34].”
BackgroundVideo Object Segmentation-based Visual Servo Control and Object Depth Estimation on a Mobile Robot
Griffin, B., Florence, V., & Corso, J. J. (2020) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv45572.2020.9093335 · Cited by 39
“asset for robot perception is that explicit actions can be taken to improve sensing and understanding of the environment. Accordingly, Active Perception (AP) [4], [3] and Interactive Perception (IP) [6] are two areas of research that exploit robot-specific capabilities to improve perception. Compared to structure from…”
BackgroundBioinspired Embodiment for Intelligent Sensing and Dexterity in Fine Manipulation: A Survey
Liu, Y., Li, Z., Liu, H., Kan, Z., & Xu, B. (2020) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2020.2971643 · Cited by 32
“4) Interactive Perception (IP) Mode: The IP mode is inspired from the human perception behaviors [35], which leverages”
BackgroundEmbodied AI: A Survey on the Evolution from Perceptive to Behavioral Intelligence
Yifan, C., Wei, M., Wang, X., Liu, Y., Wang, J., Song, H., Ma, L., Di, D., Sun, C., Liu, K., et al. (2025) · SmartBot · doi:10.1002/smb2.70003 · Cited by 43
Contact-based in-hand pose estimation using Bayesian state estimation and particle filtering
von Drigalski, F., Taniguchi, S., Lee, R., Matsubara, T., Hamaya, M., Tanaka, K., & Ijiri, Y. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9196640 · Cited by 31
“Many studies have proposed using multiple sensors [18] and sensor fusion.”
BackgroundAdvancing robotic assembly in construction: Innovations, challenges, and opportunities
Chen, Z., & Adel, A. (2025) · Automation in Construction · doi:10.1016/j.autcon.2025.106370 · Cited by 37
Survei Tingkat Persepsi Siswa Terhadap Konsep Pendidikan Jasmani Di Sekolah Menengah Atas
Fitron, M., & Mu’arifin, M. (2022) · Sport Science and Health · doi:10.17977/um062v2i52020p264-271 · Cited by 26
Grounded Affordance from Exocentric View
Luo, H., Zhai, W., Zhang, J., Cao, Y. H., & Tao, D. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01962-z · Cited by 40
“Perceiving and reasoning about possible interactions in local regions of objects is the key to the shift from passive perception systems to embodied intelligence systems that actively interact with and perceive their environment (Bohg et al., 2017; Nagarajan & Grauman, 2020).”
Structuring of tactile sensory information for category formation in robotics palpation
Scimeca, L., Maiolino, P., Bray, E., & Fumiya Iida, A. (2020) · Autonomous Robots · doi:10.1007/s10514-020-09931-y · Cited by 29
Behavior Is Everything: Towards Representing Concepts with Sensorimotor Contingencies
Hay, N., Stark, M. M., Schlegel, A. A., Wendelken, C., Park, D., Purdy, E., Silver, T., Phoenix, D. S., & George, D. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11547 · Cited by 27
Mechanical Search on Shelves using Lateral Access X-RAY
Huang, H., Dominguez-Kuhne, M., Satish, V., Danielczuk, M., Sanders, K., Ichnowski, J., Lee, A., Angelova, A., Vanhoucke, V., & Goldberg, K. Y. (2021) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9636629 · Cited by 36
“Recent work in interactive perception and mechanical search [3, 8, 19, 25, 26, 34, 41] influences our work most strongly.”
BackgroundLearning an Action-Conditional Model for Haptic Texture Generation
Heravi, N., Yuan, W., Okamura, A. M., & Bohg, J. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9197447 · Cited by 28
“Humans are conditioned to expect the sense of weight, hardness, deformability, texture, and slipperiness when interacting with an object [1, 2, 3, 4], an experience that does not fully exist in commercially available VR systems.”
Self-citationSelf-supervised Transfer Learning for Instance Segmentation through Physical Interaction
Eitel, A., Hauff, N., & Burgard, W. (2019) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros40897.2019.8967915 · Cited by 25
“resolve perception ambiguities via interaction [6], [7] and combines it with the idea of self-supervised learning.”
BackgroundDesingh, K., Lu, S., Opipari, A. W., & Jenkins, O. C. (2019) · Science Robotics · doi:10.1126/scirobotics.aaw4523 · Cited by 31
Ottenhaus, S., Renninghoff, D., Grimm, R., Ferreira, F., & Asfour, T. (2019) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/humanoids43949.2019.9035002 · Cited by 27
“tive Perception (IP) rich sensory data can be obtained [6].”
BackgroundDeep Neural Object Analysis by Interactive Auditory Exploration with a Humanoid Robot
Eppe, M., Kerzel, M., Strahl, E., & Wermter, S. (2018) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8593838 · Cited by 24
MQA: Answering the Question via Robotic Manipulation
Deng, Y., Guo, D., Guo, X., Zhang, N., Liu, H., & Sun, F. (2021) · Robotics: Science and Systems · doi:10.15607/rss.2021.xvii.044 · Cited by 27
“In the real-world environment, the perception should never be passive but an active process [4][7].”
SAGCI-System: Towards Sample-Efficient, Generalizable, Compositional, and Incremental Robot Learning
Lv, J., Yu, Q., Shao, L., Liu, W., Xu, W., & Lu, C. (2022) · 2022 International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra46639.2022.9811859 · Cited by 22
Active end-effector pose selection for tactile object recognition through Monte Carlo tree search
Zhang, M. M., Atanasov, N., & Daniilidis, K. (2017) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8206161 · Cited by 24
“Our work is closer to interactive perception, which physically contacts the environment [13].”
Ditto in the House: Building Articulation Models of Indoor Scenes through Interactive Perception
Hsu, C.-C., Jiang, Z., & Zhu, Y. (2023) · 2023 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48891.2023.10161431 · Cited by 21
“[18] is the pioneer work that introduces interactive perception [2] for articulation model estimation.”
BackgroundGrimm, R., Grotz, M., Ottenhaus, S., & Asfour, T. (2021) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48506.2021.9560889 · Cited by 23
“The survey in [11] defines IP approaches, as approaches using sensor data from multiple time frames and utilizing forceful interactions with the environment (e.”
BackgroundEnactive self: A study of engineering perspectives to obtain the sensorimotor self through enaction
Lanillos, P., Dean‐Leon, E., & Cheng, G. (2017) · IEEE International Conference on Development and Learning (ICDL) · doi:10.1109/devlrn.2017.8329790 · Cited by 24
“in contrast to other paradigms, such as active perception or interactive perception [48] arguing that real adaptability emerge from the mastery of inter-modality contingency patterns during the self-perception exploration stage [29], and executing current machine learning algorithms during interaction is not enough fo…”
ResultWu, Y., Ren, X., Zhou, H., Wang, Y., & Yi, X. (2020) · IEEE Access · doi:10.1109/access.2020.2981408 · Cited by 25
“The kind of methods have a long operating cycle, low flexibility, and highly depends on domain experts’ knowledge [14], [15].”
BackgroundSiddiqui, M. S., Coppola, C., Solak, G., & Jamone, L. (2021) · Frontiers in Robotics and AI · doi:10.3389/frobt.2021.703869 · Cited by 21
“In addition, the concept of active perception (Bajcsy et al., 2018), or interactive perception (Bohg et al., 2017), is particularly relevant in this case, because to collect useful tactile information the robot should perform relevant actions (Seminara et al., 2019), e.g. a controlled manual…”
BackgroundEstimating Mass Distribution of Articulated Objects using Non-prehensile ManipulationPreprint
Kumar, K. N., Essa, I. A., Ha, S., & Liu, C. K. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.03964 · Cited by 22
Object and relation centric representations for push effect prediction
Tekden, A. E., Erdem, A., Erdem, E., Asfour, T., & Uğur, E. (2024) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2024.104632 · Cited by 23
“In these cases, the robot can improve its perception by actions [62].”
BackgroundOpening a lockbox through physical exploration
Baum, M., Bernstein, M. A., Martín-Martín, R., Höfer, S., Kulick, J., Toussaint, M., Kacelnik, A., & Brock, O. (2017) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/humanoids.2017.8246913 · Cited by 20
“The need for active exploration in robotic manipulation stems from the fact that not all object properties can be inferred from passive observation alone [5].”
BackgroundSelf-citationLeverage Interactive Affinity for Affordance Learning
Luo, H., Zhai, W., Zhang, J., Cao, Y. H., & Tao, D. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00658 · Cited by 31
“Fini = Upsample(Linear(ci, c ′ )(Xini)), i ∈ [1, 3], (8)”
BackgroundIncremental Learning of Object Models From Natural Human–Robot Interactions
Azagra, P., Civera, J., & Murillo, A. C. (2020) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2020.2980246 · Cited by 20
“Bohg et al. [6] presented a survey on interactive perception and how it can be leveraged for robotic actions, with specific references to interactive object modeling.”
BackgroundLearning From Demonstration Based on Environmental Constraints
Li, X., & Brock, O. (2022) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2022.3196096 · Cited by 18
“Second, maintaining contact leverages the idea of Interactive Perception [12] to exploit the correlation between forceful interaction and contact-changing events, which indicates the TABLE I COMPARISON OF SUCCESS RATES ON ALL LOCKS”
BackgroundInfluentialSelf-citationDREAM Architecture: a Developmental Approach to Open-Ended Learning in RoboticsPreprint
Doncieux, S., Bredèche, N., Goff, L. K. L., Girard, B., Coninx, A., Sigaud, O., Khamassi, M., Díaz-Rodríguez, N., Filliat, D., Hospedales, T. M., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.06223 · Cited by 20
“Both steps are based on the interactive perception paradigm [67].”
Building Affordance Relations for Robotic Agents - A Review
Ardón, P., Pairet, È., Lohan, K. S., Ramamoorthy, S., & Petrick, R. P. A. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/590 · Cited by 17
Metadata
- Authors
- Bohg, J., Hausman, K., Sankaran, B., Brock, O., Kragic, D., Schaal, S., & Sukhatme, G. S.
- Affiliations
- Max Planck Institute for Intelligent Systems (DE)
- University of Southern California (US)
- Technische Universität Berlin (DE)
- Venue
- IEEE Transactions on Robotics
- Published
- 2017-12
- Volume
- 33
- Issue
- 6
- Pages
- 1273-1291
- ISSN
- 1552-3098 (ISSN-L)
- 1941-0468
- OpenAlex
- W2337977475
- Cited by
- 227
- References
- 118
- Citations by year
- 2026: 16 · 2025: 37 · 2024: 30 · 2023: 30 · 2022: 27 · 2021: 28 · 2020: 23 · 2019: 22 · 2018: 11 · 2017: 3
- FWCI
- 10.14
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Robotics (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (IIS-1205249, IIS-1017134, EECS-0926052)
- Office of Naval Research
- Okawa Foundation
- Max Planck Society
- German Science Foundation (DFG) (BR 2248/3-1)
- European Commission (H2020-ICT-645599)
- Soft Manipulation (Soma)
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Robotics and Sensor-Based Localization
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- interactive perception, robot perception, perception-action coupling, perception-action interaction, robot-environment interaction, sensorimotor learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.