Interactive Perception: Leveraging Action in Perception and Perception in Action
Open accessThe DOI matches its Crossref record; 120 of the 120 references it cites were found in a registry or an index.
Abstract
Recent approaches in robot perception follow the insight that perception is facilitated by interaction with the environment. These approaches are subsumed under the term Interactive Perception (IP). This view of perception provides the following benefits. First, interaction with the environment creates a rich sensory signal that would otherwise not be present. Second, knowledge of the regularity in the combined space of sensory data and action parameters facilitates the prediction and interpretation of the sensory signal. In this survey, we postulate this as a principle for robot perception and collect evidence in its support by analyzing and categorizing existing work in this area. We also provide an overview of the most important applications of IP. We close this survey by discussing remaining open questions. With this survey, we hope to help define the field of Interactive Perception and to provide a valuable resource for future research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Such structure is called Action Perception Regularity [1], and is believed to be the key for IP to reveal richer signals, which is impossible for passive and one-shot perception.”
From Interactive Perception for Deformable Object Manipulation (2024) · cites this paper as [1]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Interactive Perception (IP) exploits various types of forceful interactions with the environment to facilitate perception [1].”
From Interactive Perception for Deformable Object Manipulation (2024) · cites this paper as [1]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472 · cited 1×
“t given only a single input image. A method has to cope with occlusions, different lighting conditions, scale of the images, just to name a few. State-of-the-art approaches in Computer Vision as e.g. [107, 108] require enormous amounts of training data to handle these variations. Interactive Perception approaches allo…”
MethodMicrosoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 1×
“These three examples illustrate that biological perception and perceptually-guided behavior intrinsically rely on active exploration and knowledge of the relation between action and sensory response.”
BackgroundFully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
The pascal visual object classes (VOC) challenge
Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2010) · International Journal of Computer Vision · doi:10.1007/s11263-009-0275-4 · Cited by 19,599 · cited 2×
“The most successful approaches learn models from data sets that contain hundreds of thousands of semantically annotated static images, such as Pascal VOC [6], ImageNet [7] or Microsoft COCO [8].”
BackgroundDomain-Adversarial Training of Neural Networks
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., & Lempitsky, V. (2017) · Advances in Computer Vision and Pattern Recognition · doi:10.1007/978-3-319-58347-1_10 · Cited by 8,318
Contour Detection and Hierarchical Image Segmentation
Arbeláez, P., Maire, M., Fowlkes, C., & Malik, J. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.161 · Cited by 5,454 · cited 1×
“This emphasizes the blurred boundary which is traditionally drawn between manipulation and perception.”
BackgroundLaValle, S. M. (2006) · Cambridge University Press eBooks · doi:10.1017/cbo9780511546877 · Cited by 5,211
Computational modelling of visual attention
Itti, L., & Koch, C. (2001) · Nature Reviews Neuroscience · doi:10.1038/35058500 · Cited by 4,733 · cited 1×
“of freedom [17]–[19] and models of visual attention [20], [21] that these active vision systems could use for guiding their gaze.”
Background
Research works printed but not linked: 17
- [33] P. Allen and R. Bajscy, 'Two sensors are better than one: Example of integration of vision and touch,' in International Symposium on Robotics Research (ISRR) , France, October 1985. -
- [32] K. I. Goldberg and R. Bajscy, 'Active touch and robot perception,' Cognition and Brain Theory , vol. 7, no. 2, pp. 199-214, 1984. -
- [31] R. Bajscy, 'Shape from touch,' Advances in Automation and Robotics, JAI Press , 1988. -
- [24] G. Sandini, F. Gandolfo, E. Grosso, and M. Tistarelli, Vision during Action . Lawrence Erlbaum Associates, Inc, 1993, ch. 4. -
- [23] L. Natale, G. Metta, and G. Sandini, 'Learning haptic representation of objects,' in International Conference on Intelligent Manipulation and Grasping , July 2004. -
- [101] S. Otte, J. Kulick, M. Toussaint, and O. Brock, 'Entropybased strategies for physical exploration of the environments degrees of freedom,' in IEEE/RSJ International Conference on Intelligent Robots and Systems , 2014. -
- noë, 2004, Action in Perception
- [5] J. J. Gibson, The senses considered as perceptual systems . Boston: Houghton Mifflin, 1966. -
- [43] D. Katz and O. Brock, 'Interactive segmentation of articulated objects in 3d,' in Workshop on Mobile Manipulation at ICRA 2011 , 2011. -
- [128] S. Singh, M. R. James, and M. R. Rudary, 'Predictive state representations: A new theory for modeling dynamical systems,' in Proceedings of the 20th Conference on Uncertainty in Artificial Intelligence , ser. UAI '04. Arlington, Virginia, United States: AUAI Press, 2004, pp. 512-519. -
- [131] N. Kr¨ uger, C. Geib, J. Piater, R. Petrick, M. Steedman, F. W¨ org¨ otter, A. Ude, T. Asfour, D. Kraft, D. Omrˇ cen, A. Agostini, and R. Dillmann, 'Objectaction complexes: Grounded abstractions of sensorymotor processes,' Robotics and Autonomous Systems , vol. 59, no. 10, pp. 740 - 757, 2011. -
- [3] A. No¨ e, Action in Perception . The MIT Press, 2004. -
- [1] J. J. Gibson, The Ecological Approach to Visual Perception . Houghton Mifflin, 1979. -
- [103] D. Katz, A. Orthey, and O. Brock, 'Interactive perception of articulated objects,' Springer Tracts in Advanced Robotics , vol. 79, pp. 301-315, 2014. -
- [111] L. Natale and E. Torres-Jara, 'A sensitive approach to grasping,' in Proceedings of the sixth international workshop on epigenetic robotics . Citeseer, 2006, pp. 87-94. -
- [17] K. Pahlavan, T. Uhlin, and J.-O. Eklundh, 'Active vision as a methodology,' Active perception , pp. 19-46, 1993. -
- [120] E. Torres-Jara, L. Natale, and P. Fitzpatrick, 'Tapping into touch,' pp. 79-86, 2005. -
Cited by
Items 101–200 of 308
Comparing Single Touch to Dynamic Exploratory Procedures for Robotic Tactile Object Recognition
Kirby, E., Zenha, R., & Jamone, L. (2022) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2022.3151261 · Cited by 18
“For tactile perception to happen, the robot must generate the physical interaction with the objects, i.e. active [8] or interactive perception [9].”
BackgroundMa, W., Zhang, B., Han, L., Huo, S., Wang, H., & Navarro-Alarcón, D. (2022) · IEEE/ASME Transactions on Mechatronics · doi:10.1109/tmech.2022.3223990 · Cited by 17
“Assumed to be accurately executed without failure [12], action primitives are well”
BackgroundSGTM 2.0: Autonomously Untangling Long Cables using Interactive Perception
Shivakumar, K., Viswanath, V., Gu, A., Avigal, Y., Kerr, J. H., Ichnowski, J., Cheng, R., Kollar, T., & Goldberg, K. Y. (2023) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48891.2023.10160574 · Cited by 23
“To address this limitation, this paper focuses on quantifying uncertainty to enable applying interactive perception [5], which involves physically manipulating objects in a scene to better understand it.”
BackgroundThe Rise of Molecular Image–Guided Robotic Surgery
van Leeuwen, F. W. B., Buckle, T., van Oosterom, M. N., & Rietbergen, D. D. D. (2024) · Journal of Nuclear Medicine · doi:10.2967/jnumed.124.267783 · Cited by 16
“This concept can be illustrated using an example provided by Bohg et al. ( 13 ) showing that shape recognition based on static imaging provides 49% accuracy in object recognition whereas rotation of 3-dimensional vision enriches perception with a corresponding increase to 72%.”
Building an Affordances Map With Interactive Perception
Goff, L. K. L., Yaakoubi, O., Coninx, A., & Doncieux, S. (2022) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2022.504459 · Cited by 19
“[9], a robotic system, with interactive perception, isolates regularities in the combined space of sensory signals, motor commands and time.”
BackgroundParts-Based Articulated Object Localization in Clutter Using Belief Propagation
Pavlasek, J., Lewis, S., Desingh, K., & Jenkins, O. C. (2020) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9340908 · Cited by 19
“Interactive perception [25] for articulated object estimation [26] has been a problem of interest in the robotics community.”
BackgroundHaptic Knowledge Transfer Between Heterogeneous Robots using Kernel Manifold Alignment
Tatiya, G., Shukla, Y., Edegware, M., & Sinapov, J. (2020) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9340770 · Cited by 15
“language grounding (see [1], [2] for a review).”
BackgroundSelf-supervised Interactive Object Segmentation Through a Singulation-and-Grasping Approach
Yu, H., & Choi, C. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19842-7_36 · Cited by 15
Tatiya, G., Francis, J., & Sinapov, J. (2023) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48891.2023.10160811 · Cited by 24
“Recent advances in intelligent robotics consider integrating multisensory information acquired by object exploration [10], [25]–[30], where one challenge is that the implicit knowledge acquired by one robot through interactive perception cannot be directly transferred to another robot: the unique nature”
BackgroundSafe Leaf Manipulation for Accurate Shape and Pose Estimation of Occluded Fruits
Yao, S., Pan, S., Bennewitz, M., & Hauser, K. K. (2025) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra55743.2025.11128788 · Cited by 14
“A robot can explore an environment and gain informative observations by interacting with objects [2].”
Obtaining an Object’s 3D Model Using Dual-Arm Robotic Manipulation and Stationary Depth Sensing
Kobayashi, S., Wan, W., Kiyokawa, T., Koyama, K., & Harada, K. (2022) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2022.3193691 · Cited by 15
“Previously, Bohg et al. [26], Hausman et al. [27], Novkovic et al. [28], and Sinapov et al. [29] respectively surveyed or presented systems that implemented perceptive manipulation (the passive human policy).”
BackgroundSelf-Supervised Unseen Object Instance Segmentation via Long-Term Robot Interaction
Lu, Y., Khargonkar, N., Xu, Z., Averill, C., Palanisamy, K., Hang, K., Guo, Y., Ruozzi, N., & Yu, X. (2023) · Robotics: Science and Systems · doi:10.15607/rss.2023.xix.017 · Cited by 18
Robot–Camera Calibration in Tightly Constrained Environment Using Interactive Perception
Zhong, F., Li, B., Chen, W., & Liu, Y. (2023) · IEEE Transactions on Robotics · doi:10.1109/tro.2023.3299533 · Cited by 16
“The concept of IP has been systematically addressed in the work [43] and has been widely applied to robot manipulation tasks involving physical interactions to the environment [44], [45].”
BackgroundMorphEyes: Variable Baseline Stereo For Quadrotor Navigation
Sanket, N. J., Singh, C. D., Asthana, V., Fermuller, C. M., & Aloimonos, Y. (2021) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48506.2021.9561116 · Cited by 14
POMDP Planning Under Object Composition Uncertainty: Application to Robotic Manipulation
Pajarinen, J., Lundell, J., & Kyrki, V. (2022) · IEEE Transactions on Robotics · doi:10.1109/tro.2022.3188168 · Cited by 17
“A promising approach toward solving object discovery in clutter is interactive perception [1], where the object configuration is examined actively, typically by manipulating the objects and observing the results.”
BackgroundMengers, V., Roth, N., Brock, O., Obermayer, K., & Rolfs, M. (2025) · Journal of Vision · doi:10.1167/jov.25.2.6 · Cited by 16
“This way, the information of different components can be weighted in their connections, and the robot can act according to the current uncertainty, for example, by moving more carefully or actively obtaining more information (Bohg et al., 2017).”
Self-citationAction Selection for Interactive Object Segmentation in Clutter
Patten, T., Zillich, M., & Vincze, M. (2018) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8593918 · Cited by 17
“Interactive perception is the problem of interacting with the environment to improve perception quality and has been applied to many tasks such as recognition, modelling and pose estimation [2].”
BackgroundDutta, A., Burdet, E., & Kaboli, M. (2023) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10341832 · Cited by 19
“In such interactive visuo-tactile perception , active physical interactions or explorations are made to enhance object perception [3]–[6].”
BackgroundDistributional Depth-Based Estimation of Object Articulation ModelsPreprint
Jain, A., Giguere, S., Lioutikov, R., & Niekum, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.05875 · Cited by 17
Discovering affordances through perception and manipulation
Chavéz-García, R. O., Luce-Vayrac, P., & Chatila, R. (2016) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2016.7759583 · Cited by 12
Fully Convolutional One-Shot Object Segmentation for Industrial Robotics
Schnieders, B., Luo, S., Palmer, G., & Tuyls, K. (2019) · Adaptive Agents and Multi-Agents Systems · doi:10.65109/hbhp6980 · Cited by 12
Perception-Action Coupling in Usage of Telepresence Cameras
Valiton, A., & Li, Z. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9197578 · Cited by 12
“…hand, interactive perception exploits interaction with the environment to create a rich sensory signal that would otherwise not be available, and utilizes knowledge of the regularity in the combined space of sensory data and action parameters to predict and/or interpret the sensory signal [12].”
BackgroundPlaying with Food: Learning Food Item Representations Through Interactive Exploration
Sawhney, A. S., Lee, S. S. W., Zhang, K., Veloso, M., & Kroemer, O. (2021) · Springer proceedings in advanced robotics · doi:10.1007/978-3-030-71151-1_28 · Cited by 24
An information-theoretic on-line update principle for perception-action coupling
Peng, Z., Genewein, T., Leibfried, F., & Braun, D. A. (2017) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8202240 · Cited by 14
Articulated Object Manipulation using Online Axis Estimation with SAM2-Based TrackingPreprint
Wang, X., Chen, T., Yu, Q., Xu, T. X., Chen, Z., Fu, Y., He, Z., Lu, C., Mu, Y., & Luo, P. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2409.16287 · Cited by 16
“[24] highlighted the important role of interactive perception in manipulation.”
BackgroundCAPT: Category-level Articulation Estimation from a Single Point Cloud Using Transformer
Fu, L., Ishikawa, R., Sato, Y., & Oishi, T. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10611073 · Cited by 15
“Interaction methods provide sufficient dynamic information for articulation estimation, as made clear in numerous studies [5] in the observations before and after the interaction to help estimate an initial guess of the articulation model [9] or build up an implicit neural-representation of the…”
MethodShared visuo-tactile interactive perception for robust object pose estimation
Murali, P. K., Porr, B., & Kaboli, M. (2024) · The International Journal of Robotics Research · doi:10.1177/02783649241301443 · Cited by 19
“Interactive perception or perceptive manipulation is any kind of purposeful manipulation actions performed to simplify or enhance the perception of the environment (Bohg et al., 2017).”
SoftGrasp: Adaptive grasping for dexterous hand based on multimodal imitation learning
Li, Y., Guo, C., Ren, J., Chen, B., Cheng, C., Zhang, H., & Lu, H. (2025) · Biomimetic Intelligence and Robotics · doi:10.1016/j.birob.2025.100217 · Cited by 25
Yoneyama, R., Duran, A. J., & del Pobil, A. P. (2021) · Sensors · doi:10.3390/s21041437 · Cited by 11
Graph-based Cluttered Scene Generation and Interactive Exploration using Deep Reinforcement Learning
Kumar, K. N., Essa, I. A., & Ha, S. (2022) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra46639.2022.9811874 · Cited by 12
Tactile-Filter: Interactive Tactile Perception for Part Mating
Ota, K., Jha, D. K., Tung, H.-Y. F., & Tenenbaum, J. B. (2023) · Robotics: Science and Systems · doi:10.15607/rss.2023.xix.079 · Cited by 14
Active estimation of mass properties for safe cooperative lifting
Corah, M., & Michael, N. (2017) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2017.7989533 · Cited by 11
Robot-Supervised Learning for Object Segmentation
Florence, V., Corso, J. J., & Griffin, B. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9196543 · Cited by 12
“ronments [5].”
Martín-Martín, R., & Brock, O. (2017) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8206165 · Cited by 11
“a) Interactive Perception: Interactive perception methods exploit correlations between actions and changes in the sensor signals for robot perception [3].”
BackgroundSelf-citationPredictive Visuo-Tactile Interactive Perception Framework for Object Properties Inference
Dutta, A., Burdet, E., & Kaboli, M. (2025) · IEEE Transactions on Robotics · doi:10.1109/tro.2025.3531816 · Cited by 21
Viewpoint Push Planning for Mapping of Unknown Confined Spaces
Dengler, N., Pan, S., Kalagaturu, V., Menon, R., Dawood, M., & Bennewitz, M. (2023) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10341809 · Cited by 13
“In contrast, interactive perception [31] as proposed in this paper performs additional actions to change the current object configuration to support perception and coverage of the environment.”
BackgroundYang, S., Mao, X., Wang, S., & Bai, Y. (2021) · Journal of Intelligent & Robotic Systems · doi:10.1007/s10846-021-01396-0 · Cited by 11
“An expected robotic plan is an interleaved and adjoint sequence of acting and sensing actions, with each acting action being planned upon new observations received by a sensing action [11].”
BackgroundDeveloping Intelligent Robots that Grasp Affordance
Loeb, G. E. (2022) · Frontiers in Robotics and AI · doi:10.3389/frobt.2022.951293 · Cited by 13
“Understanding rather than simply observing the world may emerge through the processes of “ interactive perception, ” recently reviewed by (Bohg et al., 2017).”
BackgroundUncertainty-Aware Manipulation Planning Using Gravity and Environment Geometry
von Drigalski, F., Kasaura, K., Beltran-Hernandez, C. C., Hamaya, M., Tanaka, K., & Matsubara, T. (2022) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2022.3207565 · Cited by 10
“Many studies have explored active perception approaches where robots interact with objects to determine the objects’ location or properties [15].”
BackgroundMobile Robot Manipulation using Pure Object Detection
Griffin, B. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00063 · Cited by 20
“Thus, passive learning is not enough, especially for robots, where visual experience is dynamic and interactive [4].”
An AI‐Enabled All‐In‐One Visual, Proximity, and Tactile Perception Multimodal Sensor
Pu, M., Zhao, T., Zhang, L., Han, C., Chai, Z., Zhou, Y., Ding, H., & Wu, Z. (2025) · Advanced Robotics Research · doi:10.1002/adrr.202500062 · Cited by 9
Li, X., Baum, M., & Brock, O. (2023) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10341625 · Cited by 18
“This is an example of interactive perception [5], where forceful interaction induces additional sensory events that are otherwise unavailable.”
BackgroundSelf-citationSpotLight: Robotic Scene Understanding Through Interaction and Affordance Detection
Engelbracht, T., Zurbrügg, R., Pollefeys, M., Blum, H., & Bauer, Z. (2025) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/humanoids65713.2025.11203097 · Cited by 14
“Interactive perception refers to the task of an agent interacting with its environment and thereby gaining new insights of the environment [30].”
Kirkland, P., Caterina, G. D., Soraghan, J. J., & Matich, G. (2020) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2020.568319 · Cited by 9
“This work aims to build on the already successful perception-action models (Nishiwaki et al., 2003; Xie, 2003; Bohg et al., 2017; Masuta et al., 2017) and add some semantic understanding to the robotic system.”
BackgroundPerception-Motion Coupling in Active Telepresence: Human Behavior and Teleoperation Interface Design
Lin, T.-C., Krishnan, A. U., & Li, Z. (2022) · ACM Transactions on Human-Robot Interaction · doi:10.1145/3571599 · Cited by 10
“Knowledge about perception-action coupling has been leveraged in human-robot interaction to a limited extent, and has already yielded effective models and approaches for predicting human intent [10], optimizing camera motions and viewpoints [80], interactive perception [15], and sensory augmentation of human-robot int…”
BackgroundLiu, W., Bansal, D., Daruna, A., & Chernova, S. (2023) · Autonomous Robots · doi:10.1007/s10514-023-10099-4 · Cited by 10
“Interactive perception can further leverage a robot’s exploratory actions to reveal sensory signals that are otherwise not observable [9, 56, 14, 61, 2, 8, 59].”
BackgroundPhysics-Based Selection of Informative Actions for Interactive Perception
Eppner, C., Martín-Martín, R., & Brock, O. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8460596 · Cited by 9
“The core idea behind all those methods is to make interactions part of the perceptual process by exploiting forceful interactions and the information rich sensory signals they generate [1].”
BackgroundSelf-citationMartín-Martín, R., & Brock, O. (2017) · National Conference on Artificial Intelligence · Cited by 12
Self-citationCloth manipulation based on category classification and landmark detection
Gustavsson, O., Ziegler, T., Welle, M. C., Bütepage, J., Varava, A., & Kragić, D. (2022) · International Journal of Advanced Robotic Systems · doi:10.1177/17298806221110445 · Cited by 12
Inferring 3D Shapes of Unknown Rigid Objects in Clutter Through Inverse Physics Reasoning
Song, C., & Boularias, A. (2018) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2018.2885579 · Cited by 13
Self-Supervised Object-in-Gripper Segmentation from Robotic MotionsPreprint
Boerdijk, W., Sundermeyer, M., Durner, M., & Triebel, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.04487 · Cited by 9
“In the context of robotics, learning from the environment by means of interacting, sensing and perceiving information is generally entitled Interactive Perception (IP) [6].”
BackgroundGrasp, See, and Place: Efficient Unknown Object Rearrangement With Policy Structure Prior
Xu, K., Zhou, Z., Wu, J., Lu, H., Xiong, R., & Wang, Y. (2024) · IEEE Transactions on Robotics · doi:10.1109/tro.2024.3502520 · Cited by 13
Robotic perception and manipulation of deformable linear objects: A survey
Caporali, A., Cuiral-Zueco, I., Nicolás, G. L., & Palli, G. (2026) · The International Journal of Robotics Research · doi:10.1177/02783649261432253 · Cited by 16
Uncertainty-based Exploring Strategy in Densely Cluttered Scenes for Vacuum Cup Grasping
Tung, K., Su, J., Cai, J., Wan, Z., & Cheng, H. (2022) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra46639.2022.9811599 · Cited by 8
Estimation of Contact Regions Between Hands and Objects During Human Multi-Digit Grasping
Hartmann, F., Maiello, G., Rothkopf, C. A., & Fleming, R. W. (2023) · Journal of Visualized Experiments · doi:10.3791/64877 · Cited by 8
Aoyama, M. Y., Moura, J., Saito, N., & Vijayakumar, S. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610502 · Cited by 11
“Force or tactile perception can reveal physical properties of objects, such as weight, texture, and compliance that are typically unavailable through visual perception [25]–[28].”
BackgroundAutonomous dishwasher loading from cluttered trays using pre‐trained deep neural networks
Voysey, I., Thuruthel, T. G., & Iida, F. (2020) · Engineering Reports · doi:10.1002/eng2.12321 · Cited by 9
“37,38 The results and dataset obtained through our study can also serve as a tool for future researchers and for developing novel deep networks. describe in detail the algorithms developed for the dishwasher loading problem built around our experimental setup (see Figure 2).”
MethodReview of the techniques used in motor‐cognitive human‐robot skill transfer
Guan, Y. H., Wang, N., & Yang, C. (2021) · Cognitive Computation and Systems · doi:10.1049/ccs2.12025 · Cited by 8
“However, whether computers can have the same cognitive abilities as humans may remain controversial for a long time in F I G U R E 1 8 Illustration of the use of active perception to obtain more information [153] the future.”
BackgroundMultimodal Learning of Keypoint Predictive Models for Visual Object Manipulation
Bechtle, S., Das, N., & Meier, F. (2023) · IEEE Transactions on Robotics · doi:10.1109/tro.2022.3204509 · Cited by 10
“However, there is more and more evidence that utilizing multimodal sensor streams improves perception and manipulation [24], [25], [26], [27].”
BackgroundPerceiving Unseen 3D Objects by Poking the Objects
Chen, L., Song, Y., Bao, H., & Zhou, X. (2023) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra48891.2023.10160338 · Cited by 12
“In contrast, several recent works in robotics propose to learn from interaction with the environment [8].”
BackgroundFuseBot: mechanical search of rigid and deformable objects via multi-modal perception
Boroushaki, T., Dodds, L., Naeem, N., & Adib, F. (2023) · Autonomous Robots · doi:10.1007/s10514-023-10137-1 · Cited by 8
“We assume that the robot knows the shape of the tagged item, and has a database with the shapes of all RFID-tagged items.”
BackgroundOnline Estimation of Articulated Objects with Factor Graphs using Vision and Proprioceptive Sensing
Buchanan, R., Röfer, A., Moura, J., Valada, A., & Vijayakumar, S. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610590 · Cited by 16
“First, we cover interactive perception [8] methods which have a long history of use with articulated objects.”
BackgroundSensorimotor Cross-Behavior Knowledge Transfer for Grounded Category Recognition
Tatiya, G., Hosseini, R., Hughes, M. C., & Sinapov, J. (2019) · IEEE International Conference on Development and Learning (ICDL) · doi:10.1109/devlrn.2019.8850715 · Cited by 8
“Despite the significant advancement in interactive and multisensory object perception for robots [8], one challenge is that multisensory representations such as haptic, proprioceptive, auditory, and tactile perceptions cannot be easily transferred from one robot to another, as different robots may…”
BackgroundEcological-Inspired System Design for Safety Manipulation Strategy in Home-care Robot
Yani, M., Besari, A. R. A., Yamada, N., & Kubota, N. (2020) · International Symposium on Community-centric Systems (CcS) · doi:10.1109/ccs49175.2020.9231354 · Cited by 8
“Safety strategy for pick and place adopted from Ikuta. et al that perceptually-guided action intrinsically depends on active observer and understanding of the relationship between behavior and sensory response [16], [17].”
BackgroundTatiya, G., Hosseini, R., Hughes, M. C., & Sinapov, J. (2020) · Frontiers in Robotics and AI · doi:10.3389/frobt.2020.522141 · Cited by 9
“…of an object is the commonly adopted approach; however, several research studies show incorporating a variety of sensory modalities is the key to further enhance the robotic capabilities in recognizing the multisensory object properties (see Bohg et al., 2017; Li et al., 2020 for a review).”
BackgroundLiu, W., Bansal, D., Daruna, A., & Chernova, S. (2021) · Robotics: Science and Systems · doi:10.15607/rss.2021.xvii.035 · Cited by 12
Interactive Perception for Deformable Object Manipulation
Weng, Z., Zhou, P., Yin, H., Kravberg, A., Varava, A., Navarro-Alarcon, D., & Kragic, D. (2024) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2024.3431943 · Cited by 7
“…concepts such as hand-eye coordination [11], visual servoing [12], active perception [13], [14], [15], [16], [17], [18] and interactive perception [1], but none of these fully addresses all the three aspects in a single framework: a deformable object, moving camera, and a robot interacting with…”
BackgroundBagging by Learning to Singulate Layers Using Interactive Perception
Chen, L. Y., Shi, B., Lin, R., Seita, D., Ahmad, A., Cheng, R., Kollar, T., Held, D., & Goldberg, K. Y. (2023) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10341634 · Cited by 17
“Interactive perception combines perception and action, enabling a robot to reduce uncertainty through active physical exploration and interaction with objects [6].”
BackgroundMethodTrafficability anticipation for quadruped robot in field operation
Wang, C., Zhang, R., Dong, W., Li, T., Jiang, L., Xu, W. M., Xu, P., Zhou, Y., & Zou, M. (2024) · Journal of Field Robotics · doi:10.1002/rob.22292 · Cited by 7
“Therefore, they can obtain rich sensor data about the force and the soil deformation at their footholds (Bartolozzi et al., 2022; Bohg et al., 2017; Hwangbo et al., 2019).”
BackgroundSchorp, V., Panitch, W., Shivakumar, K., Viswanath, V., Kerr, J. H., Avigal, Y., Fer, D., Ott, L., & Goldberg, K. Y. (2023) · IEEE International Conference on Automation Science and Engineering (CASE) · doi:10.1109/case56687.2023.10260353 · Cited by 7
“Interactive perception has been used in robotic manipulation to extract kinematic and dynamic models from physical interactions with the environment [12] and to improve the understanding of a scene in the presence of occlusions and perception uncertainty [11, 13, 14].”
BackgroundMethodRISeg: Robot Interactive Object Segmentation via Body Frame-Invariant Features
Qian, H. H., Lu, Y., Ren, K., Wang, G., Khargonkar, N., Yu, X., & Hang, K. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10611015 · Cited by 7
“Motion-based segmentation methods attempt to segment environments by utilizing a robot’s interactions with objects to detect scene changes in a sequence of images [6], [20].”
BackgroundMultimodal Cross-Attention Mechanism-Based Algorithm for Elderly Behavior Monitoring and Recognition
Líu, H., Feng, Z., & Guo, Q. (2025) · Chinese Journal of Electronics · doi:10.23919/cje.2023.00.263 · Cited by 10
NudgeSeg: Zero-Shot Object Segmentation by Repeated Physical Interaction
Singh, C. D., Sanket, N. J., Parameshwara, C. M., Fermuller, C. M., & Aloimonos, Y. (2021) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9635832 · Cited by 7
“The synergy pair to active approaches are the ones which involve interaction between the agent and its environment – interactive approach which was formally defined in [17].”
BackgroundOnline Unsupervised Learning of the 3D Kinematic Structure of Arbitrary Rigid Bodies
Nunes, U. M., & Demiris, Y. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00391 · Cited by 7
“The kinematic structure of an articulated rigid body provides a compact and meaningful representation, which is useful for robot manipulation tasks and object kinematic recognition [21], finding kinematic correspondences between objects [6], interactively perceiving and acting with the environment [3], among others.”
BackgroundRobot Embodied Dynamic Tactile Perception of Liquid in Containers
Xu, Y., Ma, Y., Lin, W., Sun, Z., Zhang, T., & Wang, Z. (2024) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/humanoids58906.2024.10769944 · Cited by 6
“The interactive perception capability allows the creation of a rich sensory signal [13].”
FuseBot: RF-Visual Mechanical Search
Boroushaki, T., Dodds, L., Naeem, N., & Adib, F. (2022) · Robotics: Science and Systems · doi:10.15607/rss.2022.xviii.067 · Cited by 6
“Researchers explored the use of perceptual completion to identify partially occluded objects [17, 28], and developed systems that perform active perception whereby a robot moves a camera around the environment in order to search for items that are partially visible [2, 3, 4].”
BackgroundA Recognition Method for Soft Objects Based on the Fusion of Vision and Haptics
Sun, T. F., Zhang, Z., Miao, Z., & Zhang, W. (2023) · Biomimetics · doi:10.3390/biomimetics8010086 · Cited by 6
“The three-layer Conv1D network parameters are [[32, 64, 3, 2], [64, 128, 3, 2] and [128, 32, 2, 1]], respectively.”
BackgroundFluid viscosity prediction leveraging computer vision and robot interaction
Park, J. H., Dalwankar, G. P., Bartsch, A., George, A., & Farimani, A. B. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108603 · Cited by 6
Segmenting Objects through an Autonomous Agnostic Exploration Conducted by a Robot
Goff, L. K. L., Mukhtar, G., Fur, P.-H. L., & Doncieux, S. (2017) · IEEE International Conference on Robotic Computing (IRC) · doi:10.1109/irc.2017.22 · Cited by 6
“Most of previous work, in Interactive Perception [4], use a passive image processing step before any interactions to produce objects hypothesis and bootstrap the system.”
BackgroundInfluentialIdentification of Invariant Sensorimotor Structures as a Prerequisite for the Discovery of Objects
Hir, N. L., Sigaud, O., & Laflaquière, A. (2018) · Frontiers in Robotics and AI · doi:10.3389/frobt.2018.00070 · Cited by 7
“These approaches have been gathered under the term Interactive Perception [28].”
BackgroundPromoting Religious Tourism Management for Creating a Soft Image of Pakistan
Sumra, K. B., & Alam, M. M. (2020) · Advances in hospitality, tourism and the services industry (AHTSI) book series · doi:10.4018/978-1-7998-5792-1.ch010 · Cited by 7
Kinematically-Informed Interactive Perception: Robot-Generated 3D Models for ClassificationPreprint
Venkataraman, A., Griffin, B., & Corso, J. J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.05580 · Cited by 6
“Accordingly, Active Perception (AP) [3, 2] and Interactive Perception (IP) [5] are two areas of research exploiting robot-specific capabilities to improve perception.”
BackgroundMinimal perception: enabling autonomy in resource-constrained robots
Singh, C. D., He, B., Fermuller, C. M., Metzler, C. A., & Aloimonos, Y. (2024) · Frontiers in Robotics and AI · doi:10.3389/frobt.2024.1431826 · Cited by 5
Benjdira, B., Koubâa, A., & Ali, A. M. (2025) · Future Generation Computer Systems · doi:10.1016/j.future.2025.107723 · Cited by 6
“These include the need for robust object recognition under varying conditions, generalization across different object categories and environments, and real-time processing for fast and adaptive robotic control [24, 25].”
Mengers, V., & Brock, O. (2025) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra55743.2025.11127552 · Cited by 13
“If we instead introduce unexpected light ( e ) or heavy ( f ) disturbances, our approach can adapt to state changes and thus remains robust where planning approaches fail ( d ). performance drops, particularly in tasks requiring view-point adjustment or grasping, as the system lost its adaptability.”
Self-citationRUMI: Rummaging Using Mutual Information
Zhong, S., Fazeli, N., & Berenson, D. (2025) · IEEE Transactions on Robotics · doi:10.1109/tro.2025.3605251 · Cited by 7
“This problem encompasses a range of scenarios, differentiated by the type of robot (e.g., mobile versus stationary), the primary sensor modality (often vision), and the specific quantity to be estimated.”
Multimodal embodied attribute learning by robots for object-centric action policies
Zhang, X., Amiri, S., Sinapov, J., Thomason, J., Stone, P., & Zhang, S. (2023) · Autonomous Robots · doi:10.1007/s10514-023-10098-5 · Cited by 7
“To address this problem, several lines of research have shown that incorporating a variety of sensory modalities is the key to further enhancing the robotic capabilities in recognizing multisensory object attributes, as reviewed in recent articles [31,41].”
BackgroundMovingCables: Moving Cable Segmentation Method and Dataset
Holešovský, O., Škoviera, R., & Hlaváč, V. (2024) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2024.3416800 · Cited by 6
Visuo-Tactile Based Predictive Cross Modal Perception for Object Exploration in Robotics
Dutta, A., Burdet, E., & Kaboli, M. (2024) · IEEE International Symposium on Robotic and Sensors Environments · doi:10.1109/rose62198.2024.10590901 · Cited by 5
“…to handle novel objects, humans frequently rely on exploratory and interactive actions, such as holding, grasping, or pushing, to infer object properties such as shape, inertial, surface texture, and viscoelasticity [1]–[5], which is formalized as interactive visuo-tactile perception [6]–[10].”
BackgroundHigh-Degrees-of-Freedom Dynamic Neural Fields for Robot Self-Modeling and Motion Planning
Schulze, L., & Lipson, H. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10611047 · Cited by 7
“In all approaches, data acquisition plays a crucial role, with strategies ranging from entirely random [7], to interactive [21], and targeted-exploratory [19, 22].”
BackgroundEmbodied Uncertainty-Aware Object Segmentation
Fang, X., Kaelbling, L. P., & Lozano‐Pérez, T. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10801562 · Cited by 6
“…robot actions to complement and enhance visual perception has a long history in robotics and is variously known as active perception, interactive perception or embodied perception; the survey by Bohg et al. [17] reviews this body of work, which includes work on interactive/embodied segmentation.”
Wang, G., Ren, K., Morgan, A. S., & Hang, K. (2025) · The International Journal of Robotics Research · doi:10.1177/02783649251343926 · Cited by 10
“Meanwhile, interactive perception has significantly reduced the requirements for certain prior knowledge and direct perception (Bohg et al. 2017; Hang et al. 2021), while still assuming reliable perception through some channels for iterative state estimation.”
Planning for Learning Object Properties
Lamanna, L., Serafini, L., Faridghasemnia, M., Saffiotti, A., Saetti, A., Gerevini, A., & Traverso, P. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i10.26416 · Cited by 5
Object-aware interactive perception for tabletop scene exploration
Koc, C., & Sarıel, S. (2024) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2024.104674 · Cited by 5
Hear the Egg - Demonstrating Robotic Interactive Auditory Perception
Strahl, E., Kerzel, M., Eppe, M., Griffiths, S., & Wermter, S. (2018) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8593959 · Cited by 5
“By using interactive auditory perception [2], a robot can detect if a salt shaker is empty or the box of biscuits really contains cookies without opening it for visual inspection.”
MethodThe Automated Inspection of Opaque Liquid Vaccines
Palmer, G., Schnieders, B., Savani, R., Tuyls, K., Fossel, J., & Flore, H. (2020) · Frontiers in artificial intelligence and applications · doi:10.3233/faia200307 · Cited by 5
Learning to Look Around: Enhancing Teleoperation and Learning with a Human-like Actuated NeckPreprint
Sen, B., Wang, M., Thakur, N., Agarwal, A. A., & Agrawal, P. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2411.00704 · Cited by 5
“The actuated neck also enables interactive perception [16], as it integrates sensor observations and action trajectories over time.”
BackgroundLiu, Y., Jia, T., Zhang, H., Yang, G., Wang, H., & Chen, D. (2025) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2025.3604455 · Cited by 7
“Active perception is a complex control problem in which a robot equipped with onboard sensors maximizes the information obtained by interacting with the underlying environment [3], [4], [5], [6].”
Liu, H., Liu, X., Huang, K., & Guo, D. (2025) · Machine learning · doi:10.1007/978-981-96-5871-8_1 · Cited by 4
C., J. L. M., Kluß, T., & Zetzsche, C. (2019) · Frontiers in Robotics and AI · doi:10.3389/frobt.2019.00063 · Cited by 4
“In the context of interactive perception, Bohg et al. (2017) state that there is a regular relationship between actions and their sensory response, which can be exploited by robotic systems for the prediction and interpretation of these sensory signals.”
Background
Metadata
- Authors
- Bohg, J., Hausman, K., Sankaran, B., Brock, O., Kragic, D., Schaal, S., & Sukhatme, G. S.
- Affiliations
- Max Planck Institute for Intelligent Systems (DE)
- University of Southern California (US)
- Technische Universität Berlin (DE)
- Venue
- IEEE Transactions on Robotics
- Published
- 2017-12
- Volume
- 33
- Issue
- 6
- Pages
- 1273-1291
- ISSN
- 1552-3098 (ISSN-L)
- 1941-0468
- OpenAlex
- W2337977475
- Cited by
- 227
- References
- 118
- Citations by year
- 2026: 16 · 2025: 37 · 2024: 30 · 2023: 30 · 2022: 27 · 2021: 28 · 2020: 23 · 2019: 22 · 2018: 11 · 2017: 3
- FWCI
- 10.14
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Robotics (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (IIS-1205249, IIS-1017134, EECS-0926052)
- Office of Naval Research
- Okawa Foundation
- Max Planck Society
- German Science Foundation (DFG) (BR 2248/3-1)
- European Commission (H2020-ICT-645599)
- Soft Manipulation (Soma)
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Robotics and Sensor-Based Localization
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- interactive perception, robot perception, perception-action coupling, perception-action interaction, robot-environment interaction, sensorimotor learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.