Interactive Perception for Deformable Object Manipulation
Open accessThe DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index; 12 of the 21 sources its claims cite were checked for support.
Abstract
Interactive perception enables robots to manipulate the environment and objects to bring them into states that benefit the perception process. Deformable objects pose challenges to this due to manipulation difficulty and occlusion in vision-based perception. In this work, we address such a problem with a setup involving both an active camera and an object manipulator. Our approach is based on a sequential decision-making framework and explicitly considers the motion regularity and structure in coupling the camera and manipulator. We contribute a method for constructing and computing a subspace, called Dynamic Active Vision Space (DAVS), for effectively utilizing the regularity in motion exploration. The effectiveness of the framework and approach are validated in both a simulation and a real dual-arm robot setup. Our results confirm the necessity of an active camera and coordinative motion in interactive perception for deformable objects.
Overview
CoChat Summary
TL;DR
The paper formulates interactive perception for deformable object manipulation as a POMDP with an active camera and a manipulator, and proposes a Dynamic Active Vision Space (DAVS) to constrain camera actions using object structure of interest. Simulation and real dual-arm experiments show DAVS improves task efficiency, success, and generalization over baselines.
Key findings
- In CubeBagClean, DAVS improved all four camera/end-effector birth-mode subtasks: e.g., Fix Cam/Fix EE episode length 200.0→39.0, reward 0.3→16107.1, success 0.0%→100.0%; Rand Cam/Fix EE 104.4→64.0, 9561.8→13600.7, 66.0%→84.0%.
- In CubeBagObst, DAVS outperformed no-DAVS and static-vision baselines, especially when the camera started from a fixed challenging side perspective; Table II reports total success 50.5% without DAVS, 67.5% for DAVS ω=0, and 71.5% for DAVS ω=0.5, with DAVS ω=1 described as the best overall balance.
- DAVS generalized to unseen dynamics and 36 unseen bag shapes; e.g., New Shapes Fix Cam/Fix EE: without DAVS 188.8 episode length / 1125.0 reward / 11.9% success vs. with DAVS 97.9 / 10213.0 / 66.4%.
- On real hardware, after approximately 5,000 fine-tuning episodes, the perceiver and actor on a dual UR10e setup reliably located a blue cube inside a fabric bag.
Methods
- The problem is modeled as a POMDP with action space factorized into camera and end-effector actions; DAVS is a manifold-with-boundary camera action subspace built from SOI keypoints, ray tracing, geodesic paths, and a centroid computed via Fréchet variance/trust-region optimization, then used to bias PPO action exploration.
- Simulation uses PyBullet with CubeBagClean and CubeBagObst, observations of 2D depth image, 2D SOI heatmap, and end-effector position, a 200-step episode limit, 100 random scenes per subtask, and baselines including PPO without DAVS, static vision, and visual servoing.
- Real experiments use two UR10e arms: one actor manipulates a bag handle, one perceiver carries a RealSense D435 depth camera, with camera pitch/yaw constrained and a policy fine-tuned for about 5,000 episodes.
Limitations
- The paper states future directions: improving hand-eye coordination, explicitly incorporating dynamics prediction into the IP framework, and developing more effective parameterized DAVS constructions with task dependency.
- The paper reports failure cases: simulated gripper movement that was too fast overstretched deformable handles; slow camera movement made object localization difficult, especially with obstacles; and generalization failed when stiffness differed significantly or the bag opening was small.
- Evident from the text: the method assumes annotated SOI keypoints and a single-loop SOI structure, and the real setup relies on an overhead camera to identify the bag’s green rim and a fixed transformation; experiments are limited to bag-like objects and a specific dual-arm configuration.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
8 Valid · 2 Not valid · 3 Not checked · 37 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“Notably, advancements in complex 3D deformable bag manipulation have been presented [27], [28], [29], [30], [31]. These works have shown significant progress in the domain but do not incorporate an active movable camera for task achievement.”
What the sources say
- Supportedcites [27] source
The Task Description describes 3D bag-opening, lifting, and handle-motion tasks. Dataset Generation describes recording the simulated bag’s vertices and rigid objects’ positions as the complete scene state, while Dynamics Learning and Prediction uses that state to predict action effects. This supports the claim as applied to this cited work: its described method does not incorporate an active movable camera for task achievement. It does not establish the same point for the other works cited in the sentence. (LLM, conf=0.86)
- Supportedcites [28] source
The Method Overview (Fig. 2, reproduced in the Conclusion) describes a top camera providing top-down images for the bagging policies. The Simulation benchmark section likewise describes top-down images from an Azure Kinect sensor; neither describes moving the camera as part of the task. Together with the paper’s deformable-bag manipulation system, this supports the claim as applied to this cited work. (LLM, conf=0.87)
- Supportedcites [29] source
The Task Configurations section (p. 3) identifies bag opening as the paper’s 3D deformable-object task and says the policy selects actions from top-down images. The System and Task Setup section (p. 3) describes three view-specific cameras and robot arms equipped with grippers or an air pump; it does not describe moving a camera to achieve the task. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The recent benchmarks that build upon advanced simulation engines again focus primarily on a static camera setup [32], [33].”
What the sources say
- Partly supportedcites [32] Lin, 2021, SoftGym: Benchmarking deep reinforcement learning for deformable object manipulation sourceExperimental — accuracy not yet measured
Section 3 (p. 2) says SoftGym is a benchmark built on the Nvidia FleX physics simulator. Appendix A.1 (p. 13) describes image observations rendered by that simulator, but does not say the camera is static or that static setups predominate. The camera-setup portion of the attributed claim is therefore unsupported by our copy. (LLM, conf=0.83)
- Not matchedcites [33] Coumans, 20162021, PyBullet, a Python module for physics simulation for games, robotics and machine learning
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Our work relates to concepts such as hand-eye coordination [11], visual servoing [12], active perception [13], [14], [15], [16], [17], [18] and interactive perception [1], but none of these fully addresses all the three aspects in a single framework: a deformable object, moving camera, and a robot interacting with the object.”
What the sources say
- Supportedcites [11] source
The citation is offered to identify the concept 'hand-eye coordination', and our copy is exactly that work: the Abstract (p. 1) describes 'a learning-based approach to hand-eye coordination for robotic grasping from monocular images' that servos the gripper in real time. Our copy also shows the paper does not combine all three aspects the citing authors list: the camera is fixed, 'mounted behind the arm looking over the shoulder' (5.1 Hardware Setup, p. 6), and objects are treated as grasp targets — soft ones such as paper tissues and sponges are handled empirically (7. Discussion, p. 10) with no deformable-object model — while the robot-object interaction is present via continuous servoing (1. Introduction, p. 1). The negative framing is thus consistent with the scope of the source. (LLM, conf=0.85)
- Not checked — no full text of the sourcecites [12] Kragic, 2002, Survey on visual servoing for manipulation
- Not checked — no full text of the sourcecites [13] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While ω = 0 significantly improves performance in simpler scenarios (CubeBagClean), higher values of ω (DAVS ω = 0 . 5 and DAVS ω = 1 ) are crucial for handling more complex environments (CubeBagObst).”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As presented in Table I, the IP policy, when utilizing DAVS, not only consistently outperforms its counterpart without DAVS but also demonstrates superior generalizability, underscoring the robustness and effectiveness of our methodology.”
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
19 more claims
Create a free account to see all 50 claims and to verify the 2 references still open.
- Probably not a substantial claim. 14
- It may state two separate claims as one. 15
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ D. Result · p. 6
“The last row of the figure illustrates the dynamic evolution of the manifold in a CubeBagClean episode in the first row.”
§ A. Environment Setup · p. 5
“The second row is the 3D manifold visualization corresponding to the first row (BlueDot: Camera; Green: 3D SOI; Red: Projected SOI; BlueCurve: Boundary of DAVS).”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Algorithm 1: Action Exploration with DAVS · p. 3
“12 Get next observation ok + 1 and reward rk from the environment ;”
§ Algorithm 1: Action Exploration with DAVS · p. 3
“13 Provide the transition tuple ( ok -1 , α k , ok , rk ) to the RL algorithm ;”
Citations
Cites
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Proximal Policy Optimization Algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017) · arXiv · doi:10.48550/arxiv.1707.06347 · Cited by 10,625 · cited 1×
“To assess the effectiveness of the policy trained using DAVS for deformable object manipulation with active perception, we selected a policy fine-tuned within the CubeBagClean simulation environment (featuring random camera and end-effector positions) for real-world experimentation.”
MethodLevine, S., Pastor, P., Krizhevsky, A., Ibarz, J., & Quillen, D. (2018) · The International Journal of Robotics Research · doi:10.1177/0278364917710318 · Cited by 1,971 · cited 2×
“Our work relates to concepts such as hand-eye coordination [11], visual servoing [12], active perception [13], [14], [15], [16], [17], [18] and interactive perception [1], but none of these fully addresses all the three aspects in a single framework: a deformable object, moving camera, and a robot…”
BackgroundPolicy Invariance Under Reward Transformations: Theory and Application to Reward Shaping
Ng, A. Y., Harada, D., & Russell, S. (1999) · Cited by 1,744
Ballard, D. H. (1991) · Artificial Intelligence · doi:10.1016/0004-3702(91)90080-4 · Cited by 1,022 · cited 1×
“Our work relates to concepts such as hand-eye coordination [11], visual servoing [12], active perception [13], [14], [15], [16], [17], [18] and interactive perception [1], but none of these fully addresses all the three aspects in a single framework: a deformable object, moving camera, and a robot…”
BackgroundAloimonos, J., Weiss, I., & Bandyopadhyay, A. (1988) · International Journal of Computer Vision · doi:10.1007/bf00133571 · Cited by 931 · cited 1×
“Our work relates to concepts such as hand-eye coordination [11], visual servoing [12], active perception [13], [14], [15], [16], [17], [18] and interactive perception [1], but none of these fully addresses all the three aspects in a single framework: a deformable object, moving camera, and a robot…”
BackgroundPanoptic Studio: A Massively Multiview System for Social Motion Capture
Joo, H., Liu, H., Tan, L., Gui, L., Nabbe, B., Matthews, I., Kanade, T., Nobuhara, S., & Sheikh, Y. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.381 · Cited by 546 · cited 1×
“Specifically,theend-effectorcanbearbitrarilymoved whiletheactivecameraistraversingthehalf-spheresurface, similartotheCMU’spanopticmassivecameragrid[42]setting.”
Introduction to Topological Manifolds
Lee, J. M. (2011) · Graduate Texts in Mathematics · doi:10.1007/978-1-4419-7940-7 · Cited by 511 · cited 1×
“To this end, we augment the POMDP ( S, O, Z, T, R, γ ) , such that z ∈ Z = M is the constrained action space subjecting to M .”
MethodPanoptic Studio: A Massively Multiview System for Social Interaction Capture
Joo, H., Simon, T., Li, X., Liu, H., Tan, L., Gui, L., Banerjee, S., Godisart, T., Nabbe, B., Matthews, I., et al. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.48550/arxiv.1612.03153 · Cited by 482
Learning to Simulate Complex Physics with Graph Networks
Sánchez‐González, Á., Godwin, J., Pfaff, T., Ying, R., Leskovec, J., & Battaglia, P. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.09405 · Cited by 413
Research works printed but not linked: 1
- Yuille, 1992, Active Vision
Cited by
Qi, J., Lu, L., Wang, F., Lee, H.-Y., Navarro-Alarcón, D., Zhang, Z., & Zhou, P. (2025) · Robotics and Computer-Integrated Manufacturing · doi:10.1016/j.rcim.2025.103096 · Cited by 22
Self-citationBimanual Deformable Bag Manipulation Using a Structure-of-Interest Based Neural Dynamics Model
Zhou, P., Zheng, P., Qi, J., Li, C., Lee, H.-Y., Pan, Y., Yang, C., Navarro-Alarcón, D., & Pan, J. (2024) · IEEE/ASME Transactions on Mechatronics · doi:10.1109/tmech.2024.3485471 · Cited by 22
“Last, Weng et al. [27] explored the concept of interactive perception for DOM, where the perception and interaction are tightly integrated to enhance manipulation capabilities.”
Self-citationRobotic perception and manipulation of deformable linear objects: A survey
Caporali, A., Cuiral-Zueco, I., Nicolás, G. L., & Palli, G. (2026) · The International Journal of Robotics Research · doi:10.1177/02783649261432253 · Cited by 14
Chen, Z., Hari, K., Dasari, T., Shieh, K., Jain, R., Fer, D., Guthart, G. S., & Goldberg, K. (2025) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros60139.2025.11246566 · Cited by 6
“Weng et al. [32] explored deformable object manipulation using interactive perception within a system integrating an active camera.”
A Perspective on Open Challenges in Deformable Object Manipulation
McKennaa, R. P., & Oyekan, J. (2026) · arXiv · doi:10.48550/arxiv.2602.22998 · Cited by 1
“An example of this is introduced by Weng et al [39].”
Influential
Metadata
- Authors
- Weng, Z., Zhou, P., Yin, H., Kravberg, A., Varava, A., Navarro-Alarcon, D., & Kragic, D.
- Affiliations
- KTH Royal Institute of Technology (SE)
- Hong Kong Polytechnic University (HK)
- CAS/RPL, KTH, Stocholm, Sweden
- ORCID
- Venue
- IEEE Robotics and Automation Letters
- Published
- 2024-09
- Volume
- 9
- Issue
- 9
- Pages
- 7763-7770
- ISSN
- 2377-3766 (ISSN-L)
- 2377-3774
- OpenAlex
- W4400877829
- Cited by
- 7
- References
- 49
- Citations by year
- 2026: 3 · 2025: 3 · 2024: 1
- FWCI
- 1.09
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Robotics and Automation Letters (publishedVersion)
- Research at the University of Copenhagen (University of Copenhagen) (publishedVersion)
- PolyU Institutional Research Archive (Hong Kong Polytechnic University) (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- interactive perception, deformable object manipulation, active camera, active vision, sequential decision making, occlusion, dual-arm robot
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-11-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Interactive Perception for Deformable Object Manipulation. https://app.cochat.ai/papers/10.1109/lra.2024.3431943
- The paper
- Weng, Z., Zhou, P., Yin, H., Kravberg, A., Varava, A., Navarro-Alarcon, D., & Kragic, D. (2024). Interactive Perception for Deformable Object Manipulation. IEEE Robotics and Automation Letters. https://doi.org/10.1109/lra.2024.3431943
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.