CoChat

Interactive Perception for Deformable Object Manipulation

Open access
Weng, Zehang, Zhou, Peng, Yin, Hang, Kravberg, Alexander, Varava, Anastasiia, Navarro-Alarcon, David, Kragic, Danica · 2024 · IEEE Robotics and Automation Letters

The DOI matches its Crossref record; 53 of the 53 references it cites were found in a registry or an index; 12 of the 21 sources its claims cite were checked for support.

Abstract

Interactive perception enables robots to manipulate the environment and objects to bring them into states that benefit the perception process. Deformable objects pose challenges to this due to manipulation difficulty and occlusion in vision-based perception. In this work, we address such a problem with a setup involving both an active camera and an object manipulator. Our approach is based on a sequential decision-making framework and explicitly considers the motion regularity and structure in coupling the camera and manipulator. We contribute a method for constructing and computing a subspace, called Dynamic Active Vision Space (DAVS), for effectively utilizing the regularity in motion exploration. The effectiveness of the framework and approach are validated in both a simulation and a real dual-arm robot setup. Our results confirm the necessity of an active camera and coordinative motion in interactive perception for deformable objects.

Overview

CoChat Summary

TL;DR

The paper formulates interactive perception for deformable object manipulation as a POMDP with an active camera and a manipulator, and proposes a Dynamic Active Vision Space (DAVS) to constrain camera actions using object structure of interest. Simulation and real dual-arm experiments show DAVS improves task efficiency, success, and generalization over baselines.

Key findings

  • In CubeBagClean, DAVS improved all four camera/end-effector birth-mode subtasks: e.g., Fix Cam/Fix EE episode length 200.0→39.0, reward 0.3→16107.1, success 0.0%→100.0%; Rand Cam/Fix EE 104.4→64.0, 9561.8→13600.7, 66.0%→84.0%.
  • In CubeBagObst, DAVS outperformed no-DAVS and static-vision baselines, especially when the camera started from a fixed challenging side perspective; Table II reports total success 50.5% without DAVS, 67.5% for DAVS ω=0, and 71.5% for DAVS ω=0.5, with DAVS ω=1 described as the best overall balance.
  • DAVS generalized to unseen dynamics and 36 unseen bag shapes; e.g., New Shapes Fix Cam/Fix EE: without DAVS 188.8 episode length / 1125.0 reward / 11.9% success vs. with DAVS 97.9 / 10213.0 / 66.4%.
  • On real hardware, after approximately 5,000 fine-tuning episodes, the perceiver and actor on a dual UR10e setup reliably located a blue cube inside a fabric bag.

Methods

  • The problem is modeled as a POMDP with action space factorized into camera and end-effector actions; DAVS is a manifold-with-boundary camera action subspace built from SOI keypoints, ray tracing, geodesic paths, and a centroid computed via Fréchet variance/trust-region optimization, then used to bias PPO action exploration.
  • Simulation uses PyBullet with CubeBagClean and CubeBagObst, observations of 2D depth image, 2D SOI heatmap, and end-effector position, a 200-step episode limit, 100 random scenes per subtask, and baselines including PPO without DAVS, static vision, and visual servoing.
  • Real experiments use two UR10e arms: one actor manipulates a bag handle, one perceiver carries a RealSense D435 depth camera, with camera pitch/yaw constrained and a policy fine-tuned for about 5,000 episodes.

Limitations

  • The paper states future directions: improving hand-eye coordination, explicitly incorporating dynamics prediction into the IP framework, and developing more effective parameterized DAVS constructions with task dependency.
  • The paper reports failure cases: simulated gripper movement that was too fast overstretched deformable handles; slow camera movement made object localization difficult, especially with obstacles; and generalization failed when stiffness differed significantly or the bag opening was small.
  • Evident from the text: the method assumes annotated SOI keypoints and a single-loop SOI structure, and the real setup relies on an overhead camera to identify the bag’s green rim and a fixed transformation; experiments are limited to bag-like objects and a specific dual-arm configuration.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/lra.2024.3431943

Cites

53 of 53 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

8 Valid · 2 Not valid · 3 Not checked · 37 Own results and interpretations

19 more claims

Create a free account to see all 50 claims and to verify the 2 references still open.

See all claims — free account
  • Probably not a substantial claim. 14
  • It may state two separate claims as one. 15

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 53 references

Research works printed but not linked: 1

  1. Yuille, 1992, Active Vision

Cited by

See all 8

Metadata

Authors
Weng, Z., Zhou, P., Yin, H., Kravberg, A., Varava, A., Navarro-Alarcon, D., & Kragic, D.
Affiliations
  • KTH Royal Institute of Technology (SE)
  • Hong Kong Polytechnic University (HK)
  • CAS/RPL, KTH, Stocholm, Sweden
Venue
IEEE Robotics and Automation Letters
Published
2024-09
Volume
9
Issue
9
Pages
7763-7770
ISSN
  • 2377-3766 (ISSN-L)
  • 2377-3774
OpenAlex
W4400877829
Cited by
7
References
49
Citations by year
2026: 3 · 2025: 3 · 2024: 1
FWCI
1.09
Type
journal article
Language
en
Open access
Locations
  • IEEE Robotics and Automation Letters (publishedVersion)
  • Research at the University of Copenhagen (University of Copenhagen) (publishedVersion)
  • PolyU Institutional Research Archive (Hong Kong Polytechnic University) (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
  • Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
Keywords
interactive perception, deformable object manipulation, active camera, active vision, sequential decision making, occlusion, dual-arm robot

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-11-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol and qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Interactive Perception for Deformable Object Manipulation. https://app.cochat.ai/papers/10.1109/lra.2024.3431943
The paper
Weng, Z., Zhou, P., Yin, H., Kravberg, A., Varava, A., Navarro-Alarcon, D., & Kragic, D. (2024). Interactive Perception for Deformable Object Manipulation. IEEE Robotics and Automation Letters. https://doi.org/10.1109/lra.2024.3431943

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.