Human-robot collaborative manipulation planning using early prediction of human motion
The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
In this paper we present a framework that allows a human and a robot to perform simultaneous manipulation tasks safely in close proximity. The proposed framework is based on early prediction of the human's motion. The prediction system, which builds on previous work in the area of gesture recognition, generates a prediction of human workspace occupancy by computing the swept volume of learned human motion trajectories. The motion planner then plans robot trajectories that minimize a penetration cost in the human workspace occupancy while interleaving planning and execution. Multiple plans are computed in parallel, one for each robot task available at the current time, and the trajectory with the least cost is selected for execution. We test our framework in simulation using recorded human motions and a simulated PR2 robot. Our results show that our framework enables the robot to avoid the human while still accomplishing the robot's task, even in cases where the initial prediction of the human's motion is incorrect. We also show that taking into account the predicted human workspace occupancy in the robot's motion planner leads to safer and more efficient interactions between the user and the robot than only considering the human's current configuration.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Anticipating human activities has been shown to improve human-robot collaboration [40, 21, 25, 10].”
From Recurrent Neural Networks for driver activity anticipation via sensory-fusion architecture (2016) · cites this paper as [25]
Citations
Cites
Pattern Recognition and Machine Learning
(2006) · doi:10.1007/978-0-387-45528-0 · Cited by 9,824
(2006) · Springer eBooks · Cited by 8,123
Active Learning with Statistical Models
Cohn, D. A., Ghahramani, Z., & Jordan, M. I. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.295 · Cited by 1,317 · cited 1×
“To obtain a motion reconstructed from the GMMs that represents the predicted trajectory for each class, we use the GMR procedure [25].”
MethodOn Learning, Representing, and Generalizing a Task in a Humanoid Robot
Calinon, S., Guenter, F., & Billard, A. (2007) · IEEE Transactions on Systems, Man and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2006.886952 · Cited by 1,090 · cited 2×
“human’s motion and querying it using Gaussian Mixture Regression (GMR) is similar to [21], although, again, we extend this work to predict the workspace occupancy of the human, as well as integrating the prediction with the robot motion planning.”
MethodA System for the Notation of Proxemic Behavior1
HALL, E. T. (1963) · American Anthropologist · doi:10.1525/aa.1963.65.5.02a00020 · Cited by 1,006 · cited 1×
“In the closely-related topic of robot navigation in the presence of humans, early works were inspired by the social behavior demonstrated by humans [9].”
BackgroundCHOMP: Gradient optimization techniques for efficient motion planning
Ratliff, N., Zucker, M., Bagnell, J. A., & Srinivasa, S. (2009) · 2009 IEEE International Conference on Robotics and Automation · doi:10.1109/robot.2009.5152817 · Cited by 992 · cited 1×
“We rely on recent developments in trajectory optimization for motion planning [1] [22] [23] to compute low-cost robot motion plans quickly.”
MethodSTOMP: Stochastic trajectory optimization for motion planning
Kalakrishnan, M., Chitta, S., Theodorou, E., Pastor, P., & Schaal, S. (2011) · 2011 IEEE International Conference on Robotics and Automation · doi:10.1109/icra.2011.5980280 · Cited by 876 · cited 16×
“The initial algorithm presented in [23] optimizes a combination of two classical criteria, namely obstacle cost and smoothness cost.”
BackgroundMethodInfluentialAnticipating Human Activities Using Object Affordances for Reactive Robotic Response
Koppula, H. S., & Saxena, A. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2430335 · Cited by 596
A Human Aware Mobile Robot Motion Planner
Sisbot, E. A., Marin-Urias, L. F., Alami, R., & Simeon, T. (2007) · IEEE Transactions on Robotics · doi:10.1109/tro.2007.904911 · Cited by 499 · cited 1×
“clearance, but also about how visible the robot is along the robot motion to avoid the effect of surprise [10] leading to more legible navigation plans.”
BackgroundPlanning-based prediction for pedestrians
Ziebart, B. D., Ratliff, N., Gallagher, G., Mertz, C., Peterson, K., Bagnell, J. A., Hebert, M., Dey, A. K., & Srinivasa, S. (2009) · 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems · doi:10.1109/iros.2009.5354147 · Cited by 456 · cited 1×
“Similar to our approach, researchers have explored how to learn models of human motion and use them for online online navigation planning [12].”
BackgroundRequirements for Safe Robots: Measurements, Analysis and New Insights
Haddadin, S., Albu-Schäffer, A., & Hirzinger, G. (2009) · The International Journal of Robotics Research · doi:10.1177/0278364909343970 · Cited by 444 · cited 1×
“[7], [8], have led to the characterization of safety criteria.”
BackgroundElastic Strips: A Framework for Motion Generation in Human Environments
Brock, O., & Khatib, O. (2002) · The International Journal of Robotics Research · doi:10.1177/027836402128964189 · Cited by 352 · cited 1×
“In the closely-related topic of robot navigation in the presence of humans, early works were inspired by the social behavior demonstrated by humans [9].”
BackgroundInevitable collision states — a step towards safer robots?
Fraichard, T., & Asama, H. (2004) · Advanced Robotics · doi:10.1163/1568553042674662 · Cited by 324
Active Learning with Statistical Models.
Cohn, D. A., Ghahramani, Z., & Jordan, M. I. (1995) · doi:10.21236/ada295617 · Cited by 222
Improved human-robot team performance using chaski, a human-inspired plan execution system
Shah, J., Wiken, J., Williams, B., & Breazeal, C. (2011) · Proceedings of the 6th international conference on Human-robot interaction · doi:10.1145/1957656.1957668 · Cited by 218 · cited 1×
“To account for varying action costs at the process level, we seek to explore integration with task-level planning for human-robot teams such as [27].”
BackgroundHuman-robot interaction in handing-over tasks
Huber, M., Rickert, M., Knoll, A., Brandt, T., & Glasauer, S. (2008) · RO-MAN 2008 - The 17th IEEE International Symposium on Robot and Human Interactive Communication · doi:10.1109/roman.2008.4600651 · Cited by 216 · cited 1×
“Simultaneous motion for human-robot collaboration has been studied in terms of hand-over tasks [15], [16], [17].”
BackgroundPre-collision safety strategies for human-robot interaction
Kulić, D., & Croft, E. (2007) · Autonomous Robots · doi:10.1007/s10514-006-9009-4 · Cited by 200 · cited 1×
“Some of these tasks, such as electronics or aircraft assembly, are difficult to automate because they require workers to collaborate in close proximity and adapt to each other’s decisions and motions, which robots cannot currently do.”
BackgroundHuman-Robot Interaction for Cooperative Manipulation: Handing Objects to One Another
Edsinger, A., & Kemp, C. C. (2007) · RO-MAN 2007 - The 16th IEEE International Symposium on Robot and Human Interactive Communication · doi:10.1109/roman.2007.4415256 · Cited by 193 · cited 1×
“Simultaneous motion for human-robot collaboration has been studied in terms of hand-over tasks [15], [16], [17].”
BackgroundUsing spatial and temporal contrast for fluent robot-human hand-overs
Çakmak, M., Srinivasa, S. S., Lee, M. K., Kiesler, S. B., & Forlizzi, J. (2011) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/1957656.1957823 · Cited by 143 · cited 1×
“Simultaneous motion for human-robot collaboration has been studied in terms of hand-over tasks [15], [16], [17].”
BackgroundAnticipating Human Activities using Object Affordances for Reactive Robotic Response
Koppula, H., & Saxena, A. (2013) · Robotics: Science and Systems IX · doi:10.15607/rss.2013.ix.006 · Cited by 130
Planning human-aware motions using a sampling-based costmap planner
Mainprice, J., Sisbot, E. A., Jaillet, L., Cortés, J., Alami, R., & Siméon, T. (2011) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2011.5980048 · Cited by 120 · cited 1×
“Recently this approach was extended to combine planning and more reactive schemes [11], allowing more natural crossings between the robot and the human.”
BackgroundSelf-citationMove3D: A generic platform for path planning
Siméon, T., Laumond, J., & Lamiraux, F. (2002) · IEEE International Symposium on Assembly and Task Planning (ISATP) · doi:10.1109/isatp.2001.928961 · Cited by 109 · cited 2×
“The online classifier, STOMP, and a simulated version of the execution framework have been implemented in Move3D [26].”
MethodLegible robot navigation in the proximity of moving humans
Kruse, T., Basili, P., Glasauer, S., & Kirsch, A. (2012) · IEEE Workshop on Advanced Robotics and its Social Impacts (ARSO) · doi:10.1109/arso.2012.6213404 · Cited by 96 · cited 1×
“Recently this approach was extended to combine planning and more reactive schemes [11], allowing more natural crossings between the robot and the human.”
BackgroundSharing effort in planning human-robot handover tasks
Mainprice, J., Gharbi, M., Siméon, T., & Alami, R. (2012) · IEEE RO-MAN · doi:10.1109/roman.2012.6343844 · Cited by 94 · cited 1×
“A motion planning method that allows to share the effort by exploring the combined configuration space of the human and the robot has been proposed in [18].”
MethodSelf-citationDragan, A. D., & Srinivasa, S. S. (2018) · doi:10.1184/r1/6554969.v1 · Cited by 93
Real-time optimization-based planning in dynamic environments using GPUs
Park, C., Pan, J., & Manocha, D. (2013) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2013.6631154 · Cited by 87 · cited 1×
“In the closely-related topic of robot navigation in the presence of humans, early works were inspired by the social behavior demonstrated by humans [9].”
BackgroundEarly Recognition and Prediction of Gestures
Mori, A., Uchida, S., Kurazume, R., Taniguchi, R., Hasegawa, T., & Sakoe, H. (2006) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2006.467 · Cited by 76 · cited 1×
“Researchers have investigated early recognition of gestures [19] [20], where the goal is to classify a type of gesture being performed by the human.”
BackgroundPandey, A. K., & Alami, R. (2010) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2010.5651503 · Cited by 61 · cited 1×
“Recently this approach was extended to combine planning and more reactive schemes [11], allowing more natural crossings between the robot and the human.”
BackgroundMaking planned paths look more human-like in humanoid robot manipulation planning
Zacharias, F., Schlette, C., Schmidt, F., Borst, C., Roßmann, J., & Hirzinger, G. (2011) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2011.5979553 · Cited by 59 · cited 1×
“Recent work has explored computing more human-like motions [13] using a reachability map representation.”
BackgroundSorel, A. (2012) · theses.fr (ABES) · doi:10.70675/4e4066bdz1e6bz45e3z8bd5z82a149263798 · Cited by 10 · cited 2×
“Researchers have investigated early recognition of gestures [19] [20], where the goal is to classify a type of gesture being performed by the human.”
Background
Research works printed but not linked: 2
- huber, 2008, Humanrobot interaction in handing-over tasks, ROMAN
- 10.1177/0278364902021012002 · doi:10.1177/0278364902021012002 · Paper page
Cited by
Robot Collisions: A Survey on Detection, Isolation, and Identification
Haddadin, S., De Luca, A., & Albu-Schaffer, A. (2017) · IEEE Transactions on Robotics · doi:10.1109/tro.2017.2723903 · Cited by 791
“Other approaches like [11], [21]–[23] aim for effectively planning collision-free robot motions also incorporating the prediction of human behavior.”
BackgroundOutlook on human-centric manufacturing towards Industry 5.0
Lu, Y., Zheng, H., Chand, S., Xia, W., Liu, Z., Xu, X., Wang, L., Qin, Z., & Bao, J. (2022) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2022.02.001 · Cited by 671
Symbiotic human-robot collaborative assembly
Wang, L., Gao, R., Váncza, J., Krüger, J., Wang, X. V., Makris, S., & Chryssolouris, G. (2019) · CIRP Annals · doi:10.1016/j.cirp.2019.05.002 · Cited by 565
“Mainprice and Berenson [146] categorised human actions through the use of Gaussian Mixture Models (GMMs) and Regression (GMR).”
MethodBi, L., Feleke, A. >., & Guan, C. (2019) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2019.02.011 · Cited by 418
A Survey of Methods for Safe Human-Robot Interaction
Lasota, P. A., Fong, T., & Shah, J. A. (2017) · Foundations and Trends® in Robotics · doi:10.1561/2300000052 · Cited by 403
“Mainprice and Berenson (2013) developed a framework that utilizes labeled demonstrations of reaching motions to generate models for prediction of workspace occupancy.”
Background
Metadata
- Authors
- Mainprice, J., & Berenson, D.
- Affiliations
- Worcester Polytechnic Institute (US)
- Venue
- 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems
- Publisher
- IEEE
- Published
- 2013-11
- Pages
- 299-306
- ISSN
- 2153-0858 (ISSN-L)
- OpenAlex
- W1992461594
- Cited by
- 301
- References
- 30
- Citations by year
- 2026: 8 · 2025: 12 · 2024: 24 · 2023: 20 · 2022: 27 · 2021: 32 · 2020: 30 · 2019: 30 · 2018: 31 · 2017: 37 · 2016: 26 · 2015: 17 · 2014: 6 · 2013: 1
- FWCI
- 22.77
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (publishedVersion)
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Engineering › Control and Systems Engineering › Robotic Path Planning Algorithms
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Keywords
- human-robot collaboration, manipulation planning, human motion prediction, gesture recognition, swept volume, collision avoidance, PR2 robot, human-robot interaction safety
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.