Benchmarking Protocol for Grasp Planning Algorithms
Abstract
Numerous grasp planning algorithms have been proposed since the 1980s. The grasping literature has expanded rapidly in recent years, building on greatly improved vision systems and computing power. Methods have been proposed to plan stable grasps on known objects (exact 3D model is available), familiar objects (e.g. exploiting a-priori known grasps for different objects of the same category), or novel object shapes observed during task execution. Few of these methods have ever been compared in a systematic way, and objective performance evaluation of such complex systems remains problematic. Difficulties and confounding factors include different assumptions and amounts of a-priori knowledge in different algorithms; different robots, hands, vision systems and setups in different labs; and different choices or application needs for grasped objects. Also, grasp planning can use different grasp quality metrics (including empirical or theoretical stability measures) or other criteria, e.g., computational speed, or combination of grasps with reachability considerations. While acknowledging and discussing the outstanding difficulties surrounding this complex topic, we propose a methodology for reproducible experiments to compare the performance of a variety of grasp planning algorithms. Our protocol attempts to improve the objectivity with which different grasp planners are compared by minimizing the influence of key components in the grasping pipeline, e.g., vision and pose estimation. The protocol is demonstrated by evaluating two different grasp planners: a state-of-the-art model-free planner and a popular open-source model-based planner. We show results from real-robot experiments with a 7-DoF arm and 2-finger hand, as well as simulation-based evaluations.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Grasp stability is evaluated using various tests as in [18].
“Once grasped, we perform various tests as in [18] to evaluate grasp stability.”
From Dual Quaternion-Based Visual Servoing for Grasping Moving Objects (2021) · cites this paper as [18]
The results are reported following the evaluation protocol presented in [18].
“We report the results following the evaluation protocol presented in [18].”
From Dual Quaternion-Based Visual Servoing for Grasping Moving Objects (2021) · cites this paper as [18]
Cites
Levine, S., Pastor, P., Krizhevsky, A., Ibarz, J., & Quillen, D. (2018) · The International Journal of Robotics Research · doi:10.1177/0278364917710318 · Cited by 1,971 · see citing works
Robotic grasping and contact: a review
Bicchi, A., & Kumar, V. (2002) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/robot.2000.844081 · Cited by 1,091 · see citing works
Data-Driven Grasp Synthesis—A Survey
Bohg, J., Morales, A., Asfour, T., & Kragic, D. (2014) · IEEE Transactions on Robotics · doi:10.1109/tro.2013.2289018 · Cited by 929 · see citing works
Adaptive synergies for the design and control of the Pisa/IIT SoftHand
Catalano, M. G., Grioli, G., Farnioli, E., Serio, A., Piazza, C., & Bicchi, A. (2014) · The International Journal of Robotics Research · doi:10.1177/0278364913518998 · Cited by 673 · see citing works
Benchmarking in Manipulation Research: Using the Yale-CMU-Berkeley Object and Model Set
Çallı, B., Walsman, A., Singh, A., Srinivasa, S. S., Abbeel, P., & Dollar, A. M. (2015) · IEEE Robotics & Automation Magazine · doi:10.1109/mra.2015.2448951 · Cited by 602 · see citing works
Robotic grasp detection using deep convolutional neural networks
Kumra, S., & Kanan, C. (2017) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8202237 · Cited by 557 · see citing works
1Data-Driven Grasp Synthesis- A SurveyPreprint
Bohg, J., Morales, A., Asfour, T., & Kragić, D. (2014) · Cited by 532 · see citing works
An overview of 3D object grasp synthesis algorithms
Sahbani, A., El-Khoury, S., & Bidaud, P. (2011) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2011.07.016 · Cited by 479 · see citing works
Analysis and Observations From the First Amazon Picking Challenge
Correll, N., Bekris, K. E., Berenson, D., Brock, O., Causo, A., Hauser, K., Okada, K., Rodriguez, A., Romano, J. M., & Wurman, P. R. (2018) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2016.2600527 · Cited by 425 · cited 1× · see citing works
“Competitions such as the Amazon Picking Challenge (APC) [17] and Robotic Grasping and Manipulation Competition at IROS [18] have proposed different”
BackgroundOpenRAVE: A Planning Architecture for Autonomous Robotics
Diankov, R., & Kuffner, J. (2008) · Cited by 381 · see citing works
Review of Deep Learning Methods in Robotic Grasp Detection
Caldera, S., Rassau, A., & Chai, D. (2018) · Multimodal Technologies and Interaction · doi:10.3390/mti2030057 · Cited by 225 · see citing works
Robot grasp planning based on demonstrated grasp strategies
Lin, Y. Q., & Sun, Y. (2014) · The International Journal of Robotics Research · doi:10.1177/0278364914555544 · Cited by 108 · see citing works
Dynamic grasp and trajectory planning for moving objects
Marturi, N., Kopicki, M., Rastegarpanah, A., Rajasekaran, V., Adjigble, M., Stolkin, R., Leonardis, A., & Bekiroglu, Y. (2019) · Autonomous Robots · doi:10.1007/s10514-018-9799-1 · Cited by 104 · see citing works
Self-citationHierarchical Fingertip Space: A Unified Framework for Grasp Planning and In-Hand Grasp Adaptation
Hang, K., Li, M., Stork, J. A., Bekiroglu, Y., Pokorny, F. T., Billard, A., & Kragić, D. (2016) · IEEE Transactions on Robotics · doi:10.1109/tro.2016.2588879 · Cited by 103 · see citing works
Self-citationThe ACRV picking benchmark: A robotic shelf picking benchmark to foster reproducible research
Leitner, J., Tow, A. W., Sünderhauf, N., Dean, J. E., Durham, J. W., Cooper, M., Eich, M., Lehnert, C., Mangels, R., McCool, C., et al. (2017) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2017.7989545 · Cited by 83 · see citing works
Falco, J., Wyk, K. V., Liu, S., et al. (2015) · IEEE Robotics & Automation Magazine · doi:10.1109/mra.2015.2460891 · Cited by 73 · see citing works
The Synthesis of Stable Force-Closure Grasps
Nguyen, 。., & Van-Duc (1986) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 64 · see citing works
The SDM Hand: A Highly Adaptive Compliant Grasper for Unstructured Environments
Dollar, A. M., & Howe, R. D. (2009) · Springer tracts in advanced robotics · doi:10.1007/978-3-642-00196-3_2 · Cited by 54 · see citing works
Ulbrich, S., Kappler, D., Asfour, T., Vahrenkamp, N., Bierbaum, A., Przybylski, M., & Dillmann, R. (2011) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2011.6094894 · Cited by 46 · see citing works
The Synthesis of Stable Force-Closure Grasps
Nguyen, V. T. (1986) · Cited by 46 · see citing works
Model-free and learning-free grasping by Local Contact Moment matching
Adjigble, M., Marturi, N., Ortenzi, V., Rajasekaran, V., Corke, P., & Stolkin, R. (2018) · 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8594226 · Cited by 44 · see citing works
Self-citationAnalytic grasp success prediction with tactile feedback
Krug, R., Lilienthal, A. J., Kragić, D., et al. (2016) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2016.7487130 · Cited by 32 · see citing works
VisGraB: A Benchmark for Vision-Based Grasping
Kootstra, G., Popović, M., Jørgensen, J. A., Kragić, D., Petersen, H. G., & Krüger, N. (2012) · Paladyn Journal of Behavioral Robotics · doi:10.2478/s13230-012-0020-5 · Cited by 31 · see citing works
VIsion force control in task-oriented grasping and manipulation
Prats, M., Martinet, P., del Pobil, A. P., et al. (2007) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2007.4399256 · Cited by 24 · see citing works
An assisted telemanipulation approach: combining autonomous grasp planning with haptic cues
Adjigble, M., Marturi, N., Ortenzi, V., & Stolkin, R. (2019) · 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros40897.2019.8968454 · Cited by 19 · see citing works
Self-citationSotiropoulos, P., Roa, M. A., Martins, M. F., Fried, W., Mnyusiwalla, H., Triantafyllou, P. D., & Deacon, G. (2018) · IEEE-RAS International Conference on Humanoid Robots (Humanoids) · doi:10.1109/humanoids.2018.8624924 · Cited by 17 · see citing works
Self-citationRobotic Grasping and Manipulation Competition: Task Pool
Sun, Y., Falco, J., Cheng, N. G., et al. (2018) · Communications in computer and information science · doi:10.1007/978-3-319-94568-2_1 · Cited by 16 · see citing works
Triantafyllou, P. D., Mnyusiwalla, H., Sotiropoulos, P., Roa, M. A., Russell, D., & Deacon, G. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8793993 · Cited by 13 · see citing works
Self-citationGrasp quality evaluation in underactuated robotic hands
Pozzi, M., Sundaram, A. M., Malvezzi, M., Prattichizzo, D., & Roa, M. A. (2016) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2016.7759307 · Cited by 12 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Dual Quaternion-Based Visual Servoing for Grasping Moving Objects
De Farias, C., Adjigble, M., Tamadazte, B., Stolkin, R., & Marturi, N. (2021) · 2021 IEEE 17th International Conference on Automation Science and Engineering (CASE) · doi:10.1109/case49439.2021.9551631 · Cited by 1 · see citing works
“They are: (i) lifting test, in which we lift the object 20 cm above the table at the speed of 10 cm/s; (ii) rotation test, where we move the manipulator to be in 90 − 90 configuration [18] and rotate the object from 90◦ to −90◦”
MethodInfluentialSelf-citation
- Authors
- Bekiroglu, Y., Marturi, N., Roa, M. A., Adjigble, K. J. M., Pardi, T., Grimm, C., Balasubramanian, R., Hang, K., & Stolkin, R.
- Affiliations
- University of Birmingham (GB)
- Oregon State University (US)
- Yale University (US)
- ORCID
- Venue
- IEEE Robotics and Automation Letters
- Published
- 2020-04
- Volume
- 5
- Issue
- 2
- Pages
- 315-322
- ISSN
- 2377-3766 (ISSN-L)
- 2377-3774
- OpenAlex
- W2990957401
- Cited by
- 41
- References
- 29
- Citations by year
- 2026: 2 · 2025: 2 · 2024: 7 · 2023: 7 · 2022: 7 · 2021: 15 · 2020: 1
- FWCI
- 3.18
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Robotics and Automation Letters (publishedVersion)
- elib (German Aerospace Center) (acceptedVersion)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
- Chalmers Research (Chalmers University of Technology) (submittedVersion)
- Funders
- CHIST-ERA (EP/S032428/1)
- National Centre for Nuclear Robotics (EP/R02572X/1)
- Topics
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Physical Sciences › Engineering › Biomedical Engineering › Soft Robotics and Applications
- Keywords
- grasp planning, benchmarking protocol, robotic grasping, grasp stability, experimental reproducibility, grasp quality metrics, robotic hand, simulation-based assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
