CoChat
Sign Up
Cited by 59Robotic Path Planning AlgorithmsRobot Manipulation and LearningRobotic Mechanisms and Dynamics

MotionBenchMaker: A Tool to Generate and Benchmark Motion Planning Datasets

Open access
Chamzas, Constantinos, Quintero-Pena, Carlos, Kingston, Zachary, Orthey, Andreas, Rakita, Daniel, Gleicher, Michael, Toussaint, Marc, Kavraki, Lydia E.·2022·IEEE Robotics and Automation Letters
DOI 10.1109/lra.2021.3133603
Open this paper in CoChat — free accountOpen PDF

Abstract

Recently, there has been a wealth of development in motion planning for robotic manipulation—new motion planners are continuously proposed, each with their own unique strengths and weaknesses. However, evaluating new planners is challenging and researchers often create their own ad-hoc problems for benchmarking, which is time-consuming, prone to bias, and does not directly compare against other state-of-the-art planners. We presentMotionBenchMaker, an open-source tool to generate benchmarking datasets for realistic robot manipulation problems.MotionBenchMakeris designed to be an extensible, easy-to-use tool that allows users to both generate datasets and benchmark them by comparing motion planning algorithms. Empirically, we show the benefit of usingMotionBenchMakeras a tool to procedurally generate datasets which helps in the fair evaluation of planners. We also present a suite of 40 prefabricated datasets, with 5 different commonly used robots in 8 environments, to serve as a common ground to accelerate motion planning research.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/lra.2021.3133603

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Generating large synthetic datasets has been a problem of great interest in other domains as well, and has also been used as a tool for benchmarking motion planning.

    “Generating large synthetic datasets has been a problem of great interest in other domains as well [71-77], and has also been used as a tool for benchmarking motion planning [78].”

    From MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations (2023) · cites this paper as [78]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 52

  1. [1][1] M. T. Mason, 'Toward robotic manipulation,' Annual Review of Control, Robot., and Autom. Syst. , vol. 1, pp. 1-28, 2018. -
  2. [2][2] J. Leitner, A. W. Tow, N. S¨ underhauf, J. E. Dean, J. W. Durham, M. Cooper, M. Eich, C. Lehnert, R. Mangels, C. McCool et al. , 'The ACRV picking benchmark: A robotic shelf picking benchmark to foster reproducible research,' in IEEE Int. Conf. Robot. Autom. , 2017, pp. 47054712. -
  3. [3][3] Z. Kingston, M. Moll, and L. E. Kavraki, 'Sampling-based methods for motion planning with constraints,' Annual Review of Control, Robot., and Autom. Syst. , vol. 1, no. 1, pp. 159-185, 2018. -
  4. [4][4] C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Lozano-P´ erez, 'Integrated task and motion planning,' Annual Review of Control, Robot., and Autom. Syst. , vol. 4, pp. 265-293, 2021. -
  5. [5][5] A. Orthey and M. Toussaint, 'Section patterns: Efficiently solving narrow passage problems in multilevel motion planning,' IEEE Trans. Robot. , pp. 1-15, 2021. -
  6. [6][6] J. F. Canny, The Complexity of Robot Motion Planning . MIT Press, 1988. -
  7. [7][7] G. Antonelli, 'Robotic research: Are we applying the scientific method?' Frontiers in Robotics and AI , vol. 2, p. 13, 2015. -
  8. [8][8] C. Chamzas, Z. Kingston, C. Quintero-Pe˜ na, A. Shrivastava, and L. E. Kavraki, 'Learning Sampling Distributions Using Local 3D Workspace Decompositions for Motion Planning in High Dimensions,' in IEEE Int. Conf. Robot. Autom. , Jun. 2021. -
  9. [9][9] B. Ichter, J. Harrison, and M. Pavone, 'Learning sampling distributions for robot motion planning,' in IEEE Int. Conf. Robot. Autom. , May 2018, pp. 7087-7094. -
  10. [10][10] B. Chen, B. Dai, Q. Lin, G. Ye, H. Liu, and L. Song, 'Learning to plan in high dimensions via neural exploration-exploitation trees,' in Int. Conf. on Learn. Repr. , 2020. -
  11. [11][11] A. Hornung, K. M. Wurm, M. Bennewitz, C. Stachniss, and W. Burgard, 'OctoMap: An efficient probabilistic 3D mapping framework based on octrees,' Autonomous Robots , vol. 34, no. 3, pp. 189-206, 2013. -
  12. [12][12] M. Quigley, J. Faust, T. Foote, J. Leibs et al. , 'ROS: an open-source Robot Operating System,' in ICRA workshop on open source software , May 2009, p. 5. -
  13. [14][14] Z. Kingston and L. E. Kavraki, 'Robowflex: Robot motion planning with MoveIt made easy,' arXiv preprint arXiv:2103.12826 , 2021. - · arXiv
  14. [15][15] M. Althoff, M. Koschi, and S. Manzinger, 'CommonRoad: Composable benchmarks for motion planning on roads,' in IEEE Intelligent Vehicles Symposium (IV) , 2017, pp. 719-726. -
  15. [16][16] J. Ruiz-Sarmiento, C. Galindo, and J. Gonzalez-Jimenez, 'Robot@home, a robotic dataset for semantic mapping of home environments,' Int. J. of Robotics Research , vol. 36, no. 2, pp. 131-141, 2017. -
  16. [17][17] R. Stern, N. R. Sturtevant, A. Felner, S. Koenig, H. Ma, T. T. Walker, J. Li, D. Atzmon, L. Cohen, T. K. S. Kumar, E. Boyarski, and R. Bartak, 'Multiagent pathfinding: Definitions, variants, and benchmarks,' Symposium on Combinatorial Search (SoCS) , pp. 151-158, 2019. -
  17. [18][18] J. Tani, A. F. Daniele, G. Bernasconi, A. Camus, A. Petrov, A. Courchesne, B. Mehta, R. Suri, T. Zaluska, M. R. Walter et al. , 'Integrated benchmarking and design for reproducible and accessible evaluation of robotic agents,' in IEEE/RSJ Int. Conf. on Intell. Robots and Syst. , 2020, pp. 6229-6236. -
  18. [19][19] D. Perille, A. Truong, X. Xiao, and P. Stone, 'Benchmarking metric ground navigation,' in 2020 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR) , 2020, pp. 116-121. -
  19. [20][20] E. Heiden, L. Palmieri, L. Bruns, K. O. Arras, G. S. Sukhatme, and S. Koenig, 'Bench-MR: A motion planning benchmark for wheeled mobile robots,' IEEE Robot. Autom. Letters , vol. 6, no. 3, pp. 45364543, 2021. -
  20. [21][21] A. Toma, H. Hsueh, H. A. Jaafar, R. Murai, P. H. J. Kelly, and S. Saeedi, 'Pathbench: A benchmarking platform for classical and learned path planning algorithms,' in The Conference on Robots and Vision (CRV2021) , 2021. -
  21. [22][22] M. Moll, I. A. Sucan, and L. E. Kavraki, 'Benchmarking motion planning algorithms: An extensible infrastructure for analysis and visualization,' IEEE Robotics & Automation Magazine , vol. 22, no. 3, pp. 96-102, 2015. -
  22. [23][23] J. Weisz, Y. Huang, F. Lier, S. Sethumadhavan, and P. Allen, 'Robobench: Towards sustainable robotics system benchmarking,' in IEEE Int. Conf. Robot. Autom. IEEE, 2016, pp. 3383-3389. -
  23. [24][24] A. Mandlekar, Y. Zhu, A. Garg, J. Booher, M. Spero, A. Tung, J. Gao, J. Emmons, A. Gupta, E. Orbay et al. , 'Roboturk: A crowdsourcing platform for robotic skill learning through imitation,' in Conf. on Robot Learning , 2018, pp. 879-893. -
  24. [25][25] S. James, Z. Ma, D. R. Arrojo, and A. J. Davison, 'RLBench: The robot learning benchmark learning environment,' IEEE Robot. Autom. Letters , vol. 5, no. 2, pp. 3019-3026, 2020. -
  25. [26][26] Z. Liu, W. Liu, Y. Qin, F. Xiang, M. Gou, S. Xin, M. A. Roa, B. Calli, H. Su, Y. Sun, and P. Tan, 'OCRTOC: A cloud-based competition and benchmark for robotic grasping and manipulation,' 2021, arXiv:2104.11446. - · arXiv
  26. [27][27] S. Dasari, F. Ebert, S. Tian, S. Nair, B. Bucher, K. Schmeckpeper, S. Singh, S. Levine, and C. Finn, 'Robonet: Large-scale multi-robot learning,' in Conf. on Robot Learning . PMLR, 2020, pp. 885-897. -
  27. [28][28] H.-S. Fang, C. Wang, M. Gou, and C. Lu, 'Graspnet-1billion: A largescale benchmark for general object grasping,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2020, pp. 11 444-11 453. -
  28. [29][29] S. Murray, G. D. Konidaris, and D. J. Sorin, 'Roadmap subsampling for changing environments,' in IEEE/RSJ Int. Conf. on Intell. Robots and Syst. , pp. 5664-5670. -
  29. [30][30] A. Suarez, V. M. Vega, M. Fernandez, G. Heredia, and A. Ollero, 'Benchmarks for aerial manipulation,' IEEE Robot. Autom. Letters , vol. 5, no. 2, pp. 2650-2657, 2020. -
  30. [31][31] K. Chatzilygeroudis, B. Fichera, I. Lauzana, F. Bu, K. Yao, F. Khadivar, and A. Billard, 'Benchmark for bimanual robotic manipulation of semideformable objects,' IEEE Robot. Autom. Letters , vol. 5, no. 2, pp. 24432450, 2020. -
  31. [32][32] S. Cruciani, B. Sundaralingam, K. Hang, V. Kumar, T. Hermans, and D. Kragic, 'Benchmarking in-hand manipulation,' IEEE Robot. Autom. Letters , vol. 5, no. 2, pp. 588-595, 2020. -
  32. [33][33] C. Innes and S. Ramamoorthy, 'ProbRobScene: A probabilistic specification language for 3D robotic manipulation environments,' in IEEE Int. Conf. Robot. Autom. , 2021. -
  33. [34][34] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, 'Imagenet: A large-scale hierarchical image database,' in IEEE Conference on Computer Vision and Pattern Recognition , 2009, pp. 248-255. -
  34. [35][35] B. Wu, W. Chen, Y. Fan, Y. Zhang, J. Hou, J. Liu, and T. Zhang, 'Tencent ml-images: A large-scale multi-label image database for visual representation learning,' IEEE Access , vol. 7, pp. 172 683-172 693, 2019. -
  35. [36][36] B. Calli, A. Walsman, A. Singh, S. Srinivasa, P. Abbeel, and A. M. Dollar, 'Benchmarking in manipulation research: Using the Yale-CMU-Berkeley object and model set,' IEEE Robotics & Automation Magazine , vol. 22, no. 3, pp. 36-52, 2015. -
  36. [37][37] I. A. S ¸ ucan, M. Moll, and L. E. Kavraki, 'The Open Motion Planning Library,' IEEE Robot. Autom. Magazine , vol. 19, no. 4, pp. 72-82, 2012. -
  37. [38][38] I. Garcia-Camacho, M. Lippi, M. C. Welle, H. Yin, R. Antonova, A. Varava, J. Borras, C. Torras, A. Marino, G. Alenya et al. , 'Benchmarking bimanual cloth manipulation,' IEEE Robot. Autom. Letters , vol. 5, no. 2, pp. 1111-1118, 2020. -
  38. [39][39] B. Yang, P. E. Lancaster, S. S. Srinivasa, and J. R. Smith, 'Benchmarking robot manipulation with the rubik's cube,' IEEE Robot. Autom. Letters , vol. 5, no. 2, pp. 2094-2099, 2020. -
  39. [40][40] A. Yershova and S. M. LaValle, 'Deterministic sampling methods for spheres and so (3),' in IEEE Int. Conf. Robot. Autom. , vol. 4, 2004, pp. 3974-3980. -
  40. [42][42] S. Hart, P. Dinh, and K. Hambuchen, 'The affordance template ROS package for robot task programming,' in IEEE Int. Conf. Robot. Autom. , 2015, pp. 6227-6234. -
  41. [43][43] E. Pairet, C. Chamzas, Y. R. Petillot, and L. E. Kavraki, 'Path planning for manipulation using experience-driven random trees,' IEEE Robot. Autom. Letters , vol. 6, no. 2, p. 3295-3302, Apr. 2021. -
  42. [44][44] M. Moll, C. Chamzas, Z. Kingston, and L. E. Kavraki, 'HyperPlan: A framework for motion planning algorithm selection and parameter optimization,' in IEEE/RSJ Int. Conf. on Intell. Robots and Syst. , 2021. -
  43. [45][45] C. Quintero-Pe˜ na, A. Kyrillidis, and L. E. Kavraki, 'Robust Optimizationbased Motion Planning for high-DOF Robots under Sensing Uncertainty,' in IEEE Int. Conf. Robot. Autom. , Jun. 2021. -
  44. [46][46] C. Quintero-Pe˜ na, C. Chamzas, V. Unhelkar, and L. E. Kavraki, 'Motion Planning via Bayesian Learning in the Dark,' in ICRA: Workshop on Machine Learning for Motion Planning , Jun. 2021. -
  45. [47][47] A. Orthey, S. Akbar, and M. Toussaint, 'Multilevel motion planning: A fiber bundle formulation,' 2020, arXiv:2007.09435 [cs.RO]. - · arXiv
  46. [48][48] D. Hsu, L. E. Kavraki, J.-C. Latombe, R. Motwani, and S. Sorkin, 'On finding narrow passages with probabilistic roadmap planners,' in Int. Wksp. on the Algorithmic Foundations of Robotics . Springer, 1998. -
  47. [49][49] S. M. LaValle and J. J. Kuffner, 'Rapidly-exploring random trees: Progress and prospects,' in Algorithmic and Computational Robotics: New Directions , 2000, pp. 293-308. -
  48. [50][50] S. Karaman and E. Frazzoli, 'Sampling-based algorithms for optimal motion planning,' Int. J. of Robotics Research , vol. 30, no. 7, pp. 846894, 2011. -
  49. [51][51] J. Gammell, S. S. Srinivasa, and T. D. Barfoot, 'Batch informed trees (BIT*): Sampling-based optimal planning via the heuristically guided search of implicit random geometric graphs,' in IEEE Int. Conf. Robot. Autom. , 2015, pp. 3067-3074. -
  50. [52][52] M. P. Strub and J. D. Gammell, 'Adaptively Informed Trees (AIT*): Fast asymptotically optimal path planning through adaptive heuristics,' in IEEE Int. Conf. Robot. Autom. , 2020, pp. 3191-3198. -
  51. [53][53] D. Hsu, J.-C. Latombe, and R. Motwani, 'Path planning in expansive configuration spaces,' in IEEE Int. Conf. Robot. Autom. , vol. 3. IEEE, 1997, pp. 2719-2726. -
  52. [54][54] I. A. S ¸ ucan and L. E. Kavraki, 'Kinodynamic motion planning by interiorexterior cell exploration,' in Int. Wksp. on the Algorithmic Foundations of Robotics . Springer, 2008.

Cited by

Sorted by citations

Authors
Chamzas, C., Quintero-Pena, C., Kingston, Z., Orthey, A., Rakita, D., Gleicher, M., Toussaint, M., & Kavraki, L. E.
Affiliations
  • Rice University (US)
  • Technische Universität Berlin (DE)
  • University of Wisconsin–Madison (US)
ORCID
Venue
IEEE Robotics and Automation Letters
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2022-04
Volume
7
Issue
2
Pages
882-889
ISSN
  • 2377-3766 (ISSN-L)
  • 2377-3774
DOI
10.1109/lra.2021.3133603
OpenAlex
W4206575475
Cited by
59
References
0
Citations by year
2026: 8 · 2025: 14 · 2024: 16 · 2023: 14 · 2022: 7
FWCI
3.33
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Robotics and Automation Letters (publishedVersion)
  • Rice Research Repository (Rice University) (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
  • Rice University (NSF 1718478, NSF 2008720, NSF-GRFP 1842494)
  • NSF (1830242)
  • Deutsche Forschungsgemeinschaft (EXC 2002/1-390523135)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Robotic Path Planning Algorithms
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Engineering › Control and Systems Engineering › Robotic Mechanisms and Dynamics
Keywords
motion planning, robotic manipulation, benchmarking, dataset construction, open source tools

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review