The ACRV picking benchmark: A robotic shelf picking benchmark to foster reproducible research
The DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
Robotic challenges like the Amazon Picking Challenge (APC) or the DARPA Challenges are an established and important way to drive scientific progress. They make research comparable on a well-defined benchmark with equal test conditions for all participants. However, such challenge events occur only occasionally, are limited to a small number of contestants, and the test conditions are very difficult to replicate after the main event. We present a new physical benchmark challenge for robotic picking: the ACRV Picking Benchmark. Designed to be reproducible, it consists of a set of 42 common objects, a widely available shelf, and exact guidelines for object arrangement using stencils. A well-defined evaluation protocol enables the comparison of complete robotic systems - including perception and manipulation - instead of sub-systems only. Our paper also describes and reports results achieved by an open baseline system based on a Baxter robot.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vision meets robotics: The KITTI dataset
Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913491297 · Cited by 9,913 · cited 1×
“[8], [9] OpenGrasp [1] YCB [11] Amazon Picking ACRV Picking [10], [12] VisGraB [13] Challenge [6] Benchmark (ours)”
BackgroundThe Pascal Visual Object Classes Challenge: A Retrospective
Everingham, M., Eslami, S. M. A., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-014-0733-5 · Cited by 7,139 · cited 5×
“In parallel to activity within the robotics community, the vision research community has made enormous progress in problems such as object recognition and detection using dataset challenges such as the ImageNet-based ILSVRC [8], COCO [9], and Pascal-VOC [10].”
BackgroundInfluentialVQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
KinectFusion: Real-time dense surface mapping and tracking
Newcombe, R. A., Fitzgibbon, A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A. J., Kohi, P., Shotton, J., & Hodges, S. (2011) · 2011 10th IEEE International Symposium on Mixed and Augmented Reality · doi:10.1109/ismar.2011.6092378 · Cited by 4,008 · cited 1×
“The pipline was split into three distinct stages: a Kinect Fusion (KinFu) [20] approach whilst running a scan motion, a pointcloud-based segmentation algorithm, and a classification stage using a deep convolutional network (CNN).”
MethodEdge Boxes: Locating Object Proposals from Edges
Zitnick, C. L., & Dollár, P. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_26 · Cited by 2,656
VQA: Visual Question Answering: www.visualqa.org
Agrawal, A., Lu, J., Antol, S., Mitchell, M., Zitnick, C. L., Parikh, D., & Batra, D. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0966-6 · Cited by 570
A dataset for improved RGBD-based object detection and pose estimation for warehouse pick-and-place
Rennie, C., Shome, R., Bekris, K. E., & De Souza, A. F. (2016) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2016.2532924 · Cited by 182
(2009) · doi:10.1075/is.10.3.06wis · Cited by 124
Cited by
Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis Approach
Morrison, D., Leitner, J., & Corke, P. (2018) · Robotics: Science and Systems XIV · doi:10.15607/rss.2018.xiv.021 · Cited by 630
Self-citationLearning robust, real-time, reactive robotic grasping
Morrison, D., Corke, P., & Leitner, J. (2020) · The International Journal of Robotics Research · doi:10.1177/0278364919859066 · Cited by 489
Self-citationRLBench: The Robot Learning Benchmark & Learning Environment
James, S., Ma, Z., Arrojo, D. R., & Davison, A. J. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.2974707 · Cited by 405
A Survey on Learning-Based Robotic Grasping
Kleeberger, K., Bormann, R., Kraus, W., & Huber, M. F. (2020) · Current Robotics Reports · doi:10.1007/s43154-020-00021-6 · Cited by 290
Deep Learning Approaches to Grasp Synthesis: A Review
Newbury, R., Gu, M., Chumbley, L., Mousavian, A., Eppner, C., Leitner, J., Bohg, J., Morales, A., Asfour, T., Kragic, D., et al. (2023) · IEEE Transactions on Robotics · doi:10.1109/tro.2023.3280597 · Cited by 255
Self-citation
Metadata
- Authors
- Leitner, J., Tow, A. W., Sunderhauf, N., Dean, J. E., Durham, J. W., Cooper, M., Eich, M., Lehnert, C., Mangels, R., McCool, C., Kujala, P. T., Nicholson, L., Pham, T., Sergeant, J., Wu, L., Zhang, F., Upcroft, B., & Corke, P.
- Venue
- 2017 IEEE International Conference on Robotics and Automation (ICRA)
- Publisher
- IEEE
- Published
- 2017-05
- Pages
- 4705-4712
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W2964137828
- Cited by
- 78
- References
- 8
- FWCI
- 16.81
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The ACRV picking benchmark: A robotic shelf picking benchmark to foster reproducible research. https://app.cochat.ai/papers/10.1109/icra.2017.7989545
- The paper
- Leitner, J., Tow, A. W., Sunderhauf, N., Dean, J. E., Durham, J. W., Cooper, M., Eich, M., Lehnert, C., Mangels, R., McCool, C., Kujala, P. T., Nicholson, L., Pham, T., Sergeant, J., Wu, L., Zhang, F., Upcroft, B., & Corke, P. (2017). The ACRV picking benchmark: A robotic shelf picking benchmark to foster reproducible research. 2017 IEEE International Conference on Robotics and Automation (ICRA). https://doi.org/10.1109/icra.2017.7989545
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.