Estimating Labeling Quality with Deep Object Detectors
Abstract
Deep Learning methods are widely applied in Robotics and Automated Driving scenarios. The task of perception for Automated Driving in the real world is particularly challenging and requires a sufficient amount of high quality labeled training data for the algorithms to perform well. However, the means of obtaining real world datasets are limited. It is common practice to have human labelers involved at least to some extent. Regardless of whether the process is partially automated or not, these labels never represent perfectly accurate ground-truth. By investigating the recognition performance of a state-of-the-art object detector as a function of the quality of a labeled real world training set, we study the effect of labeling errors of various types and severity. To this end, the given labels are treated as a reference to which synthetic errors are added systematically in order to determine the performance of the object detector if trained on the erroneous dataset.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 56,035 · see citing works
You Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803 · see citing works
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684 · see citing works
(2016) · Cited by 31,590 · see citing works
Girshick, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.169 · Cited by 28,105 · see citing works
YOLO9000: Better, Faster, Stronger
Redmon, J., & Farhadi, A. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.690 · Cited by 19,892 · see citing works
The pascal visual object classes (VOC) challenge
Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2010) · International Journal of Computer Vision · doi:10.1007/s11263-009-0275-4 · Cited by 19,599 · see citing works
Are we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897 · see citing works
Vision meets robotics: The KITTI dataset
Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913491297 · Cited by 9,913 · see citing works
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2858826 · Cited by 9,556 · see citing works
Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553380 · Cited by 5,366 · cited 1× · see citing works
“In Curriculum learning [21] training examples are ordered in a meaningful way to improve training progress.”
BackgroundDeepFool: A Simple and Accurate Method to Fool Deep Neural Networks
Moosavi-Dezfooli, S.-M., Fawzi, A., & Frossard, P. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.282 · Cited by 5,317 · see citing works
Deep neural networks are easily fooled: High confidence predictions for unrecognizable images
Nguyen, A., Yosinski, J., & Clune, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298640 · Cited by 3,258 · see citing works
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838 · see citing works
Adversarial machine learning at scale
Kurakin, A., Goodfellow, I., & Bengio, S. (2016) · arXiv · doi:10.48550/arxiv.1611.01236 · Cited by 2,325 · see citing works
Joint 3D Proposal Generation and Object Detection from View Aggregation
Ku, J., Mozifian, M., Lee, J., Harakeh, A., & Waslander, S. L. (2018) · 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8594049 · Cited by 1,697 · see citing works
Self-Paced Learning for Latent Variable Models
Kumar, M., Packer, B., & Koller, D. (2010) · neural information processing systems · Cited by 1,204 · see citing works
Learning to Reweight Examples for Robust Deep LearningPreprint
Ren, M., Zeng, W., Yang, B., & Urtasun, R. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.09050 · Cited by 1,091 · see citing works
MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted LabelsPreprint
Jiang, L., Zhou, Z., Leung, T., Li, L.-J., & Fei-Fei, L. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.05055 · Cited by 834 · see citing works
The ApolloScape Dataset for Autonomous Driving
Huang, X., Cheng, X., Geng, Q., Cao, B., Zhou, D., Wang, P., Lin, Y., & Yang, R. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2018.00141 · Cited by 679 · see citing works
Deep Learning is Robust to Massive Label Noise
Rolnick, D., Veit, A., Belongie, S., & Shavit, N. (2017) · arXiv · doi:10.48550/arxiv.1705.10694 · Cited by 402 · see citing works
MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels
Jiang, L., Zhou, Z., Leung, T., Li, L.-J., & Fei-Fei, L. (2018) · International Conference on Machine Learning · Cited by 371 · see citing works
Iterative Learning with Open-set Noisy Labels
Wang, Y., Liu, W., Ma, X., Bailey, J., Hongyuan Zha, B., Song, L., & Xia, S. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00906 · Cited by 339 · see citing works
Leveraging Pre-Trained 3D Object Detection Models for Fast Ground Truth Generation
Lee, J., Walsh, S., Harakeh, A., & Waslander, S. L. (2018) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2018.8569793 · Cited by 33 · see citing works
Flagged first, then by citations
1 of the 25 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Feng, D., Haase-Schutz, C., Rosenbaum, L., Hertlein, H., Glaser, C., Timm, F., Wiesbeck, W., & Dietmayer, K. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.2972974 · Cited by 1,425 · see citing works
Self-citationWillers, O., Sudholt, S., Raafatnia, S., & Abrecht, S. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-55583-2_25 · Cited by 88 · see citing works
Labels are Not Perfect: Inferring Spatial Uncertainty in Object Detection
Feng, D., Wang, Z., Zhou, Y., Rosenbaum, L., Timm, F., Dietmayer, K., Tomizuka, M., & Zhan, W. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3096943 · Cited by 35 · see citing works
Leveraging Uncertainties for Deep Multi-modal Object Detection in Autonomous Driving
Feng, D., Cao, Y., Rosenbaum, L., Timm, F., & Dietmayer, K. (2020) · IEEE Intelligent Vehicles Symposium · doi:10.1109/iv47402.2020.9304551 · Cited by 33 · see citing works
Inferring Spatial Uncertainty in Object Detection
Wang, Z., Feng, D., Zhou, Y., Rosenbaum, L., Timm, F., Dietmayer, K., Tomizuka, M., & Zhan, W. (2020) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9340798 · Cited by 27 · see citing works
Iterative Label Improvement: Robust Training by Confidence Based Filtering and Dataset Partitioning
Haase-Schutz, C., Stal, R., Hertlein, H., & Sick, B. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9411918 · Cited by 10 · see citing works
Self-citationMohammed, S. A. K., Rahman, A. H. A., Bakar, M. A., & Razak, M. Z. A. (2024) · IEEE International Conference on Artificial Intelligence in Engineering and Technology (IICAIET) · doi:10.1109/iicaiet62352.2024.10730667 · Cited by 1 · see citing works
MLLM-Based Evaluation and Enhancement of Open-Set Object Detection Dataset Annotations
Gao, X., Nong, Q., Dong, X., & Gao, H. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-5722-6_35
- Authors
- Haase-Schütz, C., Hertlein, H., & Wiesbeck, W.
- Affiliations
- Institute of Radio Frequency Engineering and EIectronics, Karlsruhe Institute of Technology, Karlsruhe, 76131, Germany
- Robert Bosch (Germany) (DE)
- Karlsruhe Institute of Technology (DE)
- ORCID
- Wiesbeck, Werner
- Venue
- 2019 IEEE Intelligent Vehicles Symposium (IV)
- Publisher
- IEEE
- Published
- 2019-06
- Pages
- 33-38
- ISSN
- 1931-0587 (ISSN-L)
- OpenAlex
- W2971072340
- Cited by
- 8
- References
- 25
- Citations by year
- 2026: 1 · 2024: 1 · 2021: 2 · 2020: 4
- FWCI
- 0.49
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE Intelligent Vehicles Symposium (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- object detection, autonomous driving, label errors, training data quality, perception
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
