A Comparative Analysis of Object Detection Metrics with a Companion Open-Source Toolkit
Open accessAbstract
Recent outstanding results of supervised object detection in competitions and challenges are often associated with specific metrics and datasets. The evaluation of such methods applied in different contexts have increased the demand for annotated datasets. Annotation tools represent the location and size of objects in distinct formats, leading to a lack of consensus on the representation. Such a scenario often complicates the comparison of object detection methods. This work alleviates this problem along the following lines: (i) It provides an overview of the most relevant evaluation methods used in object detection competitions, highlighting their peculiarities, differences, and advantages; (ii) it examines the most used annotation formats, showing how different implementations may influence the assessment results; and (iii) it provides a novel open-source toolkit supporting different annotation formats and 15 performance metrics, making it easy for researchers to evaluate the performance of their detection algorithms in most known datasets. In addition, this work proposes a new metric, also included in the toolkit, for evaluating object detection in videos that is based on the spatio-temporal overlap between the ground-truth and detected bounding boxes.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Since drone images have aerial views with occlusions, varying sizes and orientations, employing DL models pre-trained on datasets such as Imagenet and MS-COCO might not provide high performance.
“However, since drone images have aerial views with occlusions, varying sizes and orientations, employing DL models pre-trained on datasets such as Imagenet [57] and MS-COCO [78] datasets might not provide high performance.”
From A Systematic Review of Drone Based Road Traffic Monitoring System (2022) · cites this paper as [57]
Using a pre-trained model trained on ground-viewpoint images such as MS-COCO and ImageNet leads to weak performance in detecting vehicles from UAVs because the captured images have different characteristics.
“The use of a pre-trained model, trained on images captured from a ground viewpoint such as MS-COCO [78] and Imagenet [57] datasets, leads to weak performance in the detection of vehicles from UAVs, due to the different characteristics of the captured images.”
From A Systematic Review of Drone Based Road Traffic Monitoring System (2022) · cites this paper as [57]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 56,035 · see citing works
Distinctive Image Features from Scale-Invariant Keypoints
Lowe, D. G. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000029664.99615.94 · Cited by 55,589 · see citing works
You Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803 · see citing works
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.324 · Cited by 30,581 · see citing works
Girshick, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.169 · Cited by 28,105 · see citing works
SURF: Speeded Up Robust Features
Bay, H., Tuytelaars, T., & Van Gool, L. (2006) · Lecture Notes in Computer Science · doi:10.1007/11744023_32 · Cited by 14,955 · see citing works
A Combined Corner and Edge Detector
Harris, C., & Stephens, M. (1988) · Procedings of the Alvey Vision Conference 1988 · doi:10.5244/c.2.23 · Cited by 12,901 · see citing works
EfficientDet: Scalable and Efficient Object Detection
Tan, M., Pang, R., & Le, Q. V. (2019) · arXiv · Cited by 9,841 · cited 1× · see citing works
“75; APS; APM; APL EfficientDet [77] COCO AP@[.”
Shi, J., & Tomasi (1994) · Proceedings of IEEE Conference on Computer Vision and Pattern Recognition CVPR-94 · doi:10.1109/cvpr.1994.323794 · Cited by 7,724 · cited 1× · see citing works
“A local search algorithm using gradients was devised in [29] to solve the image registration problem, which later was expanded to a tracking algorithm [30] for identifying important points in videos.”
MethodThe Pascal Visual Object Classes Challenge: A Retrospective
Everingham, M., Eslami, S. M. A., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-014-0733-5 · Cited by 7,139 · see citing works
Flow-Guided Feature Aggregation for Video Object Detection
Zhu, X., Wang, Y., Dai, J., Yuan, L., & Wei, Y. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.52 · Cited by 729 · cited 2× · see citing works
“Many works in the literature use the mAP as a metric to evaluate the performance of object detection models in video sequences [92–98].”
Background
Flagged first, then by citations
Cited by
Sorted by citations
A Systematic Review of Drone Based Road Traffic Monitoring System
Bisio, I., Garibotto, C., Haleem, H., Lavagetto, F., & Sciarrone, A. (2022) · IEEE Access · doi:10.1109/access.2022.3207282 · Cited by 121 · see citing works
Lema, D. G., Usamentiaga, R., & García, D. F. (2024) · Integration · doi:10.1016/j.vlsi.2023.102127 · Cited by 41 · see citing works
- Authors
- Padilla, R., Passos, W. L., Dias, T. L. B., Netto, S. L., & da Silva, E. A. B.
- Affiliations
- Electrical Engineering Program/Alberto Luiz Coimbra Institute for Post-Graduation and Research in Engineering (PEE/COPPE), PO Box 68504, Rio de Janeiro 21941-972, RJ, Brazil
- Corresponding author
- Padilla, Rafael
- Venue
- Electronics
- Publisher
- MDPI AG
- Published
- 2021-01-25
- Volume
- 10
- Issue
- 3
- Pages
- 279
- ISSN
- 2079-9292 (ISSN-L)
- OpenAlex
- W3124372372
- Cited by
- 735
- References
- 13
- Citations by year
- 2026: 113 · 2025: 149 · 2024: 147 · 2023: 155 · 2022: 128 · 2021: 42 · 2020: 1
- FWCI
- 42.44
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Electronics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- annotation formats, video object detection, benchmark dataset
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
