The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Framework for Performance Evaluation of Face, Text, and Vehicle Detection and Tracking in Video: Data, Metrics, and Protocol
Abstract
Common benchmark data sets, standardized performance metrics, and baseline algorithms have demonstrated considerable impact on research and development in a variety of application domains. These resources provide both consumers and developers of technology with a common framework to objectively compare the performance of different algorithms and algorithmic improvements. In this paper, we present such a framework for evaluating object detection and tracking in video: specifically for face, text, and vehicle objects. This framework includes the source video data, ground-truth annotations (along with guidelines for annotation), performance metrics, evaluation protocols, and tools including scoring software and baseline algorithms. For each detection and tracking task and supported domain, we developed a 50-clip training set and a 50-clip test set. Each data clip is approximately 2.5 minutes long and has been completely spatially/temporally annotated at the I-frame level. Each task/domain, therefore, has an associated annotated corpus of approximately 450,000 frames. The scope of such annotation is unprecedented and was designed to begin to support the necessary quantities of data for robust machine learning approaches, as well as a statistically significant comparison of the performance of algorithms. The goal of this work was to systematically address the challenges of object detection and tracking through a common evaluation framework that permits a meaningful objective comparison of techniques, provides the research community with sufficient data for the exploration of automatic modeling techniques, encourages the incorporation of objective evaluation into the development process, and contributes useful lasting resources of a scale and magnitude that will prove to be extremely useful to the computer vision research community for years to come.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A Computational Approach to Edge Detection
Canny, J. (1986) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.1986.4767851 · Cited by 30,004
Robust Real-Time Face Detection
Viola, P., & Jones, M. J. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000013087.49260.fb · Cited by 14,097
A performance evaluation of local descriptors
Mikolajczyk, K., & Schmid, C. (2005) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2005.188 · Cited by 6,723
Combinatorial Optimization: Algorithms and Complexity.
Johnson, D. S., Papadimitriou, C. H., & Steiglitz, K. (1984) · The American Mathematical Monthly · doi:10.2307/2322374 · Cited by 5,942
Survey over image thresholding techniques and quantitative performance evaluation
Sankur, B. (2004) · Journal of Electronic Imaging · doi:10.1117/1.1631315 · Cited by 4,583
Algorithms for the Assignment and Transportation Problems
Munkres, J. (1957) · Journal of the Society for Industrial and Applied Mathematics · doi:10.1137/0105003 · Cited by 3,986
The FERET evaluation methodology for face-recognition algorithms
Phillips, P. J., Moon, H., Rizvi, S. A., & Rauss, P. J. (2000) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.879790 · Cited by 3,692
Combinatorial optimization:Algorithms and complexity
Rabiner, L. (1984) · IEEE Transactions on Acoustics, Speech, and Signal Processing · doi:10.1109/tassp.1984.1164450 · Cited by 3,576
Detecting faces in images: a survey
Yang, M.-H., Kriegman, D. J., & Ahuja, N. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.982883 · Cited by 3,427
An extended set of Haar-like features for rapid object detection
Lienhart, R., & Maydt, J. (2003) · Proceedings. International Conference on Image Processing · doi:10.1109/icip.2002.1038171 · Cited by 2,977 · cited 2×
“More details of the algorithm can be found in [43], [44].”
Fibonacci heaps and their uses in improved network optimization algorithms
Fredman, M. L., & Tarjan, R. E. (1987) · Journal of the ACM · doi:10.1145/28869.28874 · Cited by 2,656
Overview of the Face Recognition Grand Challenge
Phillips, P. J., Flynn, P. J., Scruggs, T., Bowyer, K. W., Chang, J., Hoffman, K., Marques, J., Min, J., & Worek, W. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.268 · Cited by 2,438
Robust real-time object detection.
Viola, P. A. (2001) · Cited by 2,340
Design and Analysis of Experiments
(2006) · Technometrics · doi:10.1198/tech.2006.s372 · Cited by 1,584
On-road vehicle detection: a review
Sun, Z., Bebis, G., & Miller, R. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.104 · Cited by 1,150
The humanID gait challenge problem: data sets, performance, and analysis
Sarkar, S., Phillips, P. J., Liu, Z., Vega, I. R., Grother, P., & Bowyer, K. W. (2005) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2005.39 · Cited by 1,133
Stanford GraphBase: A Platform for Combinatorial Computing, The
Knuth, D. E. (1993) · Cited by 927
Hjelmås, E., & Low, B. K. (2001) · Computer Vision and Image Understanding · doi:10.1006/cviu.2001.0921 · Cited by 919
Text information extraction in images and video: a survey
Jung, K., In Kim, K., & K. Jain, A. (2004) · Pattern Recognition · doi:10.1016/j.patcog.2003.10.012 · Cited by 846
An experimental comparison of range image segmentation algorithms
Hoover, A., Jean-Baptiste, G., Jiang, X., Flynn, P. J., Bunke, H., Goldgof, D. B., Bowyer, K., Eggert, D. W., Fitzgibbon, A., & Fisher, R. B. (1996) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.506791 · Cited by 790
Self-citationA robust visual method for assessing the relative performance of edge-detection algorithms
Heath, M. D., Sarkar, S., Sanocki, T., & Bowyer, K. W. (1997) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.643893 · Cited by 494
Performance evaluation of fingerprint verification systems
Cappelli, R., Maio, D., Maltoni, D., Wayman, J. L., & Jain, A. K. (2006) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2006.20 · Cited by 372
Performance evaluation of object detection algorithms for video surveillance
Nascimento, J. C., & Marques, J. S. (2006) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2006.876287 · Cited by 283
Tools and techniques for video performance evaluation
Doermann, D., & Mihalcik, D. (2002) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2000.902888 · Cited by 190
(2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-69568-4 · Cited by 163
Evaluating Multi-Object Tracking
Smith, K., Gatica-Perez, D., Odobez, J., & Ba, S. (2006) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops · doi:10.1109/cvpr.2005.453 · Cited by 162
Fast vehicle detection with probabilistic feature grouping and its application to vehicle tracking
Kim, & Malik (2003) · Proceedings Ninth IEEE International Conference on Computer Vision · doi:10.1109/iccv.2003.1238392 · Cited by 159
Stiefelhagen, R., Bernardin, K., Bowers, R., Garofolo, J., Mostefa, D., & Soundararajan, P. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-69568-4_1 · Cited by 158
Self-citationPETS Metrics: On-Line Performance Evaluation Service
Young, D. P., & Ferryman, J. M. (2006) · IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance (VS-PETS) · doi:10.1109/vspets.2005.1570931 · Cited by 113
VISATRAM: a real-time vision system for automatic traffic monitoring
Zhu, Z., Xu, G., Yang, B., Shi, D., & Lin, X. (2000) · Image and Vision Computing · doi:10.1016/s0262-8856(99)00046-3 · Cited by 100
An Automatic Performance Evaluation Protocol for Video Text Detection Algorithms
Hua, X., Wenyin, L., & Zhang, H.-J. (2004) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2004.825538 · Cited by 91
Multi-feature Graph-Based Object Tracking
Taj, M., Maggio, E., & Cavallaro, A. (2007) · Springer eBooks · doi:10.1007/978-3-540-69568-4_15 · Cited by 37
Multiple Vehicle Tracking in Surveillance Videos
Zhai, Y., Berkowitz, P., Miller, A., et al. (2007) · Springer eBooks · doi:10.1007/978-3-540-69568-4_16 · Cited by 8
Cited by
Visual Tracking: An Experimental Survey
(2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.230 · Cited by 1,513
ICDAR 2013 Robust Reading Competition
Karatzas, D., Shafait, F., Uchida, S., Iwamura, M., Bigorda, L. G. I., Mestre, S. R., Mas, J., Mota, D. F., Almazan, J. A., & de las Heras, L. P. (2013) · 2013 12th International Conference on Document Analysis and Recognition · doi:10.1109/icdar.2013.221 · Cited by 1,360
MOT16: A Benchmark for Multi-Object Tracking
Milan, A., Leal-Taixe, L., Reid, I., Roth, S., & Schindler, K. (2016) · arXiv · doi:10.48550/arxiv.1603.00831 · Cited by 1,309
Multiple object tracking: A literature review
Luo, W., Xing, J., Milan, A., Zhang, X., Liu, W., & Kim, T.-K. (2021) · Artificial Intelligence · doi:10.1016/j.artint.2020.103448 · Cited by 1,099
Multiple Object Tracking Using K-Shortest Paths Optimization
Berclaz, J., Fleuret, F., Turetken, E., & Fua, P. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.21 · Cited by 1,015
Metadata
- Authors
- Kasturi, R., Goldgof, D., Soundararajan, P., Manohar, V., Garofolo, J., Bowers, R., Boonstra, M., Korzhova, V., & Zhang, J.
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2009-02
- Volume
- 31
- Issue
- 2
- Pages
- 319-336
- ISSN
- 0162-8828 (ISSN-L)
- OpenAlex
- W2141822239
- Cited by
- 497
- References
- 33
- FWCI
- 40.31
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.