Segmentation- and Annotation-Free License Plate Recognition With Deep Localization and Failure Identification
Abstract
Automated license plate recognition (ALPR) is essential in several roadway imaging applications. For ALPR systems deployed in the United States, variation between jurisdictions on character width, spacing, and the existence of noise sources (e.g., heavy shadows, non-uniform illumination, various optical geometries, poor contrast, and so on) present in LP images makes it challenging for the recognition accuracy and scalability of ALPR systems. Font and plate-layout variation across jurisdictions further adds to the difficulty of proper character segmentation and increases the level of manual annotation required for training classifiers for each state, which can result in excessive operational overhead and cost. In this paper, we propose a new ALPR workflow that includes novel methods for segmentation- and annotation-free ALPR, as well as improved plate localization and automation for failure identification. Our proposed workflow begins with localizing the LP region in the captured image using a two-stage approach that first extracts a set of candidate regions using a weak sparse network of winnows classifier and then filters them using a strong convolutional neural network (CNN) classifier in the second stage. Images that fail a primary confidence test for plate localization are further classified to identify localization failures, such as LP not present, LP too bright, LP too dark, or no vehicle found. In the localized plate region, we perform segmentation and optical character recognition (OCR) jointly by using a probabilistic inference method based on hidden Markov models (HMMs) where the most likely code sequence is determined by applying the Viterbi algorithm. In order to reduce manual annotation required for training classifiers for OCR, we propose the use of either artificially generated synthetic LP images or character samples acquired by trained ALPR systems already operating in other sites. The performance gap due to differences between training and target domain distributions is minimized using an unsupervised domain adaptation. We evaluated the performance of our proposed methods on LP images captured in several US jurisdictions under realistic conditions.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · see citing works
Histograms of Oriented Gradients for Human Detection
Dalal, N., & Triggs, B. (2005) · 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) · doi:10.1109/cvpr.2005.177 · Cited by 33,584 · see citing works
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684 · see citing works
Multiresolution gray-scale and rotation invariant texture classification with local binary patterns
Ojala, T., Pietikainen, M., & Maenpaa, T. (2002) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2002.1017623 · Cited by 15,398 · see citing works
Selective Search for Object Recognition
Uijlings, J. R. R., van de Sande, K. E. A., Gevers, T., & Smeulders, A. W. M. (2013) · International Journal of Computer Vision · doi:10.1007/s11263-013-0620-5 · Cited by 5,980 · see citing works
Efficient Non-Maximum Suppression
Neubeck, A., & Van Gool, L. (2006) · 18th International Conference on Pattern Recognition (ICPR'06) · doi:10.1109/icpr.2006.479 · Cited by 2,118 · see citing works
Automatic License Plate Recognition (ALPR): A State-of-the-Art Review
Du, S., Ibrahim, M., Shehata, M., & Badawy, W. M. (2012) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2012.2203741 · Cited by 836 · see citing works
Automatic license plate recognition
Chang, S., Chen, L.-S., Chung, Y.-C., & Chen, S.-W. (2004) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2004.825086 · Cited by 795 · see citing works
A License Plate-Recognition Algorithm for Intelligent Transportation System Applications
Anagnostopoulos, C., Anagnostopoulos, I., Loumos, V., et al. (2006) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2006.880641 · Cited by 735 · see citing works
License Plate Recognition From Still Images and Video Sequences: A Survey
Anagnostopoulos, C., Anagnostopoulos, I. E., Psoroulas, I., Loumos, V. G., & Kayafas, E. (2008) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2008.922938 · Cited by 646 · see citing works
A trainable feature extractor for handwritten digit recognition
Lauer, F., Suen, C. Y., & Bloch, G. (2006) · Pattern Recognition · doi:10.1016/j.patcog.2006.10.011 · Cited by 320 · see citing works
An Algorithm for License Plate Recognition Applied to Intelligent Transportation System
Wen, Y., Lu, Y., Yan, J., Zhou, Z., von Deneen, K. M., & Shi, P. (2011) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2011.2114346 · Cited by 306 · see citing works
Principal Visual Word Discovery for Automatic License Plate Detection
Zhou, W., Li, H., Lu, Y., & Tian, Q. (2012) · IEEE Transactions on Image Processing · doi:10.1109/tip.2012.2199506 · Cited by 202 · see citing works
Subspace Interpolation via Dictionary Learning for Unsupervised Domain Adaptation
Ni, J., Qiu, Q., & Chellappa, R. (2013) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2013.95 · Cited by 200 · see citing works
An Iranian License Plate Recognition System Based on Color Features
Ashtari, A. H., Nordin, M. J., & Fathy, M. (2014) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2014.2304515 · Cited by 172 · see citing works
An edge-based color-aided method for license plate detection
Abolghasemi, V., & Ahmadyfard, A. (2008) · Image and Vision Computing · doi:10.1016/j.imavis.2008.10.012 · Cited by 157 · see citing works
Learning-Based License Plate Detection Using Global and Local Features
Zhang, H., Jia, W., He, X., et al. (2006) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2006.758 · Cited by 152 · see citing works
Efficient Embedded Neural-Network-Based License Plate Recognition System
Caner, H., Geçim, H. S., & Alkar, A. Z. (2008) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2008.915524 · Cited by 150 · see citing works
Guo, J.-M., & Liu, Y.-F. (2008) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2007.909284 · Cited by 150 · see citing works
A real time vehicle's license plate recognition system
Rahman, C. A., Badawy, W. M., & Radmanesh, A. (2003) · IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS) · doi:10.1109/avss.2003.1217917 · Cited by 147 · see citing works
Face Detection using Local SMQT Features and Split up Snow Classifier
Nilsson, M., Nordberg, J., & Claesson, I. (2007) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2007.366304 · Cited by 142 · see citing works
Detection and recognition of license plate characters with different appearances
Wang, S.-Z., & Lee, H.-J. (2003) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2003.1252632 · Cited by 125 · see citing works
Vertical-Edge-Based Car-License-Plate Detection Method
Al-Ghaili, A. M., Mashohor, S., Ramli, A. R., et al. (2012) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2012.2222454 · Cited by 121 · see citing works
Unconstrained licence plate and text localization and recognition
Matas, J., & Zimmermann, K. (2005) · IEEE Intelligent Transportation Systems · doi:10.1109/itsc.2005.1520111 · Cited by 85 · see citing works
License Plate Recognition: A Brief Tutorial
Anagnostopoulos, C. (2014) · IEEE Intelligent Transportation Systems Magazine · doi:10.1109/mits.2013.2292652 · Cited by 65 · see citing works
Graphical Models for Joint Segmentation and Recognition of License Plate Characters
Fan, X., & Fan, G. (2008) · IEEE Signal Processing Letters · doi:10.1109/lsp.2008.2008486 · Cited by 52 · see citing works
Extraction and recognition of license plates of motorcycles and vehicles on highways
Lee, H.-J., Chen, S., & Wang, S.-Z. (2004) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2004.1333776 · Cited by 48 · see citing works
Chinese License Plate Localization in Multi-Lane with Complex Background Based on Concomitant Colors
Dun, J., Zhang, S., Ye, X., et al. (2015) · IEEE Intelligent Transportation Systems Magazine · doi:10.1109/mits.2015.2412146 · Cited by 44 · see citing works
A vehicle license plate recognition system based on analysis of maximally stable extremal regions
Li, B., Tian, B., Yao, Q., & Wang, K. (2012) · IEEE International Conference on Networking, Sensing and Control (ICNSC) · doi:10.1109/icnsc.2012.6204952 · Cited by 34 · see citing works
A similarity measure between vector sequences with application to handwritten word image retrieval
Rodríguez-Serrano, J. A., Perronnin, F., Lladós, J., et al. (2009) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2009.5206783 · Cited by 32 · see citing works
Color analysis for Chinese car plate recognition
Xu, J., Shao-fa, L., & Zhibin, C. (2003) · doi:10.1109/rissp.2003.1285782 · Cited by 28 · see citing works
A New Approach of the Vehicle License Plate Location
Xu, H., Fu-hua, Y., Jiao, J.-H., et al. (2005) · International Conference on Parallel and Distributed Computing, Applications and Technologies (PDCAT) · doi:10.1109/pdcat.2005.24 · Cited by 26 · see citing works
New approaches for license plate recognition system
Nathan, V. S. L., Ramkumar, J., & Priya, S. (2004) · doi:10.1109/icisip.2004.1287642 · Cited by 22 · see citing works
A deep reinforcement learning approach to character segmentation of license plate images
Abtahi, F., Zhu, Z., & Burry, A. (2015) · doi:10.1109/mva.2015.7153249 · Cited by 21 · see citing works
A Geometric Framework for Rectangular Shape Detection
Li, Q. (2014) · IEEE Transactions on Image Processing · doi:10.1109/tip.2014.2343456 · Cited by 21 · see citing works
Generative Models for License Plate Recognition by using a Limited Number of Training Samples
Mecocci, A., & Tommaso, C. (2006) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2006.313121 · Cited by 17 · see citing works
Classier training based on synthetically generated samples
Lindner, F., & Kreel, U. (2007) · Cited by 15 · see citing works
Towards Annotation Free License Plate Recognition
Bulan, O., Kozitsky, V., & Burry, A. (2015) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2015.244 · Cited by 15 · see citing works
Self-citationHandwritten Word Image Retrieval with Synthesized Typed Queries
Rodríguez-Serrano, J. A., & Perronnin, F. (2009) · Proceedings of the International Conference on Document Analysis and Recognition · doi:10.1109/icdar.2009.201 · Cited by 13 · see citing works
Image simulation for automatic license plate recognition
Bala, R., Zhao, Y., Burry, A., et al. (2012) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.912453 · Cited by 10 · see citing works
USDOT number localization and recognition from vehicle side-view NIR images
Bulan, O., Wshah, S., Palghat, R., Kozitsky, V., & Burry, A. (2015) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2015.7301301 · Cited by 3 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Zhang, J., & Tao, D. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3039359 · Cited by 774 · see citing works
- Authors
- Bulan, O., Kozitsky, V., Ramesh, P., & Shreve, M.
- Affiliations
- General Motors (United States) (US)
- PARC, Webster, NY, USA
- ORCID
- Bulan, Orhan
- Venue
- IEEE Transactions on Intelligent Transportation Systems
- Published
- 2017-09
- Volume
- 18
- Issue
- 9
- Pages
- 2351-2363
- ISSN
- 1524-9050 (ISSN-L)
- 1558-0016
- OpenAlex
- W2575354312
- Cited by
- 206
- References
- 42
- Citations by year
- 2026: 3 · 2025: 9 · 2024: 12 · 2023: 17 · 2022: 23 · 2021: 39 · 2020: 42 · 2019: 34 · 2018: 23 · 2017: 4
- FWCI
- 23.39
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Intelligent Transportation Systems (publishedVersion)
- Topics
- Physical Sciences › Engineering › Media Technology › Vehicle License Plate Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Handwritten Text Recognition Techniques
- Keywords
- license plate recognition, plate localization, segmentation-free recognition, annotation-free training, failure identification, optical character recognition, hidden Markov models, Viterbi decoding, synthetic image generation, unsupervised domain adaptation, convolutional neural networks, jurisdictional variation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
