PAN++: Towards Efficient and Accurate End-to-End Spotting of Arbitrarily-Shaped Text
The DOI matches its Crossref record; 77 of the 77 references it cites were found in a registry or an index.
Abstract
Scene text detection and recognition have been well explored in the past few years. Despite the progress, efficient and accurate end-to-end spotting of arbitrarily-shaped text remains challenging. In this work, we propose an end-to-end text spotting framework, termed PAN++, which can efficiently detect and recognize text of arbitrary shapes in natural scenes. PAN++ is based on the kernel representation that reformulates a text line as a text kernel (central region) surrounded by peripheral pixels. By systematically comparing with existing scene text representations, we show that our kernel representation can not only describe arbitrarily-shaped text but also well distinguish adjacent text. Moreover, as a pixel-based representation, the kernel representation can be predicted by a single fully convolutional network, which is very friendly to real-time applications. Taking the advantages of the kernel representation, we design a series of components as follows: 1) a computationally efficient feature enhancement network composed of stacked Feature Pyramid Enhancement Modules (FPEMs); 2) a lightweight detection head cooperating with Pixel Aggregation (PA); and 3) an efficient attention-based recognition head with Masked RoI. Benefiting from the kernel representation and the tailored components, our method achieves high inference speed while maintaining competitive accuracy. Extensive experiments show the superiority of our method. For example, the proposed PAN++ achieves an end-to-end text spotting F-measure of 64.9 at 29.2 FPS on the Total-Text dataset, which significantly outperforms the previous best method. Code will be available at: git.io/PAN.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 13×
“In 13 these experiments, we use ResNet18 as the default backbone network.”
MethodInfluentialVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 1×
“In virtue of the powerful representation of deep neural networks [1], [2], scene text detection and recognition has witnessed great progress in the past a few years [3], [4], [5], [6], [7], [8].”
BackgroundHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 12×
“At time step 0 (initial time step), the decoder takes SOS’s feature vector fs and zero LSTM initial states as input.”
BackgroundInfluentialAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“All models are optimized by using ADAM [66] optimizer with a batch size of 16 on 4 GPUs.”
MethodFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472 · cited 4×
“3(f), where an attention-based text recognition head was added on Mask R-CNN to simultaneously detect and recognize irregular text lines.”
MethodInfluentialImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1×
“In these experiments, we use the ResNet [2] pre-trained on ImageNet [65] as the backbone network of our method.”
MethodGradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 2×
““Conv” and “BN” represent regular convolution [49] and Batch Normalization [50], respectively.”
BackgroundFully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
Feature Pyramid Networks for Object Detection
Lin, T.-Y., Dollar, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.106 · Cited by 30,939 · cited 8×
“Second, we compare our feature enhancement network (2- stacked FPEMs) with other widely used methods, including FPN [51], PAFPN [74], and NAS-FPN [75].”
BackgroundMethodInfluential
Research works printed but not linked: 1
- [58] C. Yao, X. Bai, W. Liu, Y. Ma, and Z. Tu, 'Detecting texts of arbitrary orientations in natural images,' in Proc. IEEE Conf. Comp. Vis. Patt. Recogn. , 2012. - · DOI · Paper page
Cited by
Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
Wang, W., Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., & Shao, L. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00061 · Cited by 5,259
Self-citationMPDIoU: A Loss for Efficient and Accurate Bounding Box RegressionPreprint
Ma, S., & Xu, Y. D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.07662 · Cited by 832
On the Arbitrary-Oriented Object Detection: Classification Based Approaches Revisited
Yang, X., & Yan, J. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01593-w · Cited by 195
“SBD [20], SegLink [89], TextField [62], DBNet [33], PAN++ [64], PolarMask++ [65]), model ensemble (e.”
BackgroundSwinTextSpotter: Scene Text Spotting via Better Synergy between Text Detection and Text Recognition
Huang, M., Liu, Y., Peng, Z., Liu, C., Lin, D., Zhu, S., Yuan, N., Ding, K., & Jin, L. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00455 · Cited by 167
DeepSolo: Let Transformer Decoder with Explicit Points Solo for Text Spotting
Ye, M., Zhang, J., Zhao, S., Liu, J., Liu, T., Du, B., & Tao, D. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01854 · Cited by 142
Metadata
- Authors
- Wang, W., Xie, E., Li, X., Liu, X., Liang, D., Zhibo, Y., Lu, T., & Shen, C.
- Affiliations
- Nanjing University (CN)
- University of Hong Kong (HK)
- Nanjing University of Science and Technology (CN)
- Sensetime (China) (CN)
- Monash University (AU)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2021
- Volume
- 44
- Issue
- 9
- Pages
- 1-1
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W3159307593
- Semantic Scholar
- 70aa3857040d2934ab7544988468ede697cb90c9
- Cited by
- 140
- References
- 73
- Citations by year
- 2026: 6 · 2025: 31 · 2024: 47 · 2023: 27 · 2022: 26 · 2021: 3
- FWCI
- 9.26
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Handwritten Text Recognition Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Vehicle License Plate Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- scene text detection, scene text recognition, lightweight detection head, pixel aggregation, real-time inference
- MeSH terms
- Algorithms
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.