Rosetta: Large Scale System for Text Detection and Recognition in Images
Open accessThe DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
Abstract
In this paper we present a deployed, scalable optical character recognition (OCR) system, which we call Rosetta , designed to process images uploaded daily at Facebook scale. Sharing of image content has become one of the primary ways to communicate information among internet users within social networks such as Facebook, and the understanding of such media, including its textual information, is of paramount importance to facilitate search and recommendation applications. We present modeling techniques for efficient detection and recognition of text in images and describe Rosetta 's system architecture. We perform extensive evaluation of presented technologies, explain useful practical approaches to build an OCR system at scale, and provide insightful intuitions as to why and how certain components work based on the lessons learnt during the development and deployment of the system.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A novel OCR system is developed, named as Rosetta, wherein faster-RCNN is utilized for detecting the text characters from millions of Facebook images, however, fully convolutional CNN is employed for generating the lexicon free transcription of each word [6].”
From Object detection using YOLO: challenges, architectural successors, datasets and applications (2023) · cites this paper as [6]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 3×
“chitectural decision and a natural choice was to use a character-based recognition model. In the initial stages of development, we tested various efficient backbone architectures, including ResNet-18 [17] and SqueezeNet [18]. We evaluated these choices on our synthetic data and ended up selecting ResNet-18 as our main…”
MethodInfluentialFaster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 73,472 · cited 3×
“Our text detection approach, based on Faster-RCNN model [24], is responsible of detecting regions of the image that contain text.”
MethodInfluentialImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · cited 1×
“ns. As we show empirically in Section 5, ShuffleNet is significantly faster than ResNet and did not lead to inferior accuracy. The ShuffleNet convolutional trunk is pre-trained using ImageNet dataset [10]. To train the end-to-end detection system, the model was bootstrapped with an in-house synthetic dataset and then…”
MethodShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
Zhang, X., Zhou, X., Lin, M., & Sun, J. (2017) · arXiv · doi:10.48550/arxiv.1707.01083 · Cited by 9,982 · cited 1×
“e encoding, RPN, and classifiers) are trained jointly in a supervised, end-to-end manner. Our text detection model uses Faster-RCNN, but replaces the ResNet convolutional body with a ShuffleNet-based [31] architecture for efficiency reasons. As we show empirically in Section 5, ShuffleNet is significantly faster than…”
MethodShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
Zhang, X., Zhou, X., Lin, M., & Sun, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00716 · Cited by 9,982
SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
Iandola, F. N., Han, S., Moskewicz, M. W., Ashraf, K., Dally, W. J., & Keutzer, K. (2016) · arXiv · doi:10.48550/arxiv.1602.07360 · Cited by 6,201 · cited 2×
“st of the k fully connected layers that perform character classification in the CHAR model. Orthogonal to this, one can use different body architectures. We experimented with ResNet-18 and SqueezeNet [18], and obtained higher accuracy with ResNet-18 while experiencing only a very small overhead in computation. Image P…”
MethodGraves, A., Fernández, S., Gomez, F., & Schmidhuber, J. (2006) · Proceedings of the 23rd international conference on Machine learning - ICML '06 · doi:10.1145/1143844.1143891 · Cited by 5,370
Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553380 · Cited by 5,366 · cited 1×
“The second approach was based on curriculum learning [6], i.e ., starting with a simpler problem and increasing the difficulty as the model improves.”
BackgroundShi, B., Bai, X., & Yao, C. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2646371 · Cited by 3,172
Edge Boxes: Locating Object Proposals from Edges
Zitnick, C. L., & Dollár, P. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_26 · Cited by 2,656 · cited 1×
“Before the recognition stage, a combination of Edge Boxes [33] and an aggregate channel features (ACF) detector framework [11] is used to detect candidate locations for the words.”
Method
Research works printed but not linked: 1
- [2] Caffe2, 2017. -
Cited by
Object detection using YOLO: challenges, architectural successors, datasets and applications
Diwan, T., Anirudh, G., & Tembhurne, J. V. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-022-13644-y · Cited by 1,549
Towards VQA Models That Can Read
Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., & Rohrbach, M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00851 · Cited by 711
“We also note that the drop in TextVQA for number of answers of a particular answer length is more gradual than in VQA 2.0 which drops sharply after answer length Finally, we analyze the OCR tokens produced by the Rosetta OCR system [6].”
MethodInfluentialWhat Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model Analysis
Baek, J., Kim, G., Lee, J., Park, S., Han, D., Yun, S., Oh, S. J., & Lee, H. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00481 · Cited by 617
“On the other hand, Rosetta [3] removed the BiLSTM to reduce computational complexity and memory consumption.”
BackgroundMethodInfluentialScene Text Detection and Recognition: The Deep Learning Era
Long, S., He, X., & Yao, C. (2021) · International Journal of Computer Vision · doi:10.1007/s11263-020-01369-0 · Cited by 527
“ing such scenario-specific APIs, including Baidu Inc., Tencent Inc. and Megvii Inc.. Recent development of fast and efficient methods [119], [183] has also allowed the deployment of large-scale systems [10]. Companies including Google Inc. and Amazon Inc. are providing text extraction APIs. Despite the success so far, a…”
BackgroundFUNSD: A Dataset for Form Understanding in Noisy Scanned Documents
Jaume, G., Kemal Ekenel, H., & Thiran, J.-P. (2019) · 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW) · doi:10.1109/icdarw.2019.10029 · Cited by 367
“Modern engines that also support handwritten text recognition usually use a Connectionist Temporal Classification (CTC) loss to cope with the alignment problem [18].”
BackgroundScene Text Visual Question Answering
Biten, A. F., Tito, R., Mafla, A., Gomez, L., Rusinol, M., Jawahar, C. V., Valveny, E., & Karatzas, D. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00439 · Cited by 293
“Approaches that use Connectionist Temporal Classification have also been widely used in scene text recognition, in works such as [47, 7, 57, 12, 38], among others.”
MethodSVTR: Scene Text Recognition with a Single Visual Model
Du, Y., Chen, Z., Jia, C., Yin, X., Zheng, T., Li, C., Du, Y., & Jiang, Y.-G. (2022) · Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/124 · Cited by 271
Scene Text Recognition with Permuted Autoregressive Sequence Models
Bautista, D., & Atienza, R. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19815-1_11 · Cited by 267
TextCaps: A Dataset for Image Captioning with Reading Comprehension
Sidorov, O., Hu, R., Rohrbach, M., & Singh, A. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58536-5_44 · Cited by 225
Iterative Answer Prediction With Pointer-Augmented Multimodal Transformers for TextVQA
Hu, R., Singh, A., Darrell, T., & Rohrbach, M. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01001 · Cited by 217
Park, J., Naumov, M., Basu, P., Deng, S., Kalaiah, A., Khudia, D. S., Law, J., Malani, P., Malevich, A., Nadathur, S., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.09886 · Cited by 214
“One of the largest scale object detection systems running in data centers today is the text detection part of the Rosetta system used to understand text in images [8].”
BackgroundSelf-citationVision Transformer for Fast and Efficient Scene Text Recognition
Atienza, R. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-86549-8_21 · Cited by 211
TAP: Text-Aware Pre-training for Text-VQA and Text-Caption
Yang, Z., Lu, Y., Wang, J., Yin, X. P., Florencio, D. A. F., Wang, L., Zhang, C., Zhang, L., & Luo, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00864 · Cited by 209
“In this study, we experiment with Rosetta and the Microsoft Azure OCR system (Microsoft-OCR).”
MethodInfluentialScalable Deep Learning on Distributed Infrastructures: Challenges, Techniques, and Tools
Mayer, R., & Jacobsen, H.-A. (2021) · ACM Computing Surveys · doi:10.1145/3363554 · Cited by 192
“Deep Learning (DL) has recently gained a lot of attention due to its superior performance in tasks like speech recognition [65, 69], optical character recognition [20], and object detection [95].”
BackgroundSequence-to-Sequence Contrastive Learning for Text Recognition
Aberdam, A., Litman, R., Tsiper, S., Anschel, O., Slossberg, R., Mazor, S., Manmatha, R., & Perona, P. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01505 · Cited by 191
Multi-Modal Graph Neural Network for Joint Reasoning on Vision and Scene Text
Gao, D., Li, K., Wang, R., Shan, S., & Chen, X. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01276 · Cited by 124
Beskow, D. M., Kumar, S., & Carley, K. M. (2019) · Information Processing & Management · doi:10.1016/j.ipm.2019.102170 · Cited by 173
SCATTER: Selective Context Attentional Scene Text Recognizer
Litman, R., Anschel, O., Tsiper, S., Litman, R., Mazor, S., & Manmatha, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01198 · Cited by 164
“STR has attracted considerable attention over the past few years [34, 36, 3, 20].”
BackgroundTextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text
Singh, A., Pang, G., Toh, M., Huang, J., Galuba, W., & Hassner, T. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00869 · Cited by 158
Sewer-ML: A Multi-Label Sewer Defect Classification Dataset and Benchmark
Haurum, J. B., & Moeslund, T. B. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.01325 · Cited by 158
TRIE: End-to-End Text Reading and Information Extraction for Document Understanding
Zhang, P., Xu, Y., Cheng, Z., Pu, S., Lu, J., Liang, Q., Niu, Y., & Wu, F. Q. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413900 · Cited by 149
Multi-granularity Prediction for Scene Text Recognition
Wang, P., Da, C., & Yao, C. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19815-1_20 · Cited by 133
MMOCR: A Comprehensive Toolbox for Text Detection, Recognition and Understanding
Kuang, Z., Sun, H., Li, Z., Yue, X., Lin, T. H., Chen, J., Wei, H., Zhu, Y., Gao, T. Y., Zhang, W., et al. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3478328 · Cited by 122
“It initially followed a traditional, step-by-step pipeline comprising the connected component analysis, text line finding, baseline fitting, DB [12] CRAFT [1] EAST [45], DB [12], SAST [31] MaskRCNN [7], PAN [34], PSENet [33] DB [12], TextSnake [18], DRRG [43], FCENet [46] Recognition convention CRNN [24] DB [12] CRNN…”
MethodInfluentialSpatially Aware Multimodal Transformers for TextVQA
Kant, Y., Batra, D., Anderson, P., Schwing, A. G., Parikh, D., Lu, J., & Agrawal, H. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58545-7_41 · Cited by 120
Deep Learning Training in Facebook Data Centers: Design of Scale-up and Scale-out SystemsPreprint
Naumov, M., Kim, J., Mudigere, D., Sridharan, S., Wang, X., Zhao, W., Yilmaz, S., Kim, C., Yuen, H., Ozdal, M., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.09518 · Cited by 117
“Notice that the increase trend is not monotonic because in order to alleviate pressure on the training, inference and serving resources, we are constantly evaluating novel techniques to improve efficiency, such as quantization and compression.”
MethodHuang, K.-H., Chan, H.-P., Fung, M., Qiu, H.-Y., Zhou, M., Joty, S. R., Chang, S.-F., & Ji, H. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3513320 · Cited by 99
Countering Extremists on Social Media: Challenges for Strategic Communication and Content Moderation
Ganesh, B., & Bright, J. (2020) · Policy & Internet · doi:10.1002/poi3.236 · Cited by 83
CLIP4STR: A Simple Baseline for Scene Text Recognition With Pre-Trained Vision-Language Model
Zhao, S., Quan, R., Zhu, L., & Yang, Y. (2024) · IEEE Transactions on Image Processing · doi:10.1109/tip.2024.3512354 · Cited by 109
“Context-free STR methods only utilize the visual features of images, such as CTC-based [40] methods [10], [41], [42], [43], segmentation-based methods [44], [45], [46], and attention-based methods with an encoder-decoder mechanism [47], [48].”
On the Cross-dataset Generalization in License Plate Recognition
Laroca, R., Cardoso, E., Lucio, D. R., Estevam, V., & Menotti, D. (2022) · International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP) · doi:10.5220/0010846800003124 · Cited by 101
“In this work, we evaluate 12 OCR models for LP recognition: RARE (Shi et al., 2016), R2AM (Lee and Osindero, 2016), STAR-Net (Liu et al., 2016), CRNN (Shi et al., 2017), GRCNN (Wang and Hu, 2017), Holistic-CNN (Špaňhel et al., 2017), Multitask (Gonçalves et al., 2019), Rosetta (Borisyuk et al., 2018), TRBA (Baek et a…”
BackgroundInfluentialDeepStore: In-Storage Acceleration for Intelligent Queries
Mailthody, V. S., Qureshi, Z., Liang, W., Feng, Z., de Gonzalo, S. G., Li, Y., Franke, H., Xiong, J., Huang, J., & Hwu, W. (2019) · IEEE/ACM International Symposium on Microarchitecture (MICRO) · doi:10.1145/3352460.3358320 · Cited by 96
“Thanks to recent advancements in deep neural networks (DNNs) and the explosive increase and ubiquitous accessibility of data, deep learning has been enabling intelligent queries in versatile data retrieval applications to improve the quality of data services [20, 58, 59, 85].”
BackgroundTowards Accurate Text-based Image Captioning with Content Diversity Exploration
Xu, G., Niu, S., Tan, M., Luo, Y., Du, Q., & Wu, Q. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01245 · Cited by 62
A survey of text detection and recognition algorithms based on deep learning technology
Wang, X., He, Z.-H., Wang, K., Wang, Y.-F., Zou, L., & Wu, Z. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126702 · Cited by 92
Multi-modal In-Context Learning Makes an Ego-evolving Scene Text Recognizer
Zhao, Z. W., Tang, J., Lin, C., Wu, B., Huang, C., Liu, H., Tan, X., Zhang, Z., & Xie, Y. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01474 · Cited by 92
“In this branch CTC-based [11] methods [5, 24] play the most prominent part.”
MethodStructured Multimodal Attentions for TextVQA
Gao, C., Zhu, Q., Wang, P., Li, H., Liu, Y., van den Hengel, A., & Wu, Q. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3132034 · Cited by 88
“Multi-language version of Rosetta system [10].”
MethodResultA systematic review of object detection from images using deep learning
Kaur, J., & Singh, W. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-15981-y · Cited by 90
HTR-Flor: A Deep Learning System for Offline Handwritten Text Recognition
Neto, A. F. D. S., Bezerra, B. L. D., Toselli, A. H., & Lima, E. B. (2020) · SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI) · doi:10.1109/sibgrapi51738.2020.00016 · Cited by 83
“Since first introduced, CRNN for HTR has been constantly breaking state-of-the-art results and being deployed in industrial application [8].”
BackgroundAttend, Copy, Parse End-to-end Information Extraction from Documents
Palm, R. B., Laws, F., & Winther, O. (2019) · IAPR International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.2019.00060 · Cited by 80
“…on document images, e.g. segmenting documents [Yang et al., 2017, Chen et al., 2017a, Wick and Puppe, 2018], spotting handwritten words [Sudholt and Fink, 2016], classifying documents [Kang et al., 2014] and more broadly detecting text in natural scenes [Liao et al., 2017, Borisyuk et al., 2018].”
BackgroundHuman-Adversarial Visual Question AnsweringPreprint
Sheng, S., Singh, A., Goswami, V., Magana, J. A. L., Galuba, W., Parikh, D., & Kiela, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.02280 · Cited by 77
“To understand if the ability to read text is crucial for AdVQA, we extract Rosetta [6] tokens on the AdVQA val set and determine how many questions can be answered using OCR tokens at an edit distance of 1 to account for OCR errors.”
MethodCascade Reasoning Network for Text-based Visual Question Answering
Liu, F., Xu, G., Wu, Q., Du, Q., Wei, J., & Tan, M. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413924 · Cited by 68
Finding the Evidence: Localization-aware Answer Prediction for Text Visual Question Answering
Han, W., Huang, H., & Han, T. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.278 · Cited by 69
From Strings to Things: Knowledge-Enabled VQA Model That Can Read and Reason
Singh, A. K., Mishra, A., Shekhar, S., & Chakraborty, A. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00470 · Cited by 66
“The modern methods [19, 7, 38, 16, 9] (e) Q.”
BackgroundMultimodal Attention with Image Text Spatial Relationship for OCR-Based Image Captioning
Wang, J., Tang, J., & Luo, J. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413753 · Cited by 64
Efficient, Lexicon-Free OCR using Deep Learning
Namysł, M., & Konya, I. (2019) · IAPR International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.2019.00055 · Cited by 64
“[20] presented a scalable OCR system called Rosetta, which employs a fully convolutional network (FCN) model followed by the CTC layer in order to extract the text depicted on input images uploaded daily at Facebook.”
MethodText Detection and Recognition in the Wild: A ReviewPreprint
Raisi, Z., Naiel, M. A., Fieguth, P., Wardell, S., & Zelek, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.04305 · Cited by 66
“Rosetta [55] used only the extracted features from convolutional neural network by applying a ResNet model as a backbone to predict the feature sequences.”
BackgroundMethodResultInfluentialTowards open-set text recognition via label-to-prototype learning
Liu, C., Yang, C. H., Qin, H., Zhu, X., Liu, C., & Yin, X.-C. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.109109 · Cited by 63
Laroca, R., Araujo, A. B., Junior, L. A. Z., de Almeida, E. C., & Menotti, D. (2021) · IEEE Access · doi:10.1109/access.2021.3077415 · Cited by 59
“In addition to the baselines mentioned above, we trained and evaluated all models for scene text recognition implemented in [51] (i.e., CRNN [28], RARE [52], R2AM [53], STAR-Net [54], GRCNN [55], Rosetta [56] and TRBA [57]), us are also publicly available at https://web.inf.ufpr.br/vri/ publications/amr-unconstrained-…”
Zhang, W., Shi, H., Guo, J., Zhang, S., Cai, Q., Li, J., Luo, S., & Zhuang, Y. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20243 · Cited by 54
OpenViVQA: Task, dataset, and multimodal fusion models for visual question answering in Vietnamese
Nguyen, N. H., Vo, D. T. D., Nguyen, K. V., & Nguyen, N. L.-T. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101868 · Cited by 55
“[19] used the Rossetta system [7] to achieve bounding boxes of scene texts available in images, then they used ROI Alignment to extract region features for each scene text from features map from the backbone of FasterRCNN [42] and they used these features as the features of describing the appearance of scene texts in…”
MethodXu, X., Chen, J., Xiao, J., Gao, L., Shen, F., & Shen, H. T. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01232 · Cited by 54
“Five state-of-the art STR models are selected as the targets for adversarial attack, including three CTCbased models (i.e., CRNN [42], Rosetta [3], STAR-Net [30] ) and two attention-based ones (i.e., RARE [43], TRBA [1]).”
Improving OCR-based Image Captioning by Incorporating Geometrical Relationship
Wang, J., Tang, J., Yang, M., Bai, X., & Luo, J. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00136 · Cited by 54
“tion (OCR) [9, 12, 19, 23, 33] in recent years, OCR-based image captioning still faces challenges and leaves much to be desired.”
BackgroundMethodInfluentialAlam, A., Ullah, I., & Lee, Y.-K. (2020) · IEEE Access · doi:10.1109/access.2020.3017135 · Cited by 52
“(CNN) based approaches have shown performance superiority in tasks like optical character recognition [111], and object detection [112].”
BackgroundMulti-Modal Reasoning Graph for Scene-Text Based Fine-Grained Image Classification and Retrieval
Mafla, A., Dey, S., Biten, A. F., Gómez, L., & Karatzas, D. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00407 · Cited by 52
“Methods that focus on an end-to-end recognition have been explored by [7] based on Faster R-CNN [46], which 1 https://github.com/AndresPMD performs text detection and incorporates a Connectionist Temporal Classification (CTC) [17] to recognize a given text instance.”
MethodBeyond OCR + VQA: Involving OCR into the Flow for Robust and Accurate TextVQA
Zeng, G., Zhang, Y., Zhou, Y., & Yang, X. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475606 · Cited by 50
“Using the same Rosetta-en OCR module, BOV performs better than existing methods including M4C, LaAP-Net [21], SMA [15] and CRN [38] both on validation and test sets, which demonstrates the superiority of our method.”
MethodInfluentialBeyond OCR + VQA: Towards end-to-end reading and reasoning for robust and accurate textvqa
Zeng, G., Zhang, Y., Zhou, Y., Yang, X., Jiang, N., Zhao, G., Wang, W., & Yin, X.-C. (2023) · Pattern Recognition · doi:10.1016/j.patcog.2023.109337 · Cited by 49
A Deep Learning Based Model to Assist Blind People in Their Navigation
Kumar, N., & Jain, A. (2022) · Journal of Information Technology Education Innovations in Practice · doi:10.28945/5006 · Cited by 47
Towards Unified Scene Text Spotting Based on Sequence Generation
Kil, T., Kim, S., Seo, S., Kim, Y., & Kim, D. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01461 · Cited by 47
AED-YOLO11: A small object detection model based on YOLO11
Gong, X., Yu, J., Zhang, H., & Dong, X. (2025) · Digital Signal Processing · doi:10.1016/j.dsp.2025.105411 · Cited by 47
Open-Set Text Recognition via Character-Context Decoupling
Liu, C., Yang, C. H., & Yin, X.-C. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00448 · Cited by 47
“In most conventional text recognition methods [3,8,12,20,34,44,48,55], class centers are mostly modeled as weights in linear classifiers, while visual information and contextual information are modeled together without explicit separation.”
You Only Recognize Once: Towards Fast Video Text Spotting
Cheng, Z., Lu, J., Niu, Y., Pu, S., Wu, F. Q., & Zhou, S. (2019) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3343031.3351093 · Cited by 47
“an be categorized as anchor-based [15,17, 24,26,29,40] and direct-regression-based [14,52,63]. For text recognition, the task is now treated as a sequence recognition problem, in which CTC [10]-based [2,41,53] and attention-based [1,4,5,42,43] methods are designed and have achieved promising results. Recently, there a…”
BackgroundMethodTraditional to transfer learning progression on scene text detection and recognition: a survey
Gupta, N., & Jalal, A. S. (2021) · Artificial Intelligence Review · doi:10.1007/s10462-021-10091-3 · Cited by 46
Localize, Group, and Select: Boosting Text-VQA by Scene Text Modeling
Lu, X., Fan, Z., Wang, Y., Oh, J., & Rosé, C. P. (2021) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw54120.2021.00297 · Cited by 46
“For the text modality, we use two OCR systems, Rosetta OCR [4] and Azure OCR 1 , to recognize scene text.”
MethodFREE: A Fast and Robust End-to-End Video Text Spotter
Cheng, Z., Lu, J., Zou, B., Liang, Q., Xu, Y., Pu, S., Niu, Y., Wu, F. Q., & Zhou, S. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.3038520 · Cited by 45
“For text recognition, the task is now treated as a sequence recognition problem, in which CTC [36]-based [37]–[39] and attention-based [40]–[44] meth-ods are designed and have achieved promising results.”
BackgroundPreSTU: Pre-Training for Scene-Text Understanding
Kil, J., Changpinyo, S., Chen, X., Hu, H., Goodman, S., Chao, W., & Soricut, R. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01401 · Cited by 46
“(Optical Character Recognition) systems [45, 7] and model the visual context using pre-computed object-detection features.”
MethodDonut: Document Understanding Transformer without OCRPreprint
Kim, G., Hong, T., Yim, M., Park, J., Yim, J., Hwang, W., Yun, S., Han, D., & Park, S. (2021) · arXiv (Cornell University) · Cited by 45
Visual Question Answering: A Survey on Techniques and Common Trends in Recent LiteraturePreprint
de Faria, A. C. A. M., Bastos, F. D. C., da Silva, J. V. N. A., Fabris, V. L., Uchôa, V. D. S., Neto, D. G. D. A., & dos Santos, C. F. G. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.11033 · Cited by 45
Are 2D-LSTM really dead for offline text recognition?
Moysset, B., & Messina, R. (2019) · International Journal on Document Analysis and Recognition (IJDAR) · doi:10.1007/s10032-019-00325-0 · Cited by 42
Which OCR toolset is good and why? A comparative study
Jain, P., Taneja, K., & Taneja, H. (2021) · Kuwait Journal of Science · doi:10.48129/kjs.v48i2.9589 · Cited by 42
Text-instance graph: Exploring the relational semantics for text-based visual question answering
Li, X., Wu, B., Song, J., Gao, L., Zeng, P., & Gan, C. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108455 · Cited by 42
Image-based Automatic Dial Meter Reading in Unconstrained Scenarios
Salomon, G., Laroca, R., & Menotti, D. (2022) · Measurement · doi:10.1016/j.measurement.2022.112025 · Cited by 41
A Robust Layout-Independent License Plate Detection and Recognition Model Based on Attention Method
Seo, T.-M., & Kang, D. (2022) · IEEE Access · doi:10.1109/access.2022.3178192 · Cited by 39
Buoy, R., Iwamura, M., Srun, S., & Kise, K. (2023) · IEEE Access · doi:10.1109/access.2023.3332361 · Cited by 38
Data Augmentation for Scene Text Recognition
Atienza, R. (2021) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw54120.2021.00181 · Cited by 40
“Unfortunately, majority of the focus has always Model Input Text Image Baseline Prediction +STRAug Prediction %Acc Gain CRNN [33] Pyice Price 1.30 R2AM [22] OLDTOW OLDTOWN 1.48 GCRNN [36] TiMmES Times 0.89 Rosetta [5] eizu eBizu 2.10 RARE [34] Washii1 Washing 1.35 TRBA [3] insiid inside 1.06 Figure 1.”
MethodInfluentialExploring Sparse Spatial Relation in Graph Inference for Text-Based VQA
Zhou, S., Guo, D., Li, J., Yang, X., & Wang, M. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3310332 · Cited by 40
“LoRRA [46] uses an off-the-shelf OCR system [6] to detect multiple”
Influential2D Positional Embedding-based Transformer for Scene Text Recognition
Raisi, Z., Naiel, M. A., Fieguth, P., Wardell, S., & Zelek, J. (2021) · Journal of Computational Vision and Imaging Systems · doi:10.15353/jcvis.v6i1.3533 · Cited by 35
“Some methods [12–14] have used connectionist temporal classification (CTC) [20], and others [15, 16], adopted an attention mechanism [21] for the prediction of character sequences.”
MethodReinterpreting CTC training as iterative fitting
Li, H., & Wang, W. (2020) · Pattern Recognition · doi:10.1016/j.patcog.2020.107392 · Cited by 35
“Introduction The connectionist temporal classification (CTC) [1] is a method to solve sequence-to-sequence learning, and is widely used in various sequence labeling tasks, such as speech recognition [2, 3, 4], text recognition [5, 6, 7], dynamic gesture recognition [8], sound event detection [9], action labeling [10]…”
MethodFirst-Generation Inference Accelerator Deployment at FacebookPreprint
Anderson, M. J., Chen, B., Chen, S. Y., Deng, S., Fix, J., Gschwind, M. K., Kalaiah, A., Kim, C., Lee, J., Liang, J., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.04140 · Cited by 35
“that contain objects of interest, for example to detect text in images [5].”
BackgroundStacMR: Scene-Text Aware Cross-Modal Retrieval
Mafla, A., de Rezende, R. S., Gómez, L., Larlus, D., & Karatzas, D. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00227 · Cited by 34
“‡ numbers obtained with method from [7].”
ODM: A Text-Image Further Alignment Pre-training Approach for Scene Text Detection and Spotting
Duan, C., Fu, P., Guo, S., Jiang, Q., & Wei, X. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01476 · Cited by 34
“In [2, 7, 12, 19, 20, 25], end-to-end implementation is achieved by training the detection and recognition separately.”
BackgroundOCR post-correction for detecting adversarial text images
Imam, N. H., Vassilakis, V. G., & Kolovos, D. S. (2022) · Journal of Information Security and Applications · doi:10.1016/j.jisa.2022.103170 · Cited by 33
SLOAN: Scale-Adaptive Orientation Attention Network for Scene Text Recognition
Dai, P., Zhang, H., & Cao, X. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.3045602 · Cited by 31
“The two-stage methods [5], [39]–[42] first localize the scene text, and then recognize the cropped images from the detected bounding boxes.”
MethodSeq-UPS: Sequential Uncertainty-aware Pseudo-label Selection for Semi-Supervised Text Recognition
Patel, G., Allebach, J. P., & Qiu, Q. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00612 · Cited by 30
“Furthermore, the framework provides not only the existing methods but also their possible variants [3], and demonstrates that most of the state-of-the-art [14, 33, 37, 53, 54, 64] methods fall under this framework.”
Graph Pooling Inference Network for Text-based VQA
Zhou, S., Guo, D., Yang, X., Dong, J., & Wang, M. (2023) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3634918 · Cited by 30
“As for OCR detection, each OCR token output by OCR systems [5, 38] tf , and W tf 4 are fully connected layers.”
BackgroundMethodInfluentialMultimodal grid features and cell pointers for scene text visual question answering
Gómez, L., Biten, A. F., Tito, R., Mafla, A., Rusiñol, M., Valveny, E., & Karatzas, D. (2021) · Pattern Recognition Letters · doi:10.1016/j.patrec.2021.06.026 · Cited by 29
Arabic Scene Text Recognition in the Deep Learning Era: Analysis on a Novel Dataset
Hassan, H., El-Mahdy, A., & Hussein, M. E. (2021) · IEEE Access · doi:10.1109/access.2021.3100717 · Cited by 29
Wang, Q., Deng, H., Wu, X. M., Yang, Z., Liu, Y., Wang, Y., & Hao, G. (2023) · Neural Networks · doi:10.1016/j.neunet.2023.03.010 · Cited by 29
Separate and Locate: Rethink the Text in Text-based Visual Question Answering
Fang, C., Li, J., Li, L., Ma, C., & Hu, D. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3611753 · Cited by 29
“However, the text in images is recognized by the Optical Character Recognition (OCR) system [6].”
BackgroundConfidence-aware Non-repetitive Multimodal Transformers for TextCaps
Wang, Z., Bao, R., Wu, Q., & Liu, S. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i4.16389 · Cited by 28
Towards Perceiving Small Visual Details in Zero-shot Visual Question Answering with Multimodal LLMsPreprint
Zhang, J., Khayatkhoei, M., Chhikara, P., & Ilievski, F. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.16033 · Cited by 28
“TextVQA provides Optical Character Recognition (OCR) annotations (Borisyuk et al., 2018) which we use to approximate the ground-truth answer bounding box for each question by selecting the OCR bounding box containing the text with the highest string similarity with the human-provided answer.”
MethodAn extended attention mechanism for scene text recognition
Xiao, Z., Nie, Z., Song, C., & Chronopoulos, A. T. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117377 · Cited by 27
COME: Clip-OCR and Master ObjEct for text image captioning
Lv, G., Sun, Y., Nian, F., Zhu, M., Tang, W., & Hu, Z. (2023) · Image and Vision Computing · doi:10.1016/j.imavis.2023.104751 · Cited by 27
FedAWR: An Interactive Federated Active Learning Framework for Air Writing Recognition
Kong, X., Zhang, W., Qu, Y., Yao, X., & Shen, G. (2023) · IEEE Transactions on Mobile Computing · doi:10.1109/tmc.2023.3320147 · Cited by 27
“(cid:2) Rosetta [41] is Facebook’s scalable OCR system.”
BackgroundSTR Transformer: A Cross-domain Transformer for Scene Text Recognition
Wu, X., Tang, B., Zhao, M. L., Wang, J., & Guo, Y. (2022) · Applied Intelligence · doi:10.1007/s10489-022-03728-5 · Cited by 26
“Rosetta (2018) [48] presents a deployed, scalable OCR system, designed to process images uploaded daily at Facebook scale.”
BackgroundInfluentialLightweight Scene Text Recognition Based on Transformer
Luan, X., Zhang, J., Xu, M., Silamu, W., & Li, Y. (2023) · Sensors · doi:10.3390/s23094490 · Cited by 26
2LSPE: 2D Learnable Sinusoidal Positional Encoding using Transformer for Scene Text Recognition
Raisi, Z., Naiel, M. A., Younes, G., Wardell, S., & Zelek, J. (2021) · Canadian Conference on Computer and Robot Vision · doi:10.1109/crv52889.2021.00024 · Cited by 26
Offline Handwritten Chinese Text Recognition with Convolutional Neural NetworksPreprint
Liu, B., Xu, X., & Zhang, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.15619 · Cited by 26
“Resizing images to fixed-length [22, 4, 12] is popular for implementation simplicity, but images with too short or long width will get much distorted in character level.”
BackgroundOptimized YOLOv8 for Automatic License Plate Recognition on Resource Constrained Devices
Satya, B., Manongga, D., Hendry, H., & Aminuddin, A. (2025) · Engineering Technology & Applied Science Research · doi:10.48084/etasr.9983 · Cited by 25
Scene Text Recognition Based on Improved CRNN
Yu, W., Ibrayim, M., & Hamdulla, A. (2023) · Information · doi:10.3390/info14070369 · Cited by 25
Li, Z., Song, J., Qiao, K., Li, C., Zhang, Y., & Li, Z. (2022) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2022.980063 · Cited by 24
“The experimental results are evaluated using the criteria of mAP, weights, GFLOPs, parameters, and accuracy density. mAP is a common measure of neural network accuracy with the model precision measured by mAP@0.5 and mAP@0.5:0.95 (Borisyuk et al., 2018).”
MethodInfluentialA Comparative Study of Attention-Based Encoder-Decoder Approaches to Natural Scene Text Recognition
Cong, F., Hu, W., Huo, Q., & Guo, L. (2019) · IAPR International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.2019.00151 · Cited by 24
“Different from a conventional evaluation metric on isolated word recognition tasks [1], [3]–[6], [28], Word Error Rate (WER) and Character Error Rate (CER) are adopted on sentence recognition tasks.”
MethodRiadh, H., Walha, R., Drira, F., & Zaied, M. (2021) · Multimedia Tools and Applications · doi:10.1007/s11042-021-10663-z · Cited by 24
“Other works use a cascading deep learning for text recognition [11].”
Background
Metadata
- Authors
- Borisyuk, F., Gordo, A., & Sivakumar, V.
- Affiliations
- Meta (United States) (US)
- Venue
- Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining
- Publisher
- ACM
- Published
- 2018-07-19
- Pages
- 71-79
- OpenAlex
- W2809273748
- Cited by
- 353
- References
- 28
- Citations by year
- 2026: 13 · 2025: 35 · 2024: 44 · 2023: 65 · 2022: 51 · 2021: 79 · 2020: 38 · 2019: 26 · 2018: 2
- FWCI
- 10.92
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Handwritten Text Recognition Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Vehicle License Plate Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- optical character recognition, Rosetta, Facebook, system architecture, text extraction from images, media understanding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.