The DOI matches its Crossref record; 65 of the 65 references it cites were found in a registry or an index.
TrOCR: Transformer-Based Optical Character Recognition with Pre-trained Models
Open accessAbstract
Text recognition is a long-standing research problem for document digitalization. Existing approaches are usually built based on CNN for image understanding and RNN for char-level text generation. In addition, another language model is usually needed to improve the overall accuracy as a post-processing step. In this paper, we propose an end-to-end text recognition approach with pre-trained image Transformer and text Transformer models, namely TrOCR, which leverages the Transformer architecture for both image understanding and wordpiece-level text generation. The TrOCR model is simple but effective, and can be pre-trained with large-scale synthetic data and fine-tuned with human-labeled datasets. Experiments show that the TrOCR model outperforms the current state-of-the-art models on the printed, handwritten and scene text recognition tasks. The TrOCR models and code are publicly available at https://aka.ms/trocr.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Randaugment: Practical automated data augmentation with a reduced search space
Cubuk, E. D., Zoph, B., Shlens, J., & Le, Q. V. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00359 · Cited by 3,223
RandAugment: Practical automated data augmentation with a reduced search space
Cubuk, E. D., Zoph, B., Shlens, J., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1909.13719 · Cited by 3,223
Shi, B., Bai, X., & Yao, C. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2646371 · Cited by 3,172
Kudo, T., & Richardson, J. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/d18-2012 · Cited by 2,869
Zero-shot text-to-image generation
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., & Sutskever, I. (2021) · arXiv · doi:10.48550/arxiv.2102.12092 · Cited by 2,417
fairseq: A Fast, Extensible Toolkit for Sequence Modeling
Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., Grangier, D., & Auli, M. (2019) · Proceedings of the 2019 Conference of the North · doi:10.18653/v1/n19-4009 · Cited by 2,344
Synthetic Data for Text Localisation in Natural Images
Gupta, A., Vedaldi, A., & Zisserman, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.254 · Cited by 1,563
ICDAR 2015 competition on Robust Reading
Karatzas, D., Gomez-Bigorda, L., Nicolaou, A., Ghosh, S., Bagdanov, A., Iwamura, M., Matas, J., Neumann, L., Chandrasekhar, V. R., Lu, S., et al. (2015) · 2015 13th International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.2015.7333942 · Cited by 1,547
BEiT: BERT Pre-Training of Image Transformers
Bao, H., Dong, L., Piao, S., & Wei, F. (2021) · arXiv · doi:10.48550/arxiv.2106.08254 · Cited by 1,392
Self-citationICDAR 2013 Robust Reading Competition
Karatzas, D., Shafait, F., Uchida, S., Iwamura, M., Bigorda, L. G. I., Mestre, S. R., Mas, J., Mota, D. F., Almazan, J. A., & de las Heras, L. P. (2013) · 2013 12th International Conference on Document Analysis and Recognition · doi:10.1109/icdar.2013.221 · Cited by 1,360
Unified Language Model Pre-training for Natural Language Understanding and Generation
Dong, L., Yang, N., Wang, W., Wei, F., Liu, X., Wang, Y., Gao, J., Zhou, M., & Hon, H.-W. (2019) · arXiv · doi:10.48550/arxiv.1905.03197 · Cited by 1,225
Self-citationEnd-to-end scene text recognition
Wang, K., Babenko, B., & Belongie, S. (2011) · 2011 International Conference on Computer Vision · doi:10.1109/iccv.2011.6126402 · Cited by 1,060
Neural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1162 · Cited by 974
Neural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2015) · arXiv · doi:10.48550/arxiv.1508.07909 · Cited by 950
ASTER: An Attentional Scene Text Recognizer with Flexible Rectification
Shi, B., Yang, M., Wang, X., Lyu, P., Yao, C., & Bai, X. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2848939 · Cited by 906
Real-Time Scene Text Detection with Differentiable Binarization
Liao, M., Wan, Z., Yao, C., Chen, K., & Bai, X. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6812 · Cited by 864
Synthetic Data and Artificial Neural Networks for Natural Scene Text Recognition
Jaderberg, M., Simonyan, K., Vedaldi, A., & Zisserman, A. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1406.2227 · Cited by 726
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., & Zhou, M. (2020) · arXiv · doi:10.48550/arxiv.2002.10957 · Cited by 703
Self-citationRobust Scene Text Recognition with Automatic Rectification
Shi, B., Wang, X., Lyu, P., Yao, C., & Bai, X. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.452 · Cited by 693
What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model Analysis
Baek, J., Kim, G., Lee, J., Park, S., Han, D., Yun, S., Oh, S. J., & Lee, H. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00481 · Cited by 617
Handwritten Optical Character Recognition (OCR): A Comprehensive Systematic Literature Review (SLR)
Memon, J., Sami, M., Khan, R. A., & Uddin, M. (2020) · IEEE Access · doi:10.1109/access.2020.3012542 · Cited by 556
Dropout improves Recurrent Neural Networks for Handwriting RecognitionPreprint
(2016) · Cited by 539
A robust arbitrary text detection system for natural scene images
Risnumawan, A., Shivakumara, P., Chan, C. S., & Tan, C. L. (2014) · Expert Systems with Applications · doi:10.1016/j.eswa.2014.07.008 · Cited by 531
Recognizing Text with Perspective Distortion in Natural Scenes
Phan, T. Q., Shivakumara, P., Tian, S., & Tan, C. L. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.76 · Cited by 473
Fang, S., Xie, H., Wang, Y., Mao, Z., & Zhang, Y. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00702 · Cited by 450
Graves, A., Fernández, S., & Gomez, F. (2017) · mediaTUM (Technical University of Munich) · Cited by 386
Towards Accurate Scene Text Recognition With Semantic Reasoning Networks
Yu, D., Li, X., Zhang, C., Liu, T., Han, J., Liu, J., & Ding, E. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01213 · Cited by 385
Self-citationTop-down and bottom-up cues for scene text recognition
Mishra, A., Alahari, K., & Jawahar, C. V. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6247990 · Cited by 335
Are Multidimensional Recurrent Layers Really Necessary for Handwritten Text Recognition?
Puigcerver, J. (2017) · 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.2017.20 · Cited by 332
Dropout Improves Recurrent Neural Networks for Handwriting Recognition
Pham, V., Bluche, T., Kermorvant, C., & Louradour, J. (2014) · International Conference on Frontiers in Handwriting Recognition · doi:10.1109/icfhr.2014.55 · Cited by 314
ESIR: End-To-End Scene Text Recognition via Iterative Image Rectification
Zhan, F., & Lu, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00216 · Cited by 296
Decoupled Attention Network for Text Recognition
Wang, T., Zhu, Y., Jin, L., Luo, C., Chen, X., Wu, Y., Wang, Q., & Cai, M. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6903 · Cited by 294
SVTR: Scene Text Recognition with a Single Visual Model
Du, Y., Chen, Z., Jia, C., Yin, X., Zheng, T., Li, C., Du, Y., & Jiang, Y.-G. (2022) · Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/124 · Cited by 271
Scene Text Recognition with Permuted Autoregressive Sequence Models
Bautista, D., & Atienza, R. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-19815-1_11 · Cited by 267
Handwriting Recognition with Large Multidimensional Long Short-Term Memory Recurrent Neural Networks
Voigtlaender, P., Doetsch, P., & Ney, H. (2016) · 2016 15th International Conference on Frontiers in Handwriting Recognition (ICFHR) · doi:10.1109/icfhr.2016.0052 · Cited by 211
Vision Transformer for Fast and Efficient Scene Text Recognition
Atienza, R. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-86549-8_21 · Cited by 211
NRTR: A No-Recurrence Sequence-to-Sequence Model for Scene Text Recognition
Sheng, F., Chen, Z., & Xu, B. (2019) · 2019 International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.2019.00130 · Cited by 207
TextScanner: Reading Characters in Order for Robust Scene Text Recognition
Wan, Z., He, M., Chen, H., Bai, X., & Yao, C. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6891 · Cited by 190
From Two to One: A New Scene Text Recognizer with Visual Language Modeling Network
Wang, Y., Xie, H., Fang, S., Wang, J., Zhu, S., & Zhang, Y. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01393 · Cited by 183
Training data-efficient image transformers & distillation through attention
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., & Jégou, H. (2020) · arXiv · doi:10.48550/arxiv.2012.12877 · Cited by 182
RobustScanner: Dynamically Enhancing Positional Clues for Robust Text Recognition
Yue, X., Kuang, Z., Lin, C., Sun, H., & Zhang, W. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58529-7_9 · Cited by 180
Joint Line Segmentation and Transcription for End-to-End Handwritten Paragraph RecognitionPreprint
Bluche, T. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1604.08352 · Cited by 178
Evaluating Sequence-to-Sequence Models for Handwritten Text Recognition
Michael, J., Labahn, R., Grüning, T., & Zollner, J. (2019) · IAPR International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.2019.00208 · Cited by 170
Pay attention to what you read: Non-recurrent handwritten text-Line recognition
Kang, L., Riba, P., Rusiñol, M., Fornés, A., & Villegas, M. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.108766 · Cited by 170
On Recognizing Texts of Arbitrary Shapes with 2D Self-Attention
Lee, J., Park, S., Baek, J., Oh, S. J., Kim, S., & Lee, H. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00281 · Cited by 168
SCATTER: Selective Context Attentional Scene Text Recognizer
Litman, R., Anschel, O., Tsiper, S., Litman, R., Mazor, S., & Manmatha, R. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01198 · Cited by 164
Gated Convolutional Recurrent Neural Networks for Multilingual Handwriting Recognition
Bluche, T., & Messina, R. (2017) · IAPR International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.2017.111 · Cited by 156
PlugNet: Degradation Aware Scene Text Recognition Supervised by a Pluggable Super-Resolution Unit
Mou, Y., Tan, L., Hui, Y., Chen, J., Liu, L., Yan, R., & Huang, Y. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58555-6_10 · Cited by 122
Self-citationRethinking Text Line Recognition ModelsPreprint
Diaz, D. H., Qin, S., Ingle, R. R., Fujii, Y., & Bissacco, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.07787 · Cited by 87
Reading scene text with fully convolutional sequence modeling
Gao, Y., Chen, Y., Wang, J., Tang, M., & Lu, H. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.01.094 · Cited by 73
Joint Visual Semantic Reasoning: Multi-Stage Decoder for Text Recognition
Bhunia, A. K., Sain, A., Kumar, A., Ghose, S., Chowdhury, P. N., & Song, Y.-Z. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01467 · Cited by 69
AutoSTR: Efficient Backbone Search for Scene Text Recognition
Zhang, H., Yao, Q., Yang, M., Xu, Y., & Bai, X. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58586-0_44 · Cited by 68
Generating Synthetic Data for Text RecognitionPreprint
Krishnan, P., & Jawahar, C. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.04224 · Cited by 35
Towards the Unseen: Iterative Text Recognition by Distilling from Errors
Bhunia, A. K., Chowdhury, P. N., Sain, A., & Song, Y.-Z. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01468 · Cited by 26
SaHAN: Scale-aware hierarchical attention network for scene text recognition
Zhang, J., Luo, C., Jin, L., Wang, T., Li, Z., & Zhou, W. (2020) · Pattern Recognition Letters · doi:10.1016/j.patrec.2020.06.009 · Cited by 20
Self-citationScene Text Recognition via Gated Cascade Attention
Wang, S., Wang, Y., Qin, X., Zhao, Q., & Tang, Z. (2019) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2019.00179 · Cited by 19
Representation and Correlation Enhanced Encoder-Decoder Framework for Scene Text Recognition
Cui, M., Wang, W., Zhang, J., & Wang, L. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86337-1_11 · Cited by 16
Bidirectional Scene Text Recognition with a Single Decoder
Bleeker, M., & de Rijke, M. (2019) · UvA-DARE (University of Amsterdam) · doi:10.48550/arxiv.1912.03656 · Cited by 15
Sang, D. V., & Cuong, L. T. B. (2019) · doi:10.1145/3368926.3369689 · Cited by 15
Revisiting Classification Perspective on Scene Text RecognitionPreprint
Cai, H., Sun, J., & Xiong, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.10884 · Cited by 15
Baek, J., Matsui, Y., & Aizawa, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.04400 · Cited by 6
Cited by
Benchmarking large language models for handwritten text recognition
Crosilla, G., Klic, L., & Colavizza, G. (2025) · Journal of Documentation · doi:10.1108/jd-03-2025-0082 · Cited by 34
GPT-4 as an Effective Zero-Shot Evaluator for Scientific Figure Captions
Hsu, T.-Y., Huang, C.-Y., Rossi, R., Kim, S., Giles, C., & Huang, T.-H. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.363 · Cited by 8
Human-in-the-loop tabular data extraction methods for historical climate data rescue
Vercruysse, B., Birkholz, J. M., Chandrasekar, K. K. T., Muheki, D., Thiery, W., Verbeeck, H., Hufkens, K., Jacobsen, K., & Verbruggen, C. (2026) · International Journal on Document Analysis and Recognition (IJDAR) · doi:10.1007/s10032-025-00524-y · Cited by 5
Metadata
- Authors
- Li, M., Lv, T., Chen, J., Cui, L., Lu, Y., Florencio, D., Zhang, C., Li, Z., & Wei, F.
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2023-06-26
- Volume
- 37
- Issue
- 11
- Pages
- 13094-13102
- ISSN
- 2159-5399 (ISSN-L)
- 2374-3468
- OpenAlex
- W4382202677
- Cited by
- 573
- References
- 65
- FWCI
- 64.24
- Type
- conference paper
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.