CoChat
Sign Up
Cited by 363Handwritten Text Recognition TechniquesMultimodal Machine Learning ApplicationsAdvanced Neural Network Applications

DocFormer: End-to-End Transformer for Document Understanding

Open access
Appalaraju, Srikar, Jasani, Bhavan, Kota, Bhargava Urala, Xie, Yusheng, Manmatha, R.·2021·2021 IEEE/CVF International Conference on Computer Vision (ICCV)
DOI 10.1109/iccv48922.2021.00103
Open this paper in CoChat — free accountOpen PDF

Abstract

We present DocFormer - a multi-modal transformer based architecture for the task of Visual Document Understanding (VDU). VDU is a challenging problem which aims to understand documents in their varied formats (forms, receipts etc.) and layouts. In addition, DocFormer is pre-trained in an unsupervised fashion using carefully designed tasks which encourage multi-modal interaction. DocFormer uses text, vision and spatial features and combines them using a novel multi-modal self-attention layer. DocFormer also shares learned spatial embeddings across modalities which makes it easy for the model to correlate text to visual tokens and vice versa. DocFormer is evaluated on 4 different datasets each with strong baselines. DocFormer achieves state-of-the-art results on all of them, sometimes beating models 4x its size (in no. of parameters).

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/iccv48922.2021.00103

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 49

  1. Muhammad Zeshan Afzal, Andreas Kölsch, Sheraz Ahmed, and Marcus Liwicki. Cutting the error by half: Investiga- tion of very deep cnn and advanced training strategies for document image classification. In 2017 14th IAPR Inter- national Conference on Document Analysis and Recognition (ICDAR), volume 1, pages 883-888. IEEE, 2017.
  2. Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R Manmatha. Docformer: End-to-end transformer for document understanding. IEEE/CVF Inter- national Conference on Computer Vision, ICCV, 2021.
  3. Srikar Appalaraju, Yi Zhu, Yusheng Xie, and István Fehérvári. Towards good practices in self-supervised repre- sentation learning. Neural Information Processing Systems (NeurIPS Self-Supervision Workshop 2020), 2020.
  4. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. · arXiv
  5. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. A Simple Framework for Contrastive Learn- ing of Visual Representations. In ICML. 2020.
  6. Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big Self-Supervised Mod- els are Strong Semi-Supervised Learners. arXiv preprint arXiv:2006.10029, 2020. · arXiv
  7. Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved Baselines with Momentum Contrastive Learning. arXiv preprint arXiv:2003.04297, 2020. · arXiv
  8. Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European Conference on Computer Vision, pages 104-120. Springer, 2020.
  9. Gabriela Csurka, Diane Larlus, Albert Gordo, and Jon Al- mazan. What is the right way to represent document images? arXiv preprint arXiv:1603.01076, 2016. · arXiv
  10. Arindam Das, Saikat Roy, Ujjwal Bhattacharya, and Swa- pan K Parui. Document image classification with intra- domain transfer learning and stacked generalization of deep convolutional neural networks. In 2018 24th International Conference on Pattern Recognition (ICPR), pages 3180- 3185. IEEE, 2018.
  11. Tyler Dauphinee, Nikunj Patel, and Mohammad Rashidi. Modular multimodal architecture for document classifica- tion. arXiv preprint arXiv:1912.04376, 2019. · arXiv
  12. Brian Davis, Bryan Morse, Scott Cohen, Brian Price, and Chris Tensmeyer. Deep visual template-free form parsing. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 134-141. IEEE, 2019.
  13. Timo I Denk and Christian Reisswig. Bertgrid: Contextual- ized embedding for 2d document representation and under- standing. arXiv preprint arXiv:1909.04948, 2019. · arXiv
  14. Łukasz Garncarek, Rafał Powalski, Tomasz Stanisławek, Bartosz Topolski, Piotr Halama, and Filip Graliński. Lam- bert: Layout-aware language modeling using bert for infor- mation extraction. arXiv preprint arXiv:2002.08087, 2020. · arXiv
  15. Filip Graliński, Tomasz Stanisławek, Anna Wróblewska, Dawid Lipiński, Agnieszka Kaliska, Paulina Rosalska, Bar- tosz Topolski, and Przemysław Biecek. Kleister: A novel task for information extraction involving long documents with complex layout. arXiv preprint arXiv:2003.02356, 2020. · arXiv
  16. Maroua Hammami, Pierre Héroux, Sébastien Adam, and Vincent Poulain d'Andecy. One-shot field spotting on col- ored forms using subgraph isomorphism. In 2015 13th Inter- national Conference on Document Analysis and Recognition (ICDAR), pages 586-590. IEEE, 2015.
  17. Adam W Harley, Alex Ufkes, and Konstantinos G Derpanis. Evaluation of deep convolutional nets for document image classification and retrieval. In International Conference on Document Analysis and Recognition (ICDAR).
  18. Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum Contrast for Unsupervised Visual Rep- resentation Learning. In CVPR. 2020.
  19. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pages 630-645. Springer, 2016.
  20. Jonathan Herzig, Paweł Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Martin Eisenschlos. Tapas: Weakly supervised table parsing via pre-training. arXiv preprint arXiv:2004.02349, 2020. · arXiv
  21. Teakgyu Hong, DongHyun Kim, Mingi Ji, Won- seok Hwang, Daehyun Nam, and Sungrae Park. Bros: A pre-trained language model for un- derstanding texts in document. under review https://openreview.net/references/pdf?id=uCz3OR6CJT, 2020.
  22. Teakgyu Hong, DongHyun Kim, Mingi Ji, Wonseok Hwang, Daehyun Nam, and Sungrae Park. Bros: A pre- trained language model for understanding texts in document. https://openreview.net/forum?id=punMXQEsPr0, 2020.
  23. Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthe- nis Karatzas, Shijian Lu, and CV Jawahar. Icdar2019 compe- tition on scanned receipt ocr and information extraction. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 1516-1520. IEEE, 2019.
  24. Anoop Raveendra Katti, Christian Reisswig, Cordula Guder, Sebastian Brarda, Steffen Bickel, Johannes Höhne, and Jean Baptiste Faddoul. Chargrid: Towards understanding 2d documents. arXiv preprint arXiv:1809.08799, 2018. · arXiv
  25. Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, and Davide Testuggine. Supervised multimodal bitransformers for clas- sifying images and text. arXiv preprint arXiv:1909.02950, 2019. · arXiv
  26. David Lewis, Gady Agam, Shlomo Argamon, Ophir Frieder, D Grossman, and Jefferson Heard. Building a test collection for complex document information processing. In Proceed- ings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, pages 665-666, 2006.
  27. Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and perfor- mant baseline for vision and language. arXiv preprint arXiv:1908.03557, 2019. · arXiv
  28. Ron Litman, Oron Anschel, Shahar Tsiper, Roee Litman, Shai Mazor, and R Manmatha. Scatter: selective con- text attentional scene text recognizer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11962-11972, 2020.
  29. Devashish Lohani, Abdel Belaïd, and Yolande Belaïd. An invoice reading system using a graph convolutional network. In Asian Conference on Computer Vision, pages 144-158. Springer, 2018.
  30. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. · arXiv
  31. Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic represen- tations for vision-and-language tasks. arXiv preprint arXiv:1908.02265, 2019. · arXiv
  32. Bodhisattwa Prasad Majumder, Navneet Potti, Sandeep Tata, James Bradley Wendt, Qi Zhao, and Marc Najork. Repre- sentation learning for information extraction from form-like documents. In proceedings of the 58th annual meeting of the Association for Computational Linguistics, pages 6495- 6504, 2020.
  33. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An im- perative style, high-performance deep learning library. arXiv preprint arXiv:1912.01703, 2019. · arXiv
  34. Rafał Powalski, Łukasz Borchmann, Dawid Jurkiewicz, Tomasz Dwojak, Michał Pietruszka, and Gabriela Pałka. Go- ing full-tilt boogie on document understanding with text- image-layout transformer. arXiv preprint arXiv:2102.09550, 2021. · arXiv
  35. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. arXiv preprint arXiv:2103.00020, 2021. · arXiv
  36. Pau Riba, Anjan Dutta, Lutz Goldmann, Alicia Fornés, Oriol Ramos, and Josep Lladós. Table detection in invoice docu- ments by graph neural networks. In 2019 International Con- ference on Document Analysis and Recognition (ICDAR), pages 122-127. IEEE, 2019.
  37. Ritesh Sarkhel and Arnab Nandi. Deterministic routing be- tween layout abstractions for multi-scale classification of vi- sually rich documents. In 28th International Joint Confer- ence on Artificial Intelligence (IJCAI), 2019, 2019.
  38. Sebastian Schreiber, Stefan Agne, Ivo Wolf, Andreas Den- gel, and Sheraz Ahmed. Deepdesrt: Deep learning for de- tection and structure recognition of tables in document im- ages. In 2017 14th IAPR international conference on doc- ument analysis and recognition (ICDAR), volume 1, pages 1162-1167. IEEE, 2017.
  39. Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander Alemi. Inception-v4, inception-resnet and the im- pact of residual connections on learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
  40. Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida. Vi- sualmrc: Machine reading comprehension on document im- ages. arXiv preprint arXiv:2101.11272, 2021. · arXiv
  41. Chris Tensmeyer and Tony Martinez. Analysis of convo- lutional neural networks for document image classification. In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), volume 1, pages 388- 393. IEEE, 2017.
  42. Wolf Thomas, Debut Lysandre, Sanh Victor, Chaumond Julien, Delangue Clement, Moi Anthony, Cistac Pierric, Rault Tim, Louf Rémi, Morgan Funtowicz, et al. Hugging- face's transformers: State-of-the-art natural language pro- cessing. arXiv preprint arXiv:1910.03771, 2019. · arXiv
  43. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Il- lia Polosukhin. Attention is all you need. arXiv preprint arXiv:1706.03762, 2017. · arXiv
  44. Andreas Veit, Michael Wilber, and Serge Belongie. Resid- ual networks behave like ensembles of relatively shallow net- works. arXiv preprint arXiv:1605.06431, 2016. · arXiv
  45. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridg- ing the gap between human and machine translation. CoRR, abs/1609.08144, 2016. · arXiv
  46. Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. Layoutlm: Pre-training of text and layout for document image understanding. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1192-1200, 2020.
  47. Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, et al. Layoutlmv2: Multi-modal pre-training for visually-rich document understanding. arXiv preprint arXiv:2012.14740, 2020. · arXiv
  48. Xu Zhong, Jianbin Tang, and Antonio Jimeno Yepes. Pub- laynet: largest dataset ever for document layout analysis. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 1015-1022. IEEE, 2019.
  49. Jun Zhou, Han Yu, Cheng Xie, Hongming Cai, and Lihong Jiang. irmp: From printed forms to relational data model. In 2016 IEEE 18th International Conference on High Perfor- mance Computing and Communications; IEEE 14th Inter- national Conference on Smart City; IEEE 2nd International Conference on Data Science and Systems (HPCC/SmartCi- ty/DSS), pages 1394-1401. IEEE, 2016.

Cited by

Sorted by citations

Authors
Appalaraju, S., Jasani, B., Kota, B. U., Xie, Y., & Manmatha, R.
Affiliations
Amazon (United States) (US)
ORCID
Venue
2021 IEEE/CVF International Conference on Computer Vision (ICCV)
Publisher
IEEE
Published
2021-10
Pages
973-983
DOI
10.1109/iccv48922.2021.00103
OpenAlex
W3176664887
Cited by
363
References
6
Citations by year
2026: 70 · 2025: 85 · 2024: 86 · 2023: 86 · 2022: 34 · 2021: 2
FWCI
16.67
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Handwritten Text Recognition Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
visual document understanding, multimodal transformer, unsupervised pre-training, spatial embedding, document layout analysis, end-to-end learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review