The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index.
Towards VQA Models That Can Read
Open accessAbstract
Studies have shown that a dominant class of questions asked by visually impaired users on images of their surroundings involves reading text in the image. But today’s VQA models can not read! Our paper takes a first step towards addressing this problem. First, we introduce a new “TextVQA” dataset to facilitate progress on this important problem. Existing datasets either have a small proportion of questions about text (e.g., the VQA dataset) or are too small (e.g., the VizWiz dataset). TextVQA contains 45,336 questions on 28,408 images that require reasoning about text to answer. Second, we introduce a novel model architecture that reads text in the image, reasons about it in the context of the image and the question, and predicts an answer which might be a deduction based on the text and the image or composed of the strings found in the image. Consequently, we call our approach Look, Read, Reason & Answer (LoRRA). We show that LoRRA outperforms existing state-of-the-art VQA models on our TextVQA dataset. We find that the gap between human performance and machine performance is significantly larger on TextVQA than on VQA 2.0, suggesting that TextVQA is well-suited to benchmark progress along directions complementary to VQA 2.0.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Backpropagation Applied to Handwritten Zip Code Recognition
LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., & Jackel, L. D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.4.541 · Cited by 11,891
(2017) · Cited by 11,625
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., & Zhang, L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00636 · Cited by 4,948
Bag of Tricks for Efficient Text Classification
Joulin, A., Grave, E., Bojanowski, P., & Mikolov, T. (2017) · Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers · doi:10.18653/v1/e17-2068 · Cited by 4,545
Cited by
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
Improved Baselines with Visual Instruction Tuning
Liu, H., Li, C., Li, Y., & Lee, Y. J. (2023) · arXiv · doi:10.48550/arxiv.2310.03744 · Cited by 1,733
Improved Baselines with Visual Instruction Tuning
Liu, H., Li, C., Li, Y., & Lee, Y. J. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02484 · Cited by 1,748
VisualBERT: A Simple and Performant Baseline for Vision and Language
Li, L. H., Yatskar, M., Yin, D., Hsieh, C.-J., & Chang, K.-W. (2019) · arXiv · doi:10.48550/arxiv.1908.03557 · Cited by 1,198
Flamingo: a visual language model for few-shot learning
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1723 · Cited by 966
Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02283 · Cited by 734
MMBench: Is Your Multi-modal Model an All-Around Player?
Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al. (2025) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-72658-3_13 · Cited by 668
FLAVA: A Foundational Language And Vision Alignment Model
Singh, A., Hu, R., Goswami, V., Couairon, G., Galuba, W., Rohrbach, M., & Kiela, D. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01519 · Cited by 554
Self-citationDocVQA: A Dataset for VQA on Document Images
Mathew, M., Karatzas, D., & Jawahar, C. V. (2021) · 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv48630.2021.00225 · Cited by 513
Multimodal intelligence: Representation learning, information fusion, and applications
Zhang, C., Yang, Z., He, X., & Deng, L. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.2987728 · Cited by 511
MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
Yue, X., Ni, Y., Zheng, T., Zhang, K., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.00913 · Cited by 429
In Defense of Grid Features for Visual Question Answering
Jiang, H., Misra, I., Rohrbach, M., Learned-Miller, E., & Chen, X. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01028 · Cited by 358
Self-citationThe hateful memes challenge: Detecting hate speech in multimodal memes
Kiela, D., Firooz, H., Mohan, A., Goswami, V., Singh, A., Ringshia, P., & Testuggine, D. (2020) · arXiv · doi:10.48550/arxiv.2005.04790 · Cited by 352
“In the past few years there has been a surge of interest in multimodal problems, from image captioning [8, 43, 88, 28, 68] to visual question answering (VQA) [3, 26, 27, 33, 71] and beyond.”
BackgroundInfluentialSelf-citationMVBench: A Comprehensive Multi-modal Video Understanding Benchmark
Li, K., Wang, Y., He, Y., Li, Y., Wang, Y., Liu, Y., Wang, Z., Xu, J., Chen, G., Lou, P., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02095 · Cited by 300
Scene Text Visual Question Answering
Biten, A. F., Tito, R., Mafla, A., Gomez, L., Rusinol, M., Jawahar, C. V., Valveny, E., & Karatzas, D. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00439 · Cited by 293
A-OKVQA: A Benchmark for Visual Question Answering Using World Knowledge
Schwenk, D., Khandelwal, A., Clark, C., Marino, K., & Mottaghi, R. (2022) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-20074-8_9 · Cited by 269
CogAgent: A Visual Language Model for GUI Agents
Hong, W., Wang, W., Lv, Q., Xu, J., Yu, W., Ji, J., Wang, Y., Wang, Z., Dong, Y., Ding, M., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01354 · Cited by 202
Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs
Tong, S., Liu, Z., Zhai, Y., Ma, Y., LeCun, Y., & Xie, S. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.00914 · Cited by 189
Effective Techniques for Multimodal Data Fusion: A Comparative Analysis
Pawłowski, M., Wróblewska, A., & Sysko-Romańczuk, S. (2023) · Sensors · doi:10.3390/s23052381 · Cited by 180
OCRBench: on the hidden mystery of OCR in large multimodal models
Liu, Y., Li, Z., Huang, M., Yang, B., Yu, W., Li, C., Yin, X.-C., Liu, C.-L., Jin, L., & Bai, X. (2024) · Science China Information Sciences · doi:10.1007/s11432-024-4235-6 · Cited by 167
“Scene Text-Centric VQA : We test LMMs on STVQA [51], TextVQA [52], OCRVQA [53] and ESTVQA [54].”
Multimodal learning with graphs
Ektefaie, Y., Dasoulas, G., Noori, A., Farhat, M., & Zitnik, M. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00624-6 · Cited by 166
Vision-Language Pre-Training: Basics, Recent Advances, and Future Trends
Gan, Z., Li, L., Li, C., Wang, L., Liu, Z., & Gao, J. (2022) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000105 · Cited by 155
Generative Multimodal Models are In-Context Learners
Sun, Q., Cui, Y., Zhang, X., Zhang, F., Yu, Q., Wang, Y., Rao, Y., Liu, J., Huang, T., & Wang, X. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01365 · Cited by 147
The Revolution of Multimodal Large Language Models: A Survey
Caffagni, D., Cocchi, F., Barsellotti, L., Moratelli, N., Sarto, S., Baraldi, L., Baraldi, L., Cornia, M., & Cucchiara, R. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.807 · Cited by 98
“TextVQA (VQA T ) (Singh et al., 2019) is a dataset based on pictures from Open Images (Kuznetsova et al., 2020) Comprehensively describing the visual input is another important skill desired in MLLMs.”
OpenVLA: An Open-Source Vision-Language-Action Model
Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., Sanketi, P., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.09246 · Cited by 62
Clinical insights: A comprehensive review of language models in medicine
Neveditsin, N., Lingras, P., & Mago, V. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000800 · Cited by 37
Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models
Karamcheti, S., Nair, S., Balakrishna, A., Liang, P., Kollar, T., & Sadigh, D. (2024) · arXiv · doi:10.48550/arxiv.2402.07865 · Cited by 7
“For TextVQA, we also run a variant of the evaluation where VLMs are additionally prompted with input tokens parsed by an OCR-system.”
BackgroundMethodInfluentialWhat Matters When Building Vision Language Models for Product Image Analysis?
Trabelsi, A., Zontak, M., Qian, Y., Jackson, B., Khan, S., & Batur, U. (2025) · 2025 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw65960.2025.00151 · Cited by 3
RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics
Yuan, W., Duan, J., Blukis, V., Pumacay, W., Krishna, R., Murali, A., Mousavian, A., & Fox, D. (2024) · arXiv · doi:10.48550/arxiv.2406.10721 · Cited by 7
Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction
Bui, N., Nguyen, H. T., Cohan, A., & Ying, R. (2026) · arXiv · doi:10.48550/arxiv.2605.09649
“We evaluate on three standard short-form VQA benchmarks: VQAText [Singh et al., 2019], MME [Fu et al., 2025a], and GQA [Hudson and Manning, 2019].”
Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.21783 · Cited by 505
Self-citationEnhancing Vision Foundation Models via Multimodal Continual Pre-Training
Chen, Y., Meng, L., Peng, W., Tao, J., Xu, C., Wu, Z., & Jiang, Y.-G. (2025) · arXiv · doi:10.48550/arxiv.2503.18931
Self-citationGemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Gemini Team, Georgiev, P., Lei, V. I., Burnell, R., Bai, L., Gulati, A., Tanzer, G., Vincent, D., Pan, Z., Wang, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.05530 · Cited by 306
Self-citation
Metadata
- Authors
- Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., & Rohrbach, M.
- Venue
- 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2019-06
- Pages
- 8309-8318
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2936135081
- Cited by
- 711
- References
- 46
- FWCI
- 16.45
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.