The DOI matches its Crossref record; 62 of the 62 references it cites were found in a registry or an index.
Scene Text Visual Question Answering
Open accessAbstract
Current visual question answering datasets do not consider the rich semantic information conveyed by text within an image. In this work, we present a new dataset, ST-VQA, that aims to highlight the importance of exploiting high-level semantic information present in images as textual cues in the Visual Question Answering process. We use this dataset to define a series of tasks of increasing difficulty for which reading the scene text in the context provided by the visual information is necessary to reason and generate an appropriate answer. We propose a new evaluation metric for these tasks to account both for reasoning errors as well as shortcomings of the text recognition module. In addition we put forward a series of baseline methods, which provide further insight to the newly released dataset, and set the scene for further research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Binary Codes capable of currecting deletions, insertions, and reversals
Levenshtein, V. I. (1965) · Soviet physics. Doklady · Cited by 14,807
Enriching Word Vectors with Subword Information
Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00051 · Cited by 9,645
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 2×
“In contrast to prior datasets, and similar to our proposal, SQuAD [42] does not provide a list of answer choices for each question.”
BackgroundResultVisual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
Teaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., & Parikh, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.670 · Cited by 2,287
Stacked Attention Networks for Image Question Answering
Yang, Z., He, X., Gao, J., Deng, L., & Smola, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.10 · Cited by 1,998
EAST: An Efficient and Accurate Scene Text Detector
Zhou, X., Yao, C., Wen, H., Wang, Y., Zhou, S., He, W., & Liang, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.283 · Cited by 1,850
CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.215 · Cited by 1,781
ICDAR 2015 competition on Robust Reading
Karatzas, D., Gomez-Bigorda, L., Nicolaou, A., Ghosh, S., Bagdanov, A., Iwamura, M., Matas, J., Neumann, L., Chandrasekhar, V. R., Lu, S., et al. (2015) · 2015 13th International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.2015.7333942 · Cited by 1,547
Self-citationICDAR 2013 Robust Reading Competition
Karatzas, D., Shafait, F., Uchida, S., Iwamura, M., Bigorda, L. G. I., Mestre, S. R., Mas, J., Mota, D. F., Almazan, J. A., & de las Heras, L. P. (2013) · 2013 12th International Conference on Document Analysis and Recognition · doi:10.1109/icdar.2013.221 · Cited by 1,360
Self-citationReading text in the wild with convolutional neural networks
Jaderberg, M., Simonyan, K., Vedaldi, A., & Zisserman, A. (2016) · International Journal of Computer Vision · doi:10.1007/s11263-015-0823-z · Cited by 1,191
TextBoxes: A Fast Text Detector with a Single Deep Neural Network
Liao, M., Shi, B., Bai, X., Wang, X., & Liu, W. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11196 · Cited by 955
VQA: Visual Question Answering
Agrawal, A., Lu, J., Antol, S., Mitchell, M., Zitnick, C. L., Batra, D., & Parikh, D. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.00468 · Cited by 900
TextBoxes++: A Single-Shot Oriented Scene Text Detector
Liao, M., Shi, B., & Bai, X. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2825107 · Cited by 897
Towards VQA Models That Can Read
Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., & Rohrbach, M. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00851 · Cited by 711
Robust Scene Text Recognition with Automatic Rectification
Shi, B., Wang, X., Lyu, P., Yao, C., & Bai, X. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.452 · Cited by 693
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · arXiv · doi:10.48550/arxiv.1606.05250 · Cited by 678
Skinner, B. F. (1963) · American Psychologist · doi:10.1037/h0045185 · Cited by 675
MovieQA: Understanding Stories in Movies through Question-Answering
Tapaswi, M., Zhu, Y., Stiefelhagen, R., Torralba, A., Urtasun, R., & Fidler, S. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.501 · Cited by 639
VizWiz Grand Challenge: Answering Visual Questions from Blind People
Gurari, D., Li, Q., Stangl, A. J., Guo, A., Lin, C., Grauman, K., Luo, J., & Bigham, J. P. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00380 · Cited by 621 · cited 2×
“In total, we used six different datasets, namely: ICDAR 2013[23] and ICDAR2015[22], ImageNet [6], VizWiz[13], IIIT Scene Text Retrieval[37], Visual Genome [29] and COCO-Text [50].”
MethodDon't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering
Agrawal, A., Batra, D., Parikh, D., & Kembhavi, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00522 · Cited by 566
FOTS: Fast Oriented Text Spotting with a Unified Network
Liu, X., Liang, D., Yan, S., Chen, D., Qiao, Y., & Yan, J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00595 · Cited by 554
Mask TextSpotter: An End-to-End Trainable Neural Network for Spotting Text with Arbitrary Shapes
Lyu, P., Liao, M., Yao, C., Wu, W., & Bai, X. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01264-9_5 · Cited by 522
Word Spotting and Recognition with Embedded Attributes
Almazan, J., Gordo, A., Fornes, A., & Valveny, E. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2339814 · Cited by 517
Self-citationUndoing the Damage of Dataset Bias
Khosla, A., Zhou, T., Malisiewicz, T., Efros, A. A., & Torralba, A. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33718-5_12 · Cited by 480
Gulcehre, C., Ahn, S., Nallapati, R., Zhou, B., & Bengio, Y. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1014 · Cited by 478
Wang, K., & Belongie, S. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15549-9_43 · Cited by 449
FigureQA: An Annotated Figure Dataset for Visual ReasoningPreprint
Kahou, S. E., Michalski, V., Atkinson, A., Kádár, Á., Trischler, A., & Bengio, Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.07300 · Cited by 446
Deep Direct Regression for Multi-oriented Scene Text Detection
He, W., Zhang, X.-Y., Yin, F., & Liu, C.-L. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.87 · Cited by 432
Machine Comprehension Using Match-LSTM and Answer Pointer
Wang, S., & Jiang, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.07905 · Cited by 408
Exploring Models and Data for Image Question Answering
Ren, M., Kiros, R., & Zemel, R. S. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.02074 · Cited by 359
Rosetta: Large Scale System for Text Detection and Recognition in Images
Borisyuk, F., Gordo, A., & Sivakumar, V. (2018) · Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3219819.3219861 · Cited by 353
Yin and Yang: Balancing and Answering Binary Visual Questions
Zhang, P., Goyal, Y., Summers-Stay, D., Batra, D., & Parikh, D. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.542 · Cited by 312
ReasoNet: Learning to Stop Reading in Machine Comprehension
Shen, Y., Huang, P.-S., Gao, J., & Chen, W. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098177 · Cited by 262
Multi-Orientation Scene Text Detection with Adaptive Clustering
Yin, X.-C., Pei, W.-Y., Zhang, J., & Hao, H.-W. (2015) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2014.2388210 · Cited by 259
DVQA: Understanding Data Visualizations via Question Answering
Kafle, K., Price, B., Cohen, S., & Kanan, C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00592 · Cited by 254
Visual Madlibs: Fill in the Blank Description Generation and Question Answering
Yu, L., Park, E., Berg, A. C., & Berg, T. L. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.283 · Cited by 252
Kembhavi, A., Seo, M., Schwenk, D., Choi, J., Farhadi, A., & Hajishirzi, H. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.571 · Cited by 244 · cited 1×
“The Textbook Question Answering (TQA) dataset [29] aims at answering multimodal questions given a context of text, diagrams and images, but textual information is provided in computer readable format.”
BackgroundDeep TextSpotter: An End-to-End Trainable Scene Text Localization and Recognition Framework
Busta, M., Neumann, L., & Matas, J. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.242 · Cited by 244
An End-to-End TextSpotter with Explicit Alignment and Attention
He, T., Tian, Z., Huang, W., Shen, C., Qiao, Y., & Sun, C. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00527 · Cited by 234
A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input
Malinowski, M., & Fritz, M. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.0210 · Cited by 233
Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering
Gao, H., Mao, J., Zhou, J., Huang, Z., Wang, L., & Xu, W. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1505.05612 · Cited by 216
Show, Ask, Attend, and Answer: A Strong Baseline For Visual Question AnsweringPreprint
Kazemi, V., & Elqursh, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.03162 · Cited by 210
Counting Everyday Objects in Everyday Scenes
Chattopadhyay, P., Vedantam, R., Selvaraju, R. R., Batra, D., & Parikh, D. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.471 · Cited by 197
COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Images
Veit, A., Matera, T., Neumann, L., Matas, J., & Belongie, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1601.07140 · Cited by 191
Image Retrieval Using Textual Cues
Mishra, A., Alahari, K., & Jawahar, C. V. (2013) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2013.378 · Cited by 114
Self-citationSelf-Supervised Learning of Visual Features through Embedding Images into Text Topic Spaces
Gómez, L., Patel, Y., Rusiñol, M., Karatzas, D., & Jawahar, C. V. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.218 · Cited by 109
Self-citationReading Scene Text with Attention Convolutional Sequence ModelingPreprint
Gao, Y., Chen, Y., Wang, J., & Lu, H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.04303 · Cited by 70
TallyQA: Answering Complex Counting Questions
Acharya, M., Kafle, K., & Kanan, C. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33018076 · Cited by 69
ICDAR 2019 Competition on Scene Text Visual Question Answering
Biten, A. F., Tito, R., Mafla, A., Gómez, L., Rusiñol, M., Mathew, M., Jawahar, C. V., Valveny, E., & Karatzas, D. (2019) · IAPR International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.2019.00251 · Cited by 68
Self-citationICDAR2017 Robust Reading Challenge on COCO-Text
Gómez, R., Shi, B., Gómez, L., Numann, L., Veit, A., Matas, J., Belongie, S., & Karatzas, D. (2017) · IAPR International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.2017.234 · Cited by 59
Self-citationSingle Shot Scene Text Retrieval
Gómez, L., Mafla, A., Rusiñol, M., & Karatzas, D. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01264-9_43 · Cited by 49
Self-citation
Cited by
Intern VL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02283 · Cited by 734
DocVQA: A Dataset for VQA on Document Images
Mathew, M., Karatzas, D., & Jawahar, C. V. (2021) · 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv48630.2021.00225 · Cited by 513
Self-citationMVBench: A Comprehensive Multi-modal Video Understanding Benchmark
Li, K., Wang, Y., He, Y., Li, Y., Wang, Y., Liu, Y., Wang, Z., Xu, J., Chen, G., Lou, P., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.02095 · Cited by 300
Text Recognition in the Wild: A Survey
Chen, X., Jin, L., Zhu, Y., Luo, C., & Wang, T. (2022) · ACM Computing Surveys · doi:10.1145/3440756 · Cited by 205
“Examples of various applications of scene text recognition. our daily life include document understanding [106], information extraction [107], [108], text visual question answering (text VQA) [109], [110], image classification [111], historical documents digitization [21], [112], automatic geocoding systems [22], [113]…”
BackgroundCogAgent: A Visual Language Model for GUI Agents
Hong, W., Wang, W., Lv, Q., Xu, J., Yu, W., Ji, J., Wang, Y., Wang, Z., Dong, Y., Ding, M., et al. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01354 · Cited by 202
“• Though CogAgent focuses on GUIs, it achieves state-of-the-art generalist performance on nine visual question-answering benchmarks including VQAv2 [1], OK-VQA [23], TextVQA [34], ST-VQA [4], ChartQA [24], infoVQA [26], DocVQA [25], MM-Vet [41], and POPE [19].”
Metadata
- Authors
- Biten, A. F., Tito, R., Mafla, A., Gomez, L., Rusinol, M., Jawahar, C. V., Valveny, E., & Karatzas, D.
- Venue
- 2019 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2019-10
- Pages
- 4290-4300
- OpenAlex
- W2947555604
- Cited by
- 293
- References
- 62
- FWCI
- 9.02
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.