Visual Question Generation: The State of the Art
Abstract
Visual question generation (VQG) is an interesting problem that has recently received attention. The task of VQG involves generating meaningful questions based on the input image. It is a multi-modal problem involving image understanding and natural language generation, especially using deep learning methods. VQG can be considered as complementary task of visual question answering. In this article, we review the current state of VQG in terms of methods to understand the problem, existing datasets to train the VQG model, evaluation metrics, and algorithms to handle the problem. Finally, we discuss the challenges that need to be conquered and the possible future directions for an effective VQG.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A few specific reviews exist for particular domain areas of question generation.
“There are a few specific reviews, in particular domain areas of question generation. For example, the work done for question generation in the educational domain is discussed in [15].”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [17]
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333 · see citing works
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545 · see citing works
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition
(1993) · doi:10.1109/cvpr.1993 · Cited by 17,989 · see citing works
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318 · see citing works
Indoor segmentation and support inference from rgbd images
Silberman, N., Hoiem, D., Kohli, P., & Fergus, R. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33715-4_54 · Cited by 5,760 · see citing works
Proceedings European Conference on Computer Vision 2002
Heyden, A., Sparr, G., Nielsen, M., & Johansen, P. (2002) · Lund University Publications (Lund University) · Cited by 5,153 · see citing works
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107 · see citing works
Deep visual-semantic alignments for generating image descriptions
Karpathy, A., & Fei-Fei, L. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298932 · Cited by 4,902 · see citing works
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845 · see citing works
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487 · see citing works
Proceedings of the International Joint Conference on Artificial Intelligence 2007
Bundy, A. (2007) · International Joint Conference on Artificial Intelligence · Cited by 2,761 · see citing works
Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Fukui, A., Park, D. H., Yang, D., Rohrbach, A., Darrell, T., & Rohrbach, M. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1044 · Cited by 1,582 · see citing works
From captions to visual concepts and back
Fang, H., Gupta, S., Iandola, F., Srivastava, R. K., Deng, L., Dollár, P., Gao, J., He, X., Mitchell, M., Platt, J. C., et al. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298754 · Cited by 1,305 · see citing works
Hierarchical Question-Image Co-Attention for Visual Question AnsweringPreprint
Lu, J., Yang, J., Batra, D., & Parikh, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.00061 · Cited by 1,303 · see citing works
Scene Graph Generation by Iterative Message Passing
Xu, D., Zhu, Y., Choy, C. B., & Fei-Fei, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.330 · Cited by 1,271 · see citing works
DenseCap: Fully Convolutional Localization Networks for Dense Captioning
Johnson, J., Karpathy, A., & Fei-Fei, L. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.494 · Cited by 1,161 · see citing works
Visual7W: Grounded Question Answering in Images
Zhu, Y., Groth, O., Bernstein, M., & Fei-Fei, L. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.540 · Cited by 794 · cited 2× · see citing works
“The Visual7W dataset [78] is a sub-set of the Visual Genome dataset [49].”
BackgroundMethodGrounded Compositional Semantics for Finding and Describing Images with Sentences
Socher, R., Karpathy, A., Le, Q. V., Manning, C. D., & Ng, A. Y. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00177 · Cited by 779 · see citing works
Murray, G., Renals, S. J., Carletta, J., & Moore, J. D. (2005) · Meeting of the Association for Computational Linguistics · Cited by 716 · see citing works
Learning to Ask: Neural Question Generation for Reading Comprehension
Du, X., Shao, J., & Cardie, C. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1123 · Cited by 696 · cited 1× · see citing works
“Another inverse question answering approach is suggested in Du et al. [12] that uses Bi-LSTM.”
BackgroundAsk Your Neurons: A Neural-Based Approach to Answering Questions about Images
Malinowski, M., Rohrbach, M., & Fritz, M. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.9 · Cited by 633 · see citing works
Scene Graph Generation from Objects, Phrases and Region Captions
Li, Y., Ouyang, W., Zhou, B., Wang, K., & Wang, X. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.142 · Cited by 518 · see citing works
FVQA: Fact-Based Visual Question Answering
Wang, P., Wu, Q., Shen, C., Dick, A., & van den Hengel, A. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2754246 · Cited by 490 · see citing works
Towards Diverse and Natural Image Descriptions via a Conditional GAN
Dai, B., Fidler, S., Urtasun, R., & Lin, D. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.323 · Cited by 483 · see citing works
Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual GroundingPreprint
Fukui, A., Park, D. H., Yang, D., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.01847 · Cited by 388 · see citing works
Neural Question Generation from Text: A Preliminary Study
Zhou, Q. S., Yang, N., Wei, F., Tan, C., Bao, H., & Zhou, M. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-73618-1_56 · Cited by 353 · see citing works
Visual Turing test for computer vision systems
Geman, D., Geman, S., Hallonquist, N., & Laurent Younes, B. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1422953112 · Cited by 323 · see citing works
Image Captioning with Deep Bidirectional LSTMs
Wang, C., Yang, H., Bartz, C., & Meinel, C. (2016) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2964284.2964299 · Cited by 294 · see citing works
Generating Factoid Questions With Recurrent Neural Networks: The 30M Factoid Question-Answer Corpus
Serban, I. V., García-Durán, A., Gulcehre, C., Ahn, S., Chandar, S., Courville, A., & Bengio, Y. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1056 · Cited by 280 · see citing works
Generating Natural Questions About an Image
Mostafazadeh, N., Misra, I., Devlin, J., Mitchell, M., He, X., & Vanderwende, L. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1170 · Cited by 270 · see citing works
Speaking the Same Language: Matching Machine to Human Captions by Adversarial Training
Shetty, R., Rohrbach, M., Hendricks, L. A., Fritz, M., & Schiele, B. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.445 · Cited by 231 · see citing works
Visual Question Generation as Dual Task of Visual Question Answering
Li, Y., Duan, N., Zhou, B., Chu, X., Ouyang, W., Wang, X., & Zhou, M. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00640 · Cited by 171 · see citing works
KVQA: Knowledge-Aware Visual Question Answering
Shah, S. B., Mishra, A., Yadati, N., & Talukdar, P. P. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33018876 · Cited by 171 · see citing works
Question Answering on Knowledge Bases and Text using Universal Schema and Memory Networks
Das, R., Zaheer, M., Reddy, S., & McCallum, A. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-2057 · Cited by 156 · see citing works
Answer-Type Prediction for Visual Question Answering
Kafle, K., & Kanan, C. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.538 · Cited by 123 · see citing works
Rich Image Captioning in the Wild
Tran, K., He, X., Zhang, L., & Sun, J. Z. (2016) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2016.61 · Cited by 110 · see citing works
Image Understanding using vision and reasoning through Scene Description Graph
Aditya, S., Yang, Y., Baral, C. R., Aloimonos, Y., & Fermuller, C. M. (2017) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2017.12.004 · Cited by 103 · see citing works
Information Maximizing Visual Question Generation
Krishna, R., Bernstein, M. S., & Fei-Fei, L. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00211 · Cited by 86 · see citing works
Automatic Generation of Grounded Visual Questions
Zhang, S., Qu, L., You, S., Yang, Z., & Zhang, J. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/592 · Cited by 82 · see citing works
A Question Type Driven Framework to Diversify Visual Question Generation
Fan, Z., Wei, Z., Li, P., Lan, Y., & Huang, X. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/563 · Cited by 54 · see citing works
Infusing NLU into Automatic Question Generation
Mazidi, K., & Tarau, P. (2016) · Proceedings of the 9th International Natural Language Generation conference · doi:10.18653/v1/w16-6609 · Cited by 36 · see citing works
Neural Question Generation from Text: A Preliminary StudyPreprint
Zhou, Q. S., Yang, N., Wei, F., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.01792 · Cited by 35 · see citing works
Neural Self Talk: Image Understanding via Continuous Questioning and AnsweringPreprint
Yang, Y., Li, Y., Fermuller, C. M., & Aloimonos, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.03460 · Cited by 17 · see citing works
From Two Graphs to N Questions: A VQA Dataset for Compositional Reasoning on Vision and Commonsense.Preprint
Gao, D., Wang, R., Shan, S., & Chen, X. (2019) · arXiv (Cornell University) · Cited by 6 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Mulla, N., & Gharpure, P. (2023) · Progress in Artificial Intelligence · doi:10.1007/s13748-023-00295-9 · Cited by 132 · see citing works
A survey of methods, datasets and evaluation metrics for visual question answering
Sharma, H., & Jalal, A. S. (2021) · Image and Vision Computing · doi:10.1016/j.imavis.2021.104327 · Cited by 47 · see citing works
VQA and Visual Reasoning: An overview of approaches, datasets, and future direction
Zakari, R. Y., Owusu, J. W., Qin, K., Wang, H., Lawal, Z. K., & He, T. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129345 · Cited by 25 · see citing works
Pramanick, P., Sarkar, C., Banerjee, S., & Bhowmick, B. (2022) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2022.104183 · Cited by 18 · see citing works
Learning visual language models for video understanding
Yang, A. (2023) · theses.fr (ABES) · doi:10.70675/802872a7z8092z4556za1fdz2c7af45f0e72 · Cited by 15 · see citing works
Machine-to-Machine Visual Dialoguing with ChatGPT for Enriched Textual Image Description
Ricci, R., Bazi, Y., & Melgani, F. (2024) · Remote Sensing · doi:10.3390/rs16030441 · Cited by 14 · see citing works
Automated multiple-choice question generation in Spanish using neural language models
de-Fitero-Dominguez, D., García‐Cabot, A., & García‐López, E. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-10076-7 · Cited by 14 · see citing works
Template-based Abstractive Microblog Opinion Summarization
Bilal, I. M., Wang, B., Tsakalidis, A., Nguyen, D., Procter, R., & Liakata, M. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00516 · Cited by 13 · see citing works
A Critical Analysis of Benchmarks, Techniques, and Models in Medical Visual Question Answering
Al-Hadhrami, S. A., Menaï, M. E. B., Al-Ahmadi, S. A., & Alnafessah, A. (2023) · IEEE Access · doi:10.1109/access.2023.3335216 · Cited by 13 · see citing works
Goal-Driven Visual Question Generation from Radiology Images
Sarrouti, M., Abacha, A. B., & Demner‐Fushman, D. (2021) · Information · doi:10.3390/info12080334 · Cited by 8 · see citing works
Automated Machine Learning in a Multi-agent Environment
Bakas, N., Lavdas, S., Vavousis, K., Christodoulou, C., & Langousis, A. (2025) · Lecture notes in business information processing · doi:10.1007/978-3-031-81322-1_4 · Cited by 7 · see citing works
Enriching object-aware image–text highlight information for visual question generation
Lee, S., & Lee, D.-G. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104379 · Cited by 7 · see citing works
Attention-based Visual Question Generation
Patil, C., & Kulkarni, A. (2021) · International Conference on Emerging Smart Computing and Informatics (ESCI) · doi:10.1109/esci50559.2021.9396956 · Cited by 5 · see citing works
Self-citationMulti-Hop Question Generation Using Hierarchical Encoding-Decoding and Context Switch Mechanism
Ji, T., Lyu, C., Cao, Z., & Cheng, P. (2021) · Entropy · doi:10.3390/e23111449 · Cited by 4 · see citing works
So Many Heads, So Many Wits: Multimodal Graph Reasoning for Text-Based Visual Question Answering
Zheng, W., Yan, L., & Wang, F. (2023) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2023.3319964 · Cited by 3 · see citing works
QC 2 -VQG: Question context complement for visual question generation
Zhang, Y., Liu, X., Lu, Z., Guo, W., Liu, X., & Yan, R. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114306 · Cited by 1 · see citing works
Automatic Question Generation with Large Language Models: A Survey
de Oliveira, R. M., Hernández, A., Garbin, M. C., dos Reis, J. C., & Azevedo, R. (2026) · ACM Computing Surveys · doi:10.1145/3832773 · Cited by 1 · see citing works
Александрович, С. О. (2024) · Computers · doi:10.3390/computers13060144 · Cited by 1 · see citing works
“Some recent works on Visual Question Generation try to use an image as input for question generation instead of text, using methods of solving the problem of image understanding and summarization [19–21].”
MethodSituational Data Integration in Question Answering systems: a survey over two decades
Franciscatto, M. H., Bona, L. C. E., Trois, C., Fabro, M. D. D., & Lima, J. C. D. (2024) · Knowledge and Information Systems · doi:10.1007/s10115-024-02136-0 · Cited by 1 · see citing works
WikiFlash: Generating Flashcards from Wikipedia Articles
Cheng, Y., Ding, Y., Foucher, S., Pascual, D., Richter, O., Volk, M., & Wattenhofer, R. P. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-92273-3_12 · Cited by 4 · see citing works
Effective Generation of Visual Questions
Patil, C., & Abhyankar, A. (2022) · International Conference on Signal and Information Processing · doi:10.1109/iconsip49665.2022.10007524
Self-citationVisual Question Generation Framework for Chess Game State Identification
Padaria, A. A., Patel, D., Gupta, R., Jadav, N. K., Tanwar, S., & Garg, D. (2024) · IEEE International Conference on Contemporary Computing and Communications (InC4) · doi:10.1109/inc460750.2024.10649325
Generating visual questions with hop sequence reasoning
Patil, C. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-01083-w
“…et al. 2021), video question answering (Mihalcea et al. 2015; Jang et al. 2017), visual question generation (VQG) (Jain et al. 2017; Zhang et al. 2018; Krishna et al. 2019; Patil and Patwardhan 2020; Bi et al. 2022) In this paper, we are dealing with an interesting vision-language problem of VQG.”
Self-citation
- Authors
- Patil, C., & Patwardhan, M.
- Affiliations
- Savitribai Phule Pune University (IN)
- Venue
- ACM Computing Surveys
- Published
- 2020-05-28
- Volume
- 53
- Issue
- 3
- Pages
- 1-22
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W3029273706
- Cited by
- 25
- References
- 43
- Citations by year
- 2026: 2 · 2025: 4 · 2024: 5 · 2023: 4 · 2022: 4 · 2021: 6
- FWCI
- 1.36
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- ACM Computing Surveys (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Generative Adversarial Networks and Image Synthesis
- Keywords
- visual question generation, image understanding, natural language generation, visual question answering, multimodal learning, evaluation metrics, deep learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
