Visual Question Generation from Radiology Images
Open accessAbstract
Visual Question Generation (VQG), the task of generating a question based on image contents, is an increasingly important area that combines natural language processing and computer vision.Although there are some recent works that have attempted to generate questions from images in the open domain, the task of VQG in the medical domain has not been explored so far.In this paper, we introduce an approach to generation of visual questions about radiology images called VQGR, i.e. an algorithm that is able to ask a question when shown an image.VQGR first generates new training data from the existing examples, based on contextual word embeddings and image augmentation techniques.It then uses the variational auto-encoders model to encode images into a latent space and decode natural language questions.Experimental automatic evaluations performed on the VQA-RAD dataset of clinical visual questions show that VQGR achieves good performances compared with the baseline system.The source code is available at https://github.com/ sarrouti/vqgr.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Visual question generation has been used to generate meaningful questions on radiology images in [136].
“Visual question generation, for instance, has been used for generating meaningful questions on radiology images in [136].”
From Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications (2023) · cites this paper as [136]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067 · see citing works
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487 · see citing works
CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.215 · Cited by 1,781 · see citing works
Microsoft COCO Captions: Data Collection and Evaluation Server
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollar, P., & Zitnick, C. L. (2015) · arXiv · doi:10.48550/arxiv.1504.00325 · Cited by 1,614 · see citing works
Agreement, the F-Measure, and Reliability in Information Retrieval
Hripcsak, G. M. (2005) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1733 · Cited by 949 · see citing works
Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations
Kobayashi, S. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-2072 · Cited by 617 · see citing works
A dataset of clinically generated visual questions and answers about radiology images
Lau, J. J., Gayen, S., Abacha, A. B., & Demner‐Fushman, D. (2018) · Scientific Data · doi:10.1038/sdata.2018.251 · Cited by 496 · see citing works
Self-citationMaking the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
Goyal, Y., Khot, T., Agrawal, A., Summers-Stay, D., Batra, D., & Parikh, D. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-018-1116-0 · Cited by 487 · see citing works
Generating Factoid Questions With Recurrent Neural Networks: The 30M Factoid Question-Answer Corpus
Serban, I. V., García-Durán, A., Gulcehre, C., Ahn, S., Chandar, S., Courville, A., & Bengio, Y. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1056 · Cited by 280 · see citing works
Manual and automatic evaluation of machine translation between European languages
Koehn, P., & Monz, C. (2006) · Workshop on Statistical Machine Translation · doi:10.3115/1654650.1654666 · Cited by 272 · see citing works
Generating Natural Questions About an Image
Mostafazadeh, N., Misra, I., Devlin, J., Mitchell, M., He, X., & Vanderwende, L. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1170 · Cited by 270 · see citing works
Improving Neural Question Generation Using Answer Separation
Kim, Y., Lee, H., Shin, J., & Jung, K. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016602 · Cited by 178 · see citing works
Visual Question Generation as Dual Task of Visual Question Answering
Li, Y., Duan, N., Zhou, B., Chu, X., Ouyang, W., Wang, X., & Zhou, M. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00640 · Cited by 171 · see citing works
Harvesting Paragraph-level Question-Answer Pairs from Wikipedia
Du, X., & Cardie, C. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1177 · Cited by 168 · see citing works
Creativity: Generating Diverse Questions Using Variational Autoencoders
Jain, U., Zhang, Z., & Schwing, A. G. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.575 · Cited by 131 · see citing works
VQA-Med: Overview of the Medical Visual Question Answering Task at ImageCLEF 2019
Abacha, A. B., Hasan, S. A., Datla, V. V., Liu, J., Demner‐Fushman, D., & Müller, H. (2024) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.10499038 · Cited by 109 · see citing works
Self-citationData Augmentation via Dependency Tree Morphing for Low-Resource Languages
Şahin, G. G., & Steedman, M. J. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1545 · Cited by 107 · see citing works
Information Maximizing Visual Question Generation
Krishna, R., Bernstein, M. S., & Fei-Fei, L. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00211 · Cited by 86 · see citing works
Automatic Generation of Grounded Visual Questions
Zhang, S., Qu, L., You, S., Yang, Z., & Zhang, J. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/592 · Cited by 82 · see citing works
VQA-Med: Overview of the Medical Visual Question Answering Task at ImageCLEF 2019
Abacha, A. B., Hasan, S. A., Datla, V. V., et al. (2024) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.10499039 · Cited by 81 · see citing works
Sarrouti, M., & Alaoui, S. O. E. (2020) · Artif. Intell. Medicine · doi:10.1016/j.artmed.2019.101767 · Cited by 78 · see citing works
Self-citationSarrouti, M., & Alaoui, S. O. E. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.03.001 · Cited by 59 · see citing works
Self-citationImproving Question Generation With to the Point Context
Li, J., Gao, Y., Bing, L., King, I., & Lyu, M. R.-T. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1317 · Cited by 52 · see citing works
Evaluating Rewards for Question Generation Models
Hosking, T., & Riedel, S. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1237 · Cited by 34 · see citing works
Overview of ImageCLEF 2018 Medical Domain Visual Question Answering Task.
Hasan, S. A., Yuan, L., Farri, O., Liu, J., Müller, H., & Lungren, M. P. (2018) · ArODES (HES-SO (https://www.hes-so.ch/)) · Cited by 32 · see citing works
NLM at ImageCLEF 2018 Visual Question Answering in the Medical Domain.
Abacha, A. B., Gayen, S., Lau, J. J., Rajaraman, S., & Demner‐Fushman, D. (2018) · CLEF (Working Notes) · Cited by 30 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 2
- Saidalavi Kalady, Ajeesh Elikkottil, and Rajarshi Das. 2010. Natural language question generation using syntax and keywords. In Proceedings of QG2010: The Third Workshop on Question Generation, volume 2, pages 5-14.
- Issey Masuda-Mora, Santiago Pascual-deLaPuente, and Xavier Giró i Nieto. 2016. Towards automatic generation of question answer pairs from images. In Visual Question Answering Challenge Workshop, CVPR 2016, Las Vegas, NV, USA.
Cited by
Sorted by citations
Mulla, N., & Gharpure, P. (2023) · Progress in Artificial Intelligence · doi:10.1007/s13748-023-00295-9 · Cited by 132 · see citing works
Erdem, E., Kuyu, M., Yagcioglu, S., Frank, A., Parcalabescu, L., Plank, B., Babii, A., Turuta, O., Erdem, A., Calixto, I., et al. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12918 · Cited by 54 · see citing works
An Encoder–Decoder-Based Method for Segmentation of COVID-19 Lung Infection in CT Images
Elharrouss, O., Subramanian, N., & Al-Maadeed, S. A. (2021) · SN Computer Science · doi:10.1007/s42979-021-00874-4 · Cited by 51 · see citing works
Abacha, A. B., Datla, V. V., Hasan, S. A., Demner‐Fushman, D., & Müller, H. (2020) · ArODES (HES-SO (https://www.hes-so.ch/)) · Cited by 43 · see citing works
Self-citationIonescu, B. E., Müller, H., Péteri, R., Abacha, A. B., Sarrouti, M., Demner‐Fushman, D., Hasan, S. A., Kozlovski, S., Liauchuk, V., Cid, Y. D., et al. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-85251-1_23 · Cited by 17 · see citing works
Self-citationA Critical Analysis of Benchmarks, Techniques, and Models in Medical Visual Question Answering
Al-Hadhrami, S. A., Menaï, M. E. B., Al-Ahmadi, S. A., & Alnafessah, A. (2023) · IEEE Access · doi:10.1109/access.2023.3335216 · Cited by 13 · see citing works
Goal-Driven Visual Question Generation from Radiology Images
Sarrouti, M., Abacha, A. B., & Demner‐Fushman, D. (2021) · Information · doi:10.3390/info12080334 · Cited by 8 · see citing works
Self-citationDiversity Learning Based on Multi-Latent Space for Medical Image Visual Question Generation
Zhu, H., Togo, R., Ogawa, T., & Haseyama, M. (2023) · Sensors · doi:10.3390/s23031057 · Cited by 8 · see citing works
UAV-VQG: Visual Question Generation Framework on UAV Images
Sarkar, A., & Rahnemoonfar, M. (2021) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata52589.2021.9671493 · Cited by 5 · see citing works
Visual Question Generation Under Multi-granularity Cross-Modal Interaction
Chai, Z., Wan, X., Han, S. C., & Poon, J. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-27077-2_20 · Cited by 2 · see citing works
Zhu, H., Togo, R., Ogawa, T., & Haseyama, M. (2026) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2026.105014 · Cited by 1 · see citing works
A Medical Domain Visual Question Generation Model via Large Language Model
Zhu, H., Togo, R., Ogawa, T., & Haseyama, M. (2023) · IEEE International Conference on Consumer Electronics - Taiwan (ICCE-Taiwan) · doi:10.1109/icce-taiwan58799.2023.10227045 · Cited by 1 · see citing works
Lungs Tumor Classification using Convolutional Neural Network
Praveena, R., Babu, T. R., Reddy, A. H., Dinesh, C., & Bharath, S. M. (2024) · Journal of Innovative Image Processing · doi:10.36548/jiip.2024.2.003 · Cited by 1 · see citing works
Wu, Q., Wang, P., Wang, X. Y., He, X., & Zhu, W. (2022) · Advances in computer vision and pattern recognition · doi:10.1007/978-981-19-0964-1_13 · Cited by 1 · see citing works
- Authors
- Sarrouti, M., Ben Abacha, A., & Demner-Fushman, D.
- Affiliations
- National Institutes of Health (US)
- ORCID
- Sarrouti, Mourad
- Venue
- Proceedings of the First Workshop on Advances in Language and Vision Research
- Published
- 2020
- Pages
- 12-18
- OpenAlex
- W3045951017
- Cited by
- 15
- References
- 26
- Citations by year
- 2026: 1 · 2024: 1 · 2023: 5 · 2022: 3 · 2021: 4 · 2020: 1
- FWCI
- 1.00
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Workshop on Advances in Language and Vision Research (ALVR) (publishedVersion, cc-by)
- Funders
- National Institutes of Health
- U.S. National Library of Medicine
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- visual question generation, radiographic images, variational autoencoder, image-to-text generation, image augmentation, contextual word embeddings
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
