Multiscale Feature Extraction and Fusion of Image and Text in VQA
Open accessThe DOI matches its Crossref record; 40 of the 41 references it cites were found in a registry or an index.
Abstract
The Visual Question Answering (VQA) system is the process of finding useful information from images related to the question to answer the question correctly. It can be widely used in the fields of visual assistance, automated security surveillance, and intelligent interaction between robots and humans. However, the accuracy of VQA has not been ideal, and the main difficulty in its research is that the image features cannot well represent the scene and object information, and the text information cannot be fully represented. This paper used multi-scale feature extraction and fusion methods in the image feature characterization and text information representation sections of the VQA system, respectively to improve its accuracy. Firstly, aiming at the image feature representation problem, multi-scale feature extraction and fusion method were adopted, and the image features output of different network layers were extracted by a pre-trained deep neural network, and the optimal scheme of feature fusion method was found through experiments. Secondly, for the representation of sentences, a multi-scale feature method was introduced to characterize and fuse the word-level, phrase-level, and sentence-level features of sentences. Finally, the VQA model was improved using the multi-scale feature extraction and fusion method. The results show that the addition of multi-scale feature extraction and fusion improves the accuracy of the VQA model.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Machine learning methods are not good enough to differentiate between objects that have motion in their pixels [35,36].”
From Vehicle Detection and Classification via YOLOv8 and Deep Belief Network over Aerial Image Sequences (2023) · cites this paper as [36]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Show and tell: A neural image caption generator
Vinyals, O., Toshev, A., Bengio, S., & Erhan, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298935 · Cited by 6,318 · cited 1×
“Most of the early work applied the model of image caption generation [6–9] to the Visual Question Answering (VQA) task [10–12] and realized the ability of VQA by combining the modules of convolutional neural network and cyclic neural network.”
MethodLong-term recurrent convolutional networks for visual recognition and description
Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., & Saenko, K. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298878 · Cited by 5,228 · cited 1×
“Furthermore, scholars put forward the introduction of an external knowledge base [8, 15], among which the common external knowledge base includes the DBpedia [16], the Freebase [17], the YAGO [18], and the OpenIE [19].”
BackgroundFreebase: a collaboratively created graph database for structuring human knowledge
Bollacker, K., Evans, C., Paritosh, P., Sturge, T., & Taylor, J. (2008) · Proceedings of the 2008 ACM SIGMOD international conference on Management of data · doi:10.1145/1376616.1376746 · Cited by 4,852 · cited 1×
“Furthermore, scholars put forward the introduction of an external knowledge base [8, 15], among which the common external knowledge base includes the DBpedia [16], the Freebase [17], the YAGO [18], and the OpenIE [19].”
BackgroundBilinear CNN Models for Fine-Grained Visual Recognition
Lin, T.-Y., RoyChowdhury, A., & Maji, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.170 · Cited by 2,135 · cited 1×
“Lin first proposed the bilinear network [38], mainly used for fine-grained image classification tasks, such as distinguishing bird species and aircraft species and other extremely subtle images.”
BackgroundMultimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Fukui, A., Park, D. H., Yang, D., Rohrbach, A., Darrell, T., & Rohrbach, M. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1044 · Cited by 1,582
YAGO2: A spatially and temporally enhanced knowledge base from Wikipedia
Hoffart, J., Suchanek, F. M., Berberich, K., & Weikum, G. (2013) · Artificial Intelligence · doi:10.1016/j.artint.2012.06.001 · Cited by 1,232 · cited 1×
“Furthermore, scholars put forward the introduction of an external knowledge base [8, 15], among which the common external knowledge base includes the DBpedia [16], the Freebase [17], the YAGO [18], and the OpenIE [19].”
BackgroundHierarchical Question-Image Co-Attention for Visual Question Answering
Lu, J., Yang, J., Batra, D., & Parikh, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.00061 · Cited by 1,129 · cited 1×
“To get the feature in phrase scale, a one-dimensional convolution neural network proposed by Lu [30] is used to extract the feature of word vector space.”
MethodRethinking ImageNet Pre-Training
He, K., Girshick, R., & Dollar, P. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00502 · Cited by 1,049 · cited 1×
“He Keming [43] studied the parameter initialization method of ImageNet pre-training.”
MethodAsk Me Anything: Dynamic Memory Networks for Natural Language Processing
Kumar, A., Irsoy, O., Ondruska, P., Iyyer, M., Bradbury, J., Gulrajani, I., Zhong, V., Paulus, R., & Socher, R. (2015) · arXiv · doi:10.48550/arxiv.1506.07285 · Cited by 1,020 · cited 1×
“The combination model is to decompose VQA tasks into independent models to solve different types of tasks, among which neural module networks (NMN) [13] and dynamic memory networks (DMN) [14] are typical representatives.”
Method
Research works printed but not linked: 4
- Ren, M., Kiros, R., Zemel, R.: Exploring models and data for image question answering. Adv. Neural. Inf. Process. Syst. 28, 2953–2961 (2015)
- Auer, S., et al. DBpedia: A nucleus for a web of open data. in 6th International Semantic Web Conference/2nd Asian Semantic Web Conference (ISWC 2007/ASWC 2007). Busan, South Korea: Springer-Verlag Berlin. (2007)
- Adelson, E.H., et al.: Pyramid methods in image processing. RCA Eng 29(6), 33–41 (1984)
- Ben-younes, H., et al. MUTAN: Multimodal Tucker Fusion for Visual Question Answering. In 16th IEEE International Conference on Computer Vision (ICCV). Venice, ITALY: Ieee. (2017) · DOI · Paper page
Cited by
(1996) · Institutional Research Information System University of Turin (University of Turin) · doi:10.1111/j · Cited by 5,486
Hong, Z., & Xiao, K. (2024) · Scientific Reports · doi:10.1038/s41598-024-53760-3 · Cited by 174
Amiri, Z., Heidari, A., Navimipour, N. J., Unal, M., & Mousavi, A. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-16382-x · Cited by 189
Transforming Conversations with AI—A Comprehensive Study of ChatGPT
Bansal, G., Chamola, V., Hussain, A., Guizani, M., & Niyato, D. (2024) · Cognitive Computation · doi:10.1007/s12559-023-10236-2 · Cited by 164
Robust human locomotion and localization activity recognition over multisensory
Khan, D., Alonazi, M., Abdelhaq, M., Mudawi, N. A., Algarni, A. M., Jalal, A., & Liu, H. (2024) · Frontiers in Physiology · doi:10.3389/fphys.2024.1344887 · Cited by 220
“Kurtosis (Lu et al., 2023; Liu et al., 2023b; Liu et al., 2023c; Liu et al., 2023d) on the other hand, measures the tailedness of the distribution.”
Background
Metadata
- Authors
- Lu, S., Ding, Y., Liu, M., Yin, Z., Yin, L., & Zheng, W.
- Affiliations
- University of Electronic Science and Technology of China (CN)
- Wenzhou University of Technology (CN)
- Guizhou University (CN)
- Louisiana State University (US)
- Corresponding author
- Liu, Mingzhe
- Zheng, Wenfeng
- Venue
- International Journal of Computational Intelligence Systems
- Published
- 2023-04-11
- Volume
- 16
- Issue
- 1
- ISSN
- 1875-6883 (ISSN-L)
- OpenAlex
- W4364377616
- Cited by
- 422
- References
- 41
- Citations by year
- 2026: 12 · 2025: 44 · 2024: 66 · 2023: 114
- FWCI
- 21.44
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- International Journal of Computational Intelligence Systems (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Sichuan Science and Technology Program (2021YFQ0003)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Keywords
- visual question answering, multi-scale feature extraction, feature fusion, image-text fusion, visual feature representation, sentence representation, word-level features, sentence-level features, pre-trained deep neural networks
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Multiscale Feature Extraction and Fusion of Image and Text in VQA. https://app.cochat.ai/papers/10.1007/s44196-023-00233-6
- The paper
- Lu, S., Ding, Y., Liu, M., Yin, Z., Yin, L., & Zheng, W. (2023). Multiscale Feature Extraction and Fusion of Image and Text in VQA. International Journal of Computational Intelligence Systems. https://doi.org/10.1007/s44196-023-00233-6
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.