The DOI matches its Crossref record; 151 of the 151 references it cites were found in a registry or an index.
Multi-Modal Knowledge Graph Construction and Application: A Survey
Open accessAbstract
Recent years have witnessed the resurgence of knowledge engineering which is featured by the fast growth of knowledge graphs. However, most of existing knowledge graphs are represented with pure symbols, which hurts the machine's capability to understand the real world. The multi-modalization of knowledge graphs is an inevitable key step towards the realization of human-level machine intelligence. The results of this endeavor are Multi-modal Knowledge Graphs (MMKGs). In this survey on MMKGs constructed by texts and images, we first give definitions of MMKGs, followed with the preliminaries on multi-modal tasks and techniques. We then systematically review the challenges, progresses and opportunities on the construction and application of MMKGs respectively, with detailed analyses of the strengths and weaknesses of different solutions. We finalize this survey with open research problems relevant to MMKGs.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Learning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293
WordNet: a lexical database for English
Miller, G. A. (1995) · Communications of the ACM · doi:10.1145/219717.219748 · Cited by 14,015 · cited 1×
“WordNet [3], BabelNet [4]), encyclopedia knowledge (e.”
Emerging Properties in Self-Supervised Vision Transformers
Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00951 · Cited by 5,638 · cited 1×
“Some pre-trained vision transformer models already have strong capability of visual object segmentation and focus on the foreground object even in highly ambiguous situations, such as DINO [70], which will improve the performance of locating visual objects and aligning cross-modal knowledge.”
BackgroundVisual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., & Zhang, L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00636 · Cited by 4,948
Freebase: a collaboratively created graph database for structuring human knowledge
Bollacker, K., Evans, C., Paritosh, P., Sturge, T., & Taylor, J. (2008) · Proceedings of the 2008 ACM SIGMOD international conference on Management of data · doi:10.1145/1376616.1376746 · Cited by 4,852
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
Harnad, S. (1990) · Physica D: Nonlinear Phenomena · doi:10.1016/0167-2789(90)90087-6 · Cited by 4,235 · cited 1×
“Thus, it is necessary to ground symbols to corresponding images, sound and video data and map symbols to their corresponding referents with meanings in the physical world, enabling machines to generate similar “experiences” like a real human [12] when they are confronted with a specific entity Hand-in-waistcoat or an…”
BackgroundKnowledge Graph Embedding by Translating on Hyperplanes
Wang, Z., Zhang, J., Feng, J., & Chen, Z. (2014) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v28i1.8870 · Cited by 3,915
The MovieLens Datasets: History and Context
Harper, F. M., & Konstan, J. A. (2016) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/2827872 · Cited by 3,907
Learning Entity and Relation Embeddings for Knowledge Graph Completion
Lin, Y., Liu, Z., Sun, M., Liu, Y., & Zhu, X. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9491 · Cited by 3,749
Self-citationA Survey of Collaborative Filtering Techniques
Su, X., & Khoshgoftaar, T. M. (2009) · Advances in Artificial Intelligence · doi:10.1155/2009/421425 · Cited by 3,602
Wikidata: a free collaborative knowledgebase
Vrandečić, D., & Krötzsch, M. (2014) · Communications of the ACM · doi:10.1145/2629489 · Cited by 3,342
Bobadilla, J., Ortega, F., Hernando, A., & Gutiérrez, A. (2013) · Knowledge-Based Systems · doi:10.1016/j.knosys.2013.03.012 · Cited by 3,158
NUS-WIDE: a real-world web image database from National University of Singapore
Chua, T.-S., Tang, J., Hong, R., Li, H., Luo, Z., & Zheng, Y. (2009) · Proceedings of the ACM International Conference on Image and Video Retrieval · doi:10.1145/1646396.1646452 · Cited by 3,025
Yago: a core of semantic knowledge
Suchanek, F. M., Kasneci, G., & Weikum, G. (2007) · Proceedings of the 16th international conference on World Wide Web · doi:10.1145/1242572.1242667 · Cited by 2,512
DBpedia: A Nucleus for a Web of Open Data
Auer, S., Bizer, C., Kobilarov, G., Lehmann, J., Cyganiak, R., & Ives, Z. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-76298-0_52 · Cited by 2,454
Young, P., Lai, A., Hodosh, M., & Hockenmaier, J. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00166 · Cited by 2,426
UNITER: UNiversal Image-TExt Representation Learning
Chen, Y.-C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., & Liu, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58577-8_7 · Cited by 2,055
Toward an architecture for never-ending language learning
Carlson, A., Betteridge, J., Kisiel, B., Settles, B., Hruschka, E., & Mitchell, T. (2010) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v24i1.7519 · Cited by 2,000
80 Million Tiny Images: A Large Data Set for Nonparametric Object and Scene Recognition
Torralba, A., Fergus, R., & Freeman, W. T. (2008) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.128 · Cited by 1,921
Knowledge Graph Embedding via Dynamic Mapping Matrix
Ji, G., He, S., Xu, L., Liu, K., & Zhao, J. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.3115/v1/p15-1067 · Cited by 1,816
Collaborative Knowledge Base Embedding for Recommender Systems
Zhang, F., Yuan, N. J., Lian, D., Xie, X., & Ma, W.-Y. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939673 · Cited by 1,749
Self-citationKuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A., et al. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-020-01316-z · Cited by 1,738
ConceptNet — A Practical Commonsense Reasoning Tool-Kit
Liu, H., & Singh, P. (2004) · BT Technology Journal · doi:10.1023/b:bttj.0000047600.45421.6d · Cited by 1,558
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks
Li, X., Yin, X., Li, C., Zhang, P., Hu, X., Zhang, L., Wang, L., Hu, H., Dong, L., Wei, F., et al. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58577-8_8 · Cited by 1,542
GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering
Hudson, D. A., & Manning, C. D. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00686 · Cited by 1,451
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., & Lazebnik, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.303 · Cited by 1,444
Stacked Cross Attention for Image-Text Matching
Lee, K.-H., Chen, X., Hua, G., Hu, H., & He, X. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01225-0_13 · Cited by 1,402
Scene Graph Generation by Iterative Message Passing
Xu, D., Zhu, Y., Choy, C. B., & Fei-Fei, L. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.330 · Cited by 1,271
CLIPScore: A Reference-free Evaluation Metric for Image Captioning
Hessel, J., Holtzman, A., Forbes, M., Le Bras, R., & Choi, Y. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.595 · Cited by 1,232
Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
Kiros, R., Salakhutdinov, R., & Zemel, R. S. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.2539 · Cited by 1,187
Deep multimodal learning: a survey on recent advances and trends
Ramachandram, D., & Taylor, G. W. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2738401 · Cited by 1,139 · cited 1×
“task in multi-modal pretrained language models [45].”
Grounded Language-Image Pre-training
Li, L. H., Zhang, P., Zhang, H., Yang, J., Li, C., Zhong, Y., Wang, L., Yuan, L., Zhang, L., Hwang, J.-N., et al. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01069 · Cited by 1,086
Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data
Bender, E. M., & Koller, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.463 · Cited by 1,082
Visual Relationship Detection with Language Priors
Lu, C., Krishna, R., Bernstein, M., & Fei-Fei, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0_51 · Cited by 1,069
Neural Motifs: Scene Graph Parsing with Global Context
Zellers, R., Yatskar, M., Thomson, S., & Choi, Y. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00611 · Cited by 1,052
ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
Kim, W., Son, B., & Kim, I. (2021) · arXiv · doi:10.48550/arxiv.2102.03334 · Cited by 951
Graph R-CNN for Scene Graph Generation
Yang, J., Lu, J., Lee, S., Batra, D., & Parikh, D. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01246-5_41 · Cited by 851
BabyTalk: Understanding and Generating Simple Image Descriptions
Kulkarni, G., Premraj, V., Ordonez, V., Dhar, S., Li, S., Choi, Y., Berg, A. C., & Berg, T. L. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.162 · Cited by 839
Probase: a probabilistic taxonomy for text understanding
Wu, W., Li, H., Wang, H., & Zhu, K. Q. (2012) · Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data · doi:10.1145/2213836.2213891 · Cited by 763
Unbiased Scene Graph Generation From Biased Training
Tang, K., Niu, Y., Huang, J., Shi, J., & Zhang, H. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00377 · Cited by 755
OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge
Marino, K., Rastegari, M., Farhadi, A., & Mottaghi, R. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00331 · Cited by 703
Top-Down Neural Attention by Excitation Backprop
Zhang, J., Bargal, S. A., Lin, Z., Brandt, J., Shen, X., & Sclaroff, S. (2018) · International Journal of Computer Vision · doi:10.1007/s11263-017-1059-x · Cited by 680
A Review and Meta-Analysis of Multimodal Affect Detection Systems
D'mello, S. K., & Kory, J. (2015) · ACM Computing Surveys · doi:10.1145/2682899 · Cited by 628
Knowledge Graph Embedding for Link Prediction
Rossi, A., Barbosa, D., Firmani, D., Matinata, A., & Merialdo, P. (2021) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3424672 · Cited by 601 · cited 1×
“Then in the prediction stage, the most matching h in (?, r, t) is found by ranking all candidate head entities h∗ according to a score function like argmaxh∗ φ(h ∗, r, t), where the score function is diverse in different embedding models [154].”
BackgroundDual-path Convolutional Image-Text Embeddings with Instance Loss
Zheng, Z., Zheng, L., Garrett, M., Yang, Y., Xu, M., & Shen, Y.-D. (2020) · ACM Transactions on Multimedia Computing, Communications, and Applications · doi:10.1145/3383184 · Cited by 589
Visual Translation Embedding Network for Visual Relation Detection
Zhang, H., Kyaw, Z., Chang, S.-F., & Chua, T.-S. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.331 · Cited by 586
The (Un)reliability of Saliency Methods
Kindermans, P.-J., Hooker, S., Adebayo, J., Alber, M., Schütt, K. T., Dähne, S., Erhan, D., & Kim, B. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-28954-6_14 · Cited by 580
Modeling mutual context of object and human pose in human-object interaction activities
Yao, B., & Fei-Fei, L. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5540235 · Cited by 573
Learning to Compose Dynamic Tree Structures for Visual Contexts
Tang, K., Zhang, H., Wu, B., Luo, W., & Liu, W. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00678 · Cited by 543
Detecting Visual Relationships with Deep Relational Networks
Dai, B., Zhang, Y., & Lin, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.352 · Cited by 502
Guadarrama, S., Krishnamoorthy, N., Malkarnenkar, G., Venugopalan, S., Mooney, R., Darrell, T., & Saenko, K. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.337 · Cited by 500
FVQA: Fact-Based Visual Question Answering
Wang, P., Wu, Q., Shen, C., Dick, A., & van den Hengel, A. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2754246 · Cited by 479
Counterfactual VQA: A Cause-Effect Look at Language Bias
Niu, Y., Tang, K., Zhang, H., Lu, Z., Hua, X.-S., & Wen, J.-R. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01251 · Cited by 418
NEIL: Extracting Visual Knowledge from Web Data
Chen, X., Shrivastava, A., & Gupta, A. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.178 · Cited by 408
Adaptive Co-attention Network for Named Entity Recognition in Tweets
Zhang, Q., Fu, J., Liu, X., & Huang, X. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11962 · Cited by 381
Learning Semantic Concepts and Order for Image and Sentence Matching
Huang, Y., Wu, Q., Song, C., & Wang, L. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00645 · Cited by 342
Multimodal Convolutional Neural Networks for Matching Image and Sentence
Ma, L., Lu, Z., Shang, L., & Li, H. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.301 · Cited by 334
Grouplet: A structured image representation for recognizing human and object interactions
Yao, B., & Fei-Fei, L. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5540234 · Cited by 328
Visual Relationship Detection with Internal and External Linguistic Knowledge Distillation
Yu, R., Li, A., Morariu, V. I., & Davis, L. S. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.121 · Cited by 310
Action recognition from a distributed representation of pose and appearance
Maji, S., Bourdev, L., & Malik, J. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995631 · Cited by 302
Image-embodied Knowledge Representation Learning
Xie, R., Liu, Z., Luan, H., & Sun, M. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/438 · Cited by 268
Multi-modal Knowledge Graphs for Recommender Systems
Sun, R., Cao, X., Zhao, Y., Wan, J., Zhou, K., Zhang, F., Wang, Z., & Zheng, K. (2020) · Proceedings of the 29th ACM International Conference on Information & Knowledge Management · doi:10.1145/3340531.3411947 · Cited by 265
MMKG: Multi-modal Knowledge Graphs
Liu, Y., Li, H., Garcia-Duran, A., Niepert, M., Onoro-Rubio, D., & Rosenblum, D. S. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-21348-0_30 · Cited by 257
Yang, K., Qinami, K., Fei-Fei, L., Deng, J., & Russakovsky, O. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3375709 · Cited by 254
Graphical Contrastive Losses for Scene Graph Parsing
Zhang, J., Shih, K. J., Elgammal, A., Tao, A., & Catanzaro, B. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.01180 · Cited by 252
Gradient-Based Attribution Methods
Ancona, M., Ceolini, E., Öztireli, C., & Gross, M. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-28954-6_9 · Cited by 252
CN-DBpedia: A Never-Ending Chinese Knowledge Extraction System
Xu, B., Xu, Y., Liang, J., Xie, C., Liang, B., Cui, W., & Xiao, Y. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-60045-1_44 · Cited by 244
Self-citationCross-modal Scene Graph Matching for Relationship-aware Image-Text Retrieval
Wang, S., Wang, R., Yao, Z., Shan, S., & Chen, X. (2020) · 2020 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv45572.2020.9093614 · Cited by 244
Situation Recognition: Visual Semantic Role Labeling for Image Understanding
Yatskar, M., Zettlemoyer, L., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.597 · Cited by 242
Explicit Knowledge-based Reasoning for Visual Question Answering
Wang, P., Wu, Q., Shen, C., Dick, A., & van den Hengel, A. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/179 · Cited by 233
Towards a Relevant and Diverse Search of Social Images
Wang, M., Yang, K., Hua, X., & Zhang, H.-J. (2010) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2010.2055045 · Cited by 228
Hybrid Transformer with Multi-level Fusion for Multimodal Knowledge Graph Completion
Chen, X., Zhang, N., Li, L., Deng, S., Tan, C., Xu, C., Huang, F., Si, L., & Chen, H. (2022) · Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3477495.3531992 · Cited by 226
Huang, T.-H. (., Ferraro, F., Mostafazadeh, N., Misra, I., Agrawal, A., Devlin, J., Girshick, R., He, X., Kohli, P., Batra, D., et al. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1147 · Cited by 218
ERNIE-ViL: Knowledge Enhanced Vision-Language Representations through Scene Graphs
Yu, F., Tang, J., Yin, W., Sun, Y., Tian, H., Wu, H., & Wang, H. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i4.16431 · Cited by 218
Visual diversification of image search results
van Leuken, R. H., Garcia, L., Olivares, X., & van Zwol, R. (2009) · Proceedings of the 18th international conference on World wide web · doi:10.1145/1526709.1526756 · Cited by 214
Bastings, J., & Filippova, K. (2020) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2020.blackboxnlp-1.14 · Cited by 207
Show, Control and Tell: A Framework for Generating Controllable and Grounded Captions
Cornia, M., Baraldi, L., & Cucchiara, R. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00850 · Cited by 205
FILIP: Fine-grained Interactive Language-Image Pre-Training
Yao, L., Huang, R., Hou, L., Lu, G., Niu, M., Xu, H., Liang, X., Li, Z., Jiang, X., & Xu, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.07783 · Cited by 202
Self-citationLanguage-Driven Temporal Activity Localization: A Semantic Matching Reinforcement Learning Model
Wang, W., Huang, Y., & Wang, L. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00042 · Cited by 198
KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQA
Marino, K., Chen, X., Parikh, D., Gupta, A., & Rohrbach, M. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01389 · Cited by 185
Consensus-Aware Visual-Semantic Embedding for Image-Text Matching
Wang, H., Zhang, Y., Ji, Z., Pang, Y., & Ma, L. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58586-0_2 · Cited by 174
Embedding Multimodal Relational Data for Knowledge Base Completion
Pezeshkpour, P., Chen, L., & Singh, S. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1359 · Cited by 173
Multi-Scale Dynamic Convolutional Network for Knowledge Graph Embedding
Zhang, Z., Li, Z., Liu, H., & Xiong, N. N. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2020.3005952 · Cited by 168
Self-citationThe symbol grounding problem has been solved, so what's next?
Steels, L. (2008) · Symbols and EmbodimentDebates on meaning and cognition · doi:10.1093/acprof:oso/9780199217274.003.0012 · Cited by 164
Large-Scale Visual Relationship Understanding
Zhang, J., Kalantidis, Y., Rohrbach, M., Paluri, M., Elgammal, A., & Elhoseiny, M. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33019185 · Cited by 161
KVQA: Knowledge-Aware Visual Question Answering
Shah, S., Mishra, A., Yadati, N., & Talukdar, P. P. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33018876 · Cited by 160
A Multimodal Translation-Based Approach for Knowledge Graph Representation Learning
Mousselly Sergieh, H., Botschen, T., Gurevych, I., & Roth, S. (2018) · Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/s18-2027 · Cited by 159
Multimodal Relation Extraction with Efficient Graph Alignment
Zheng, C., Feng, J., Fu, Z., Cai, Y., Li, Q., & Wang, T. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3476968 · Cited by 159
Grounded Situation Recognition
Pratt, S. M., Yatskar, M., Weihs, L., Farhadi, A., & Kembhavi, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58548-8_19 · Cited by 148
Top-Down Visual Saliency Guided by Captions
Ramanishka, V., Das, A., Zhang, J., & Saenko, K. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.334 · Cited by 142
Recognize complex events from static images by fusing deep channels
Xiong, Y., Zhu, K., Lin, D., & Tang, X. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298768 · Cited by 139
Cross-modal knowledge reasoning for knowledge-based visual question answering
Yu, J., Zhu, Z., Wang, Y., Zhang, W., Hu, Y., & Tan, J. (2020) · Pattern Recognition · doi:10.1016/j.patcog.2020.107563 · Cited by 122
Cross-media Structured Common Space for Multimedia Event Extraction
Li, M., Zareian, A., Zeng, Q., Whitehead, S., Lu, D., Ji, H., & Chang, S. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.230 · Cited by 118
GAIA: A Fine-grained Multimedia Knowledge Extraction System
Li, M., Zareian, A., Lin, Y., Pan, X., Whitehead, S., Chen, B., Wu, B., Ji, H., Chang, S., Voss, C. R., et al. (2020) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2020.acl-demos.11 · Cited by 118
Multimodal Named Entity Disambiguation for Noisy Social Media Posts
Moon, S., Neves, L., & Carvalho, V. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1186 · Cited by 109
Richpedia: A Large-Scale, Comprehensive Multi-Modal Knowledge Graph
Wang, M., Wang, H., Qi, G., & Zheng, Q. (2020) · Big Data Research · doi:10.1016/j.bdr.2020.100159 · Cited by 109
Knowledge Aided Consistency for Weakly Supervised Phrase Grounding
Chen, K., Gao, J., & Nevatia, R. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00425 · Cited by 107
Zheng, C., Wu, Z., Feng, J., Fu, Z., & Cai, Y. (2021) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme51207.2021.9428274 · Cited by 101
MMEA: Entity Alignment for Multi-modal Knowledge Graph
Chen, L., Li, Z., Wang, Y., Xu, T., Wang, Z., & Chen, E. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-55130-8_12 · Cited by 99
Self-citationMulti-Level Multimodal Common Semantic Space for Image-Phrase Grounding
Akbari, H., Karaman, S., Bhargava, S., Chen, B., Vondrick, C., & Chang, S. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.01276 · Cited by 95
Transform and Tell: Entity-Aware News Image Captioning
Tran, A., Mathews, A., & Xie, L. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01305 · Cited by 93
Zero-Shot Learning via Visual Abstraction
Antol, S., Zitnick, C. L., & Parikh, D. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-10593-2_27 · Cited by 87
Structured Matching for Phrase Localization
Wang, M., Azab, M., Kojima, N., Mihalcea, R. F., & Deng, J. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46484-8_42 · Cited by 86
Deep Variational Matrix Factorization with Knowledge Embedding for Recommendation System
Shen, X., Yi, B., Liu, H., Zhang, W., Zhang, Z., Liu, S., & Xiong, N. N. (2019) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2019.2952849 · Cited by 84
Boosting Entity-Aware Image Captioning With Multi-Modal Knowledge Graph
Zhao, W., & Wu, X. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3301279 · Cited by 83
Jointly optimising relevance and diversity in image retrieval
Thomas Deselaers, B., Gass, T., Dreuw, P., & Ney, H. (2009) · ACM International Conference on Image and Video Retrieval (CIVR) · doi:10.1145/1646396.1646443 · Cited by 81
Knowledge Aware Semantic Concept Expansion for Image-Text Matching
Shi, B., Ji, L., Lu, P., Niu, Z., & Duan, N. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/720 · Cited by 81
Multimodal Joint Attribute Prediction and Value Extraction for E-commerce Product
Zhu, T., Wang, Y., Li, H., Wu, Y., He, X., & Zhou, B. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.166 · Cited by 74
Multi-modal entity alignment in hyperbolic space
Guo, H., Tang, J., Zeng, W., Zhao, X. H., & Liu, L. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.03.132 · Cited by 71
IMGpedia: A Linked Dataset with Content-Based Analysis of Wikimedia Images
Ferrada, S., Bustos, B., & Hogan, A. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-68204-4_8 · Cited by 67
Multi-modal Knowledge-aware Reinforcement Learning Network for Explainable Recommendation
Tao, S., Qiu, R., Ping, Y., & Ma, H. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107217 · Cited by 66
MEduKG: A Deep-Learning-Based Approach for Multi-Modal Educational Knowledge Graph Construction
Li, N., Shen, Q., Song, R., Chi, Y., & Xu, H. (2022) · Information · doi:10.3390/info13020091 · Cited by 62
Zheng, W., Yan, L., Gou, C., Zhang, Z.-C., Zhang, J. J., Hu, M., & Wang, F. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.015 · Cited by 61
Multimodal Entity Linking for Tweets
Adjali, O., Besançon, R., Ferret, O., Borgne, H. L., & Grau, B. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-45439-5_31 · Cited by 59
HAKE: Human Activity Knowledge EnginePreprint
Li, Y., Xu, L., Liu, X., Huang, X., Xu, Y., Chen, M., Ma, Z., Wang, S., Fang, H.-S., & Lu, C. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.06539 · Cited by 54
Multimodal Named Entity Recognition with Image Attributes and Image Knowledge
Chen, D., Li, Z., Gu, B., & Chen, Z. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-73197-7_12 · Cited by 52
Self-citationCui, Y., Zhou, Y., Wang, C., Zhao, Z., Zhang, J., Wang, M., & Yu, J. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475251 · Cited by 51
Improving Event Extraction via Multimodal Integration
Zhang, T., Whitehead, S., Zhang, H., Li, H., Ellis, J. G., Huang, L., Liu, W., Ji, H., & Chang, S. (2017) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3123266.3123294 · Cited by 50
Computing iconic summaries of general visual concepts
Raguram, R., & Lazebnik, S. (2008) · 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops · doi:10.1109/cvprw.2008.4562959 · Cited by 49
Wen, H., Lin, Y., Lai, T., Pan, X., Li, S., Lin, X., Zhou, B., Li, M., Wang, H., Zhang, H., et al. (2021) · doi:10.18653/v1/2021.naacl-demos.16 · Cited by 48
Knowledge-Enriched Visual Storytelling
Hsu, C.-C., Chen, Z., Hsu, C.-Y., Li, C.-C., Lin, T.-Y., Huang, T.-H., & Ku, L. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6303 · Cited by 47
Multimodal Knowledge Graph for Deep Learning Papers and Code
Kannan, A. V., Fradkin, D., Akrotirianakis, I. G., Kulahcioglu, T., Canedo, A., Roy, A., Yu, S.-Y., Malawade, A. V., & Faruque, M. A. A. (2020) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3340531.3417439 · Cited by 46
Image and Sentence Matching via Semantic Concepts and Order Learning
Huang, Y., Wu, Q., Wang, W., & Wang, L. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2883466 · Cited by 43
GAKG: A Multimodal Geoscience Academic Knowledge Graph
Deng, C., Jia, Y., Xu, H., Zhang, C., Tang, J., Fu, L., Zhang, W., Zhang, H., Wang, X., & Zhou, C. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482003 · Cited by 43
Self-citationPerona, P. (2010) · Proceedings of the IEEE · doi:10.1109/jproc.2010.2049621 · Cited by 41
VisualSem: a high-quality knowledge graph for vision and language
Alberts, H., Huang, N. T., Deshpande, Y., Liu, Y., Cho, K., Vania, C., & Calixto, I. (2021) · Workshop on Multilingual Representation Learning (MRL) · doi:10.18653/v1/2021.mrl-1.13 · Cited by 39
Attention-Based Multimodal Entity Linking with High-Quality Images
Zhang, L., Li, Z., & Yang, Q. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-73197-7_35 · Cited by 36
Self-citationJoint Multimedia Event Extraction from Video and Article
Chen, B., Lin, X., Thomas, C., Li, M., Yoshida, S., Chum, L., Ji, H., & Chang, S. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.8 · Cited by 36
Harnad, S. R. (2005) · Wiley eBooks · doi:10.1002/0470018860.s00025 · Cited by 34
Matching Image and Sentence with Multi-faceted Representations
Ma, L., Jiang, W., Jie, Z., Jiang, Y., & Liu, W. (2019) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2019.2916167 · Cited by 31
Bloem, P., Wilcke, X., Berkel, L. V., & de Boer, V. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-77385-4_37 · Cited by 31
Thoma, S., Rettinger, A., & Both, F. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-68288-4_41 · Cited by 30
Oracle Bone Inscriptions information processing based on multi-modal knowledge graph
Xiong, J., Liu, G., Liu, Y., & Liu, M. (2021) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2021.107173 · Cited by 28
Enhancing the Quality of Image Tagging Using a Visio-Textual Knowledge Base
Chaudhary, C., Goyal, P., Prasad, D. N., & Chen, Y. P. (2019) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2937181 · Cited by 26
Do Dogs have Whiskers? A New Knowledge Base of hasPart RelationsPreprint
Bhakthavatsalam, S., Richardson, K., Tandon, N., & Clark, P. E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.07510 · Cited by 20
Xu, G., Chen, H., Li, F.-L., Sun, F. X., Shi, Y., Zeng, Z., Zhou, W., Zhao, Z., & Zhang, J. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3481983 · Cited by 19
Event Specific Multimodal Pattern Mining for Knowledge Base Construction
Li, H., Ellis, J. G., Ji, H., & Chang, S. (2016) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2964284.2964287 · Cited by 16
Relational Reasoning using Prior Knowledge for Visual CaptioningPreprint
Hou, J., Wu, X., Qi, Y., Zhao, W., Luo, J., & Jia, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.01290 · Cited by 16
From Vision to Content: Construction of Domain-Specific Multi-Modal Knowledge Graph
Zhang, X., Sun, X., Xie, C., & Lun, B. (2019) · IEEE Access · doi:10.1109/access.2019.2933370 · Cited by 15
One-Shot Learning for Long-Tail Visual Relation Detection
Wang, W., Wang, M., Wang, S., Long, G., Yao, L., Qi, G., & Chen, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6904 · Cited by 15
MMEKG: Multi-modal Event Knowledge Graph towards Universal Representation across Modalities
Ma, Y., Wang, Z., Li, M., Cao, Y., Chen, M., Li, X., Sun, W., Deng, K., Wang, K., Sun, A., et al. (2022) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2022.acl-demo.23 · Cited by 12
Memory-Based Network for Scene Graph with Unbalanced Relations
Wang, W., Liu, R., Wang, M., Wang, S., Chang, X., & Chen, Y. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413507 · Cited by 11
Answering Visual-Relational Queries in Web-Extracted Knowledge Graphs
Oñoro-Rubio, D., Niepert, M., García-Durán, A., González-Sánchez, R., & López-Sastre, R. J. (2019) · Automated Knowledge Base Construction · Cited by 9
Towards Using Semantic-Web Technologies for Multi-Modal Knowledge Graph Construction
Baumgartner, M., Rossetto, L., & Bernstein, A. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3416292 · Cited by 7
Sherlock: Scalable Fact Learning in Images
Elhoseiny, M., Cohen, S. D., Chang, W. H., Price, B. L., & Elgammal, A. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11214 · Cited by 6
Jiang, X., Li, A., Liang, J., Liu, B., Xie, R., Wu, W., Li, Z., & Xiao, Y. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-00123-9_14 · Cited by 4
Self-citation
Cited by
Unifying Large Language Models and Knowledge Graphs: A Roadmap
Pan, S., Luo, L., Wang, Y., Chen, C., Wang, J., & Wu, X. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3352100 · Cited by 1,376
Multimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072
Self-citationKnowledge Graphs: Opportunities and Challenges
Peng, C., Xia, F., Naseriparsa, M., & Osborne, F. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10465-9 · Cited by 717
A Survey of Knowledge Graph Reasoning on Graph Types: Static, Dynamic, and Multi-Modal
Liang, K., Meng, L., Liu, M., Liu, Y., Tu, W., Wang, S., Zhou, S., Liu, X., Sun, F., & He, K. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3417451 · Cited by 330
Metadata
- Authors
- Zhu, X., Li, Z., Wang, X., Jiang, X., Sun, P., Wang, X., Xiao, Y., & Yuan, N. J.
- Venue
- IEEE Transactions on Knowledge and Data Engineering
- Published
- 2024-02
- Volume
- 36
- Issue
- 2
- Pages
- 715-735
- ISSN
- 1041-4347 (ISSN-L)
- 1558-2191
- 2326-3865
- OpenAlex
- W4312044727
- Cited by
- 291
- References
- 151
- FWCI
- 17.97
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.