A Review on Explainability in Multimodal Deep Neural Nets
Open accessThe DOI matches its Crossref record; 267 of the 267 references it cites were found in a registry or an index.
Abstract
Artificial Intelligence techniques powered by deep neural nets have achieved much success in several application domains, most significantly and notably in the Computer Vision applications and Natural Language Processing tasks. Surpassing human-level performance propelled the research in the applications where different modalities amongst language, vision, sensory, text play an important role in accurate predictions and identification. Several multimodal fusion methods employing deep learning models are proposed in the literature. Despite their outstanding performance, the complex, opaque and black-box nature of the deep neural nets limits their social acceptance and usability. This has given rise to the quest for model interpretability and explainability, more so in the complex tasks involving multimodal AI methods. This paper extensively reviews the present literature to present a comprehensive survey and commentary on the explainability in multimodal deep neural nets, especially for the vision and language tasks. Several topics on multimodal AI and its applications for generic domains have been covered in this paper, including the significance, datasets, fundamental building blocks of the methods and techniques, challenges, applications, and future trends in this domain.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The black box and opaque nature of deep neural networks precludes their usefulness despite excellent performance of multimodal methods [288].”
From Survey of explainable artificial intelligence techniques for biomedical imaging with deep neural networks (2023) · cites this paper as [288]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AI systems can profit significantly from assimilating multimodal data into prediction and classification models to imitate integrative human clinical decision-making.”
From Orchestrating explainable artificial intelligence for multimodal and longitudinal data in medical imaging (2024) · cites this paper as [17]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This can boost their robustness and accuracy, enable the discovery of new biomarkers and therapeutic targets6,14, as well as improve model performance15–17.”
From Orchestrating explainable artificial intelligence for multimodal and longitudinal data in medical imaging (2024) · cites this paper as [17]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“[165] Attention Mechanism Focuses and attends the regions that are important for making”
BackgroundHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“BERT Transformer (B2T2) [98], UnicodeText Unidirectional long short term memory(LSTM)[104], Bidirectional LSTM (BiLSTM), Unidirectional Gated Recurrent Units (GRU)[105],bidirectional GRU (BiGRU) VL [99], VLBERT[100], UNITER [101], ViLBert [102], LXMERT [103]”
BackgroundLeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.81 · Cited by 32,684
Greedy function approximation: A gradient boosting machine
Friedman, J. H. (2001) · The Annals of Statistics · doi:10.1214/aos/1013203451 · Cited by 30,731 · cited 3×
“Individual conditional expectation (ICE) plot [145] visualize the dependence of features on the final prediction for unique data points, providing more insights than the PDP approach of averaging on all data points.”
BackgroundA Unified Approach to Interpreting Model Predictions
Lundberg, S., & Lee, S.-I. (2017) · arXiv · doi:10.48550/arxiv.1705.07874 · Cited by 23,707 · cited 1×
“Shapley values [146] are used when each data point’s contribution is variable in the final prediction.”
MethodBLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 2×
“Automated evaluation metrics such as BLEU-4 [215] is used for automatic evaluation of machine translation that is”
Method"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839 · cited 1×
“Deep learning-based explanatory systems are broadly classified into justification [17] or introspection-based systems [155], [165].”
BackgroundLearning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 1×
“The second breakthrough in Multimodal AI introduces Open AI’s CLIPmodel [230] trained on image sentence similarity scores connecting images and text model that stands for contrastive language image pretraining classifying image text pairs.”
BackgroundGauriat, C.-M., Pencolé, Y., Ribot, P., & Brouillet, G. (2024) · HAL (Le Centre pour la Communication Scientifique Directe) · doi:10.4230/oasics.dx.2024.27 · Cited by 12,586
Learning Deep Features for Discriminative Localization
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.319 · Cited by 11,428 · cited 1×
“Deep learning-based explanatory systems are broadly classified into justification [17] or introspection-based systems [155], [165].”
BackgroundArrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321 · cited 3×
“The XAI field is continuously evolving and is critical in developing new AI algorithms and methods to explore how the models work and why they succeed or fail, to improve their design driving us towards the responsible AI paradigm of the future [24].”
BackgroundEmpirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008 · cited 1×
“BERT Transformer (B2T2) [98], UnicodeText Unidirectional long short term memory(LSTM)[104], Bidirectional LSTM (BiLSTM), Unidirectional Gated Recurrent Units (GRU)[105],bidirectional GRU (BiGRU) VL [99], VLBERT[100], UNITER [101], ViLBert [102], LXMERT [103]”
BackgroundRudin, C. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0048-x · Cited by 10,561 · cited 1×
“An enormous scope of improvement exists for building interpretable and explainable models for multimodal settings; instead of explaining the model predictions in a post hoc manner inherent and self-explainingmodel should be developed leveraging visual and textual modalities integrated benefits [228].”
BackgroundBarredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682
Large-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354 · cited 2×
“A ‘‘slow or gradual fusion’’ approach to integrating multiple fusion layers by fusing modalities from their high to low contribution performs well [112].”
BackgroundPeeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)
Adadi, A., & Berrada, M. (2018) · IEEE Access · doi:10.1109/access.2018.2870052 · Cited by 6,294 · cited 2×
“The existing literature has a number of surveys on explainable AI such as [5], [8], [24]–[30] and on explainable deep learning [31], [32].”
BackgroundHearing lips and seeing voices
MCGURK, H., & MACDONALD, J. (1976) · Nature · doi:10.1038/264746a0 · Cited by 6,183 · cited 1×
“The McGurk effect [39] highlighted that the audio and visual”
BackgroundLipton, Z. C. (2018) · Queue · doi:10.1145/3236386.3241340 · Cited by 5,427
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370 · cited 2×
“Human interpretable explanations such as gradient-based methods [140], [141] have been proposed for the convolution neural networks.”
MethodExplanation in artificial intelligence: Insights from the social sciences
Miller, T. (2019) · Artificial Intelligence · doi:10.1016/j.artint.2018.07.007 · Cited by 5,302
Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0981-7 · Cited by 5,107 · cited 2×
“0 and Visual Genome datasets for the task-specific release [192].”
BackgroundMultimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018
CIDEr: Consensus-based image description evaluation
Vedantam, R., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299087 · Cited by 4,845
VQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
Axiomatic attribution for deep networks
Sundararajan, M., Taly, A., & Yan, Q. (2017) · arXiv · doi:10.48550/arxiv.1703.01365 · Cited by 4,087 · cited 3×
“Gradient Input [151] improves the sharpness VOLUME 9, 2021 59807 of the attribution maps.”
MethodStriving for Simplicity: The All Convolutional Net
Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2014) · arXiv · doi:10.48550/arxiv.1412.6806 · Cited by 3,447 · cited 1×
“Human interpretable explanations such as gradient-based methods [140], [141] have been proposed for the convolution neural networks.”
MethodTowards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121 · cited 2×
“It is hard to understand, interpret, and explain these models’ internal processing and decision-making processes, limiting their social acceptance and usability [30].”
BackgroundExplainable AI: a review of machine learning interpretability methods
Linardatos, P., Papastefanopoulos, V., & Kotsiantis, S. (2020) · Entropy · doi:10.3390/e23010018 · Cited by 2,922 · cited 1×
“[81] Explainability methods for the critical domain of healthcare are discussed.”
MethodRecent Trends in Deep Learning Based Natural Language Processing [Review Article]
Young, T., Hazarika, D., Poria, S., & Cambria, E. (2018) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2018.2840738 · Cited by 2,921
Learning Important Features Through Propagating Activation Differences
Shrikumar, A., Greenside, P., & Kundaje, A. (2017) · arXiv · doi:10.48550/arxiv.1704.02685 · Cited by 2,761 · cited 2×
“Gradient Input [151] improves the sharpness”
(2011) · Cited by 2,752
Methods for interpreting and understanding deep neural networks
Montavon, G., Samek, W., & Müller, K.-R. (2018) · Digital Signal Processing · doi:10.1016/j.dsp.2017.10.011 · Cited by 2,665 · cited 3×
“In [32], a survey of different methods on understanding and interpreting deep neural nets are discussed.”
BackgroundA Survey on Explainable Artificial Intelligence (XAI): Toward Medical XAI
Tjoa, E., & Guan, C. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.3027314 · Cited by 2,524
Explaining Explanations: An Overview of Interpretability of Machine Learning
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2018) · 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa.2018.00018 · Cited by 2,507 · cited 3×
“humans understand and interpret their working mechanism and decision-making process by asking questions like why the system made a particular prediction? Why answer the interpretability aspect, and how justifies how the system came up to a specific decision answer the explainability part [27].”
BackgroundMethodInfluentialZero-shot text-to-image generation
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., & Sutskever, I. (2021) · arXiv · doi:10.48550/arxiv.2102.12092 · Cited by 2,417 · cited 1×
“Dall E [229] is a transformer-based image generation model from text captions that fills the gap between vision and text for a wide range of concepts explained in natural language.”
BackgroundLXMERT: Learning Cross-Modality Encoder Representations from Transformers
Tan, H., & Bansal, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1514 · Cited by 2,319 · cited 2×
“BERT Transformer (B2T2) [98], UnicodeText Unidirectional long short term memory(LSTM)[104], Bidirectional LSTM (BiLSTM), Unidirectional Gated Recurrent Units (GRU)[105],bidirectional GRU (BiGRU) VL [99], VLBERT[100], UNITER [101], ViLBert [102], LXMERT [103]”
BackgroundInfluentialMaking the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., & Parikh, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.670 · Cited by 2,287
Score normalization in multimodal biometric systems
Jain, A., Nandakumar, K., & Ross, A. (2005) · Pattern Recognition · doi:10.1016/j.patcog.2005.01.012 · Cited by 2,154 · cited 1×
“Optimal integration of modalities, involve rules such as the Bayes rule [107],max-fusion [108], averagefusion [109], a majority vote for this approach.”
MethodEuropean Union Regulations on Algorithmic Decision Making and a “Right to Explanation”
Goodman, B., & Flaxman, S. (2017) · AI Magazine · doi:10.1609/aimag.v38i3.2741 · Cited by 2,131 · cited 1×
“Explainability is a legal concern to comply with the EU General Data Protection and Regulation (GDPR) act asking for ‘‘Right to explanation’’ to the users of an automated decision-making system [133].”
BackgroundViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
Lu, J., Batra, D., Parikh, D., & Lee, S. (2019) · arXiv · doi:10.48550/arxiv.1908.02265 · Cited by 2,109
Tensor Fusion Network for Multimodal Sentiment Analysis
Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1115 · Cited by 2,102 · cited 1×
“The tensor fusion network, where the unimodal, bimodal, and trimodal interactions are modeled using a 3-fold Cartesian product, is presented in [114].”
MethodUNITER: UNiversal Image-TExt Representation Learning
Chen, Y.-C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., & Liu, J. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58577-8_7 · Cited by 2,055 · cited 1×
“BERT Transformer (B2T2) [98], UnicodeVL [99], VLBERT[100], UNITER [101], ViLBert [102], LXMERT [103] Text Unidirectional long short term memory(LSTM)[104],”
BackgroundSPICE: Semantic Propositional Image Caption Evaluation
Anderson, P., Fernando, B., Johnson, M., & Gould, S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46454-1_24 · Cited by 1,986 · cited 2×
“[218] P. Anderson, B. Fernando, M. Johnson, and S. Gould, ‘‘SPICE: Semantic propositional image caption evaluation,’’ in Proc.”
ResultA Survey of Autonomous Driving: Common Practices and Emerging Technologies
Yurtsever, E., Lambert, J., Carballo, A., & Takeda, K. (2020) · IEEE Access · doi:10.1109/access.2020.2983149 · Cited by 1,889 · cited 1×
“[74] Camera, LiDAR, Radar, IMU, GNSS Task-related to perception, localization planning, and decision making are reviewed with the tools and datasets.”
MethodGoldstein, A., Kapelner, A., Bleich, J., & Pitkin, E. (2015) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.2014.907095 · Cited by 1,861 · cited 1×
“Individual conditional expectation (ICE) plot [145] visualize the dependence of features on the final prediction for unique data points, providing more insights than the PDP approach of averaging on all data points.”
MethodMachine Learning Interpretability: A Survey on Methods and Metrics
Carvalho, D. V., Pereira, E. M., & Cardoso, J. S. (2019) · Electronics · doi:10.3390/electronics8080832 · Cited by 1,842 · cited 1×
“EXPLAINATION EVALUATION Evaluation of explanations with metrics and methods is a major point of concern to leverage explainability and user confidence as they are task-specific and subjective [25].”
BackgroundMultimodal Learning with Deep Boltzmann Machines
Srivastava, N., & Salakhutdinov, R. (2012) · Journal of Machine Learning Research · doi:10.5555/2627435.2697059 · Cited by 1,807 · cited 3×
“than the intermodal one; the greedy layer-wise pretraining approach is also used in different settings [46].”
MethodCLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.215 · Cited by 1,781
Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Fukui, A., Park, D. H., Yang, D., Rohrbach, A., Darrell, T., & Rohrbach, M. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1044 · Cited by 1,582 · cited 1×
“[116], [117], which lacks exploiting the multilinear fusion power [118].”
BackgroundUnderstanding Black-box Predictions via Influence Functions
Koh, P. W., & Liang, P. (2017) · arXiv · doi:10.48550/arxiv.1703.04730 · Cited by 1,514 · cited 1×
“They are used to understandmodel behavior, model debugging, detect dataset errors, and even create adversarial attacks [148].”
BackgroundFeng, D., Haase-Schutz, C., Rosenbaum, L., Hertlein, H., Glaser, C., Timm, F., Wiesbeck, W., & Dietmayer, K. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.2972974 · Cited by 1,425 · cited 1×
“[75] Camera, LiDAR, Radar Tasks related to deep object detection and segmentation with associated fusion challenges are detailed and addressed.”
MethodMultimodal Data Fusion: An Overview of Methods, Challenges, and Prospects
Lahat, D., Adali, T., & Jutten, C. (2015) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2460697 · Cited by 1,296 · cited 2×
“Multimodality extracts and combines vital information from the respective modality source and solves a given problem with richer representation and performance than the individual modalities [9].”
BackgroundMultimodal fusion for multimedia analysis: a survey
Atrey, P. K., Hossain, M. A., El Saddik, A., & Kankanhalli, M. S. (2010) · Multimedia Systems · doi:10.1007/s00530-010-0182-0 · Cited by 1,279 · cited 1×
“as early [11], late [42], and hybrid fusion [43] schemes are developed to fuse themodalities at data, feature, decision, and intermediate mixed levels [44].”
BackgroundA survey on deep learning for big data
Zhang, Q., Yang, L. T., Chen, Z., & Li, P. (2018) · Information Fusion · doi:10.1016/j.inffus.2017.10.006 · Cited by 1,226
Anderson, P., Wu, Q., Teney, D., Bruce, J., Johnson, M., Sunderhauf, N., Reid, I., Gould, S., & van den Hengel, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00387 · Cited by 1,187
Efficient Low-rank Multimodal Fusion With Modality-Specific Factors
Liu, Z., Shen, Y., Lakshminarasimhan, V. B., Liang, P. P., Bagher Zadeh, A., & Morency, L.-P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1209 · Cited by 1,170 · cited 1×
“The low-rank multimodal fusion technique [115] addresses the shortfall of tensor fusion networks by using low-rank tensors for fusion but results in a complex architecture with a lot of computation and processing.”
BackgroundDeep multimodal learning: a survey on recent advances and trends
Ramachandram, D., & Taylor, G. W. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2738401 · Cited by 1,139 · cited 2×
“cross-modality associations and correlations between different modalities [51].”
BackgroundInterpretable Machine Learning
Molnar, C. (2020) · Cited by 1,104
Interpretable Explanations of Black Boxes by Meaningful Perturbation
Fong, R. C., & Vedaldi, A. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.371 · Cited by 1,097 · cited 1×
“Model agnostic methods [142], [143] that are independent and irrespective of the model have general modularity in design and can be applied for other domains.”
MethodVisual Relationship Detection with Language Priors
Lu, C., Krishna, R., Bernstein, M., & Fei-Fei, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0_51 · Cited by 1,069 · cited 1×
“In [194], graphs with only objects and relations are generated.”
BackgroundCounterfactual explanations without opening the black box: automated decisions and the GDPR
Wachter, S., Mittelstadt, B., & Russell, C. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.3063289 · Cited by 1,061
Meteor: an automatic metric for MT evaluation with high levels of correlation with human judgments
Lavie, A., & Agarwal, A. (2007) · Proceedings of the Second Workshop on Statistical Machine Translation - StatMT '07 · doi:10.3115/1626355.1626389 · Cited by 1,047 · cited 2×
“The ideal summaries created by humans, CIDEr [217], are a human consensus-based image description evaluation metric for automatically evaluating image descriptions.”
BackgroundAttention is not not Explanation
Wiegreffe, S., & Pinter, Y. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1002 · Cited by 966 · cited 1×
“Attention’s explanatory power is questionable [180] as they lack an association with the attention weights and gradients mapping for generating faithful explanations.”
BackgroundVL-BERT: Pre-training of Generic Visual-Linguistic Representations
Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., & Dai, J. (2019) · arXiv · doi:10.48550/arxiv.1908.08530 · Cited by 950
Huang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I., & Lungren, M. P. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00341-z · Cited by 934 · cited 1×
“The loss is propagated back to the feature extractor network during the training process [111].”
BackgroundEarly versus late fusion in semantic video analysis
Snoek, C. G. M., Worring, M., & Smeulders, A. W. M. (2005) · Proceedings of the 13th annual ACM international conference on Multimedia · doi:10.1145/1101149.1101236 · Cited by 879 · cited 2×
“as early [11], late [42], and hybrid fusion [43] schemes are developed to fuse themodalities at data, feature, decision, and intermediate mixed levels [44].”
BackgroundMethodMemory Fusion Network for Multi-view Sequential Learning
Zadeh, A., Liang, P. P., Mazumder, N., Poria, S., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12021 · Cited by 855
Unicoder-VL: A Universal Encoder for Vision and Language by Cross-Modal Pre-Training
Li, G., Duan, N., Fang, Y., Gong, M., & Jiang, D. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6795 · Cited by 795
Mittelstadt, B., Russell, C., & Wachter, S. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287574 · Cited by 794 · cited 1×
“it is found that many explanatory systems are based on scientific modeling than explanation generation [163] and hence are not user-centric and role-based.”
BackgroundA Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems
Mohseni, S., Zarei, N., & Ragan, E. D. (2021) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3387166 · Cited by 785 · cited 1×
“Multimodal explanations play a vital role in building intelligent systems powered with understanding and reasoning capabilities inherent and integral to humans [159].”
BackgroundA Multidisciplinary Survey and Framework for Design and Evaluation of Explainable AI Systems
Mohseni, S., Zarei, N., & Ragan, E. D. (2018) · arXiv · doi:10.48550/arxiv.1811.11839 · Cited by 783
From Recognition to Cognition: Visual Commonsense Reasoning
Zellers, R., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00688 · Cited by 769 · cited 1×
“The model’s ability to reason their predictions has become essential, leading to the emergence of Explainable AI for multimodal settings [57].”
BackgroundExplainable Deep Learning Models in Medical Image Analysis
Singh, A., Sengupta, S., & Lakshminarayanan, V. (2020) · Journal of Imaging · doi:10.3390/jimaging6060052 · Cited by 718 · cited 1×
“In [33], a survey of explainable deep learning applications for medical imaging tasks from the perceptive of deep learning researcher, developer, and end-user is presented.”
BackgroundYang, G., Ye, Q., & Xia, J. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.07.016 · Cited by 717
MUTAN: Multimodal Tucker Fusion for Visual Question Answering
Ben-younes, H., Cadene, R., Cord, M., & Thome, N. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.285 · Cited by 716 · cited 1×
“Other approaches such as Multimodal Tucker fusion decompose and form a low-rank matrix decomposition overcomes bilinear polling’s drawbacks by reducing complexity through tucker decomposition [120].”
MethodVL-BERT: Pre-training of Generic Visual-Linguistic Representations
Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., & Dai, J. (2020) · International Conference on Learning Representations · Cited by 707
Ensemble of keyword extraction methods and classifiers in text classification
Onan, A., Korukoğlu, S., & Bulut, H. (2016) · Expert Systems with Applications · doi:10.1016/j.eswa.2016.03.045 · Cited by 630 · cited 1×
“Multiple deep learning and embedding approaches are reported for NLP tasks such as text mining [35],[36], sentiment analysis [37],[38],sarcasm identification [39],[40], information extraction [41],[42],[43].”
BackgroundA Survey of Methods for Explaining Black Box Models
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2019) · ACM Computing Surveys · doi:10.1145/3236009 · Cited by 629 · cited 1×
“Deep neural nets lack interpretability as it is difficult to analyze which modalities or features are driving the predictions [28].”
BackgroundExplainable Deep Learning: A Field Guide for the UninitiatedPreprint
Ras, G., Xie, N., van Gerven, M. V., & Doran, D. (2020) · Journal of Artificial Intelligence Research · doi:10.48550/arxiv.2004.14545 · Cited by 622 · cited 1×
“This growing critical need and importance have led to several reviews emphasizing the multifaceted explainability topic in black-box models such as deep neural nets [23].”
BackgroundWhat Makes Training Multi-modal Classification Networks Hard?
Wang, W., Tran, D., & Feiszli, M. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01271 · Cited by 588
Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers
Hohman, F., Kahng, M., Pienta, R., & Chau, D. H. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2843369 · Cited by 582
Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering
Agrawal, A., Batra, D., Parikh, D., & Kembhavi, A. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00522 · Cited by 566
Handbook of multisensor data fusion : theory and practice
Liggins, M. E., Hall, D. L., & Llinas, J. (2009) · Cited by 556
(2016) · neural information processing systems · Cited by 556
Explainable Artificial Intelligence (XAI)
Hossain, M. (2021) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.6371435 · Cited by 541
Deep Multimodal Representation Learning: A Survey
Guo, W., Wang, J., & Wang, S. (2019) · IEEE Access · doi:10.1109/access.2019.2916887 · Cited by 535 · cited 2×
“In multimodal tasks, such as visual captioning, visual question answering, or visual entailment, attention mechanisms play a crucial role in the alignment and fusion across different modalities [86].”
BackgroundGenerating Visual Explanations
Hendricks, L. A., Akata, Z., Rohrbach, M., Donahue, J., Schiele, B., & Darrell, T. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_1 · Cited by 534
Poria, S., Cambria, E., & Gelbukh, A. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1303 · Cited by 533 · cited 2×
“converting data sources into a fixed representation is too difficult and time-consumingwith early fusion techniques [103].”
BackgroundLearning to Reason: End-to-End Module Networks for Visual Question Answering
Hu, R., Andreas, J., Rohrbach, M., Darrell, T., & Saenko, K. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.93 · Cited by 517 · cited 1×
“Modular approaches make the system interpretable by design [19], [20] but are majorly tested on synthetic datasets such as CLEVR [21].”
BackgroundRight for the Right Reasons: Training Differentiable Models by Constraining their Explanations
Ross, A. S., Hughes, M. C., & Doshi-Velez, F. (2017) · Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/371 · Cited by 515 · cited 1×
“3) Post hoc Modelling is implemented after the model is developed by extracting explanations for already developed models through perturbations [138], Backpropagation methods [14], and proxy models [139], as it can be freely applied to any model without any constraints.”
MethodMultimodal intelligence: Representation learning, information fusion, and applications
Zhang, C., Yang, Z., He, X., & Deng, L. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.2987728 · Cited by 511 · cited 1×
“as early [11], late [42], and hybrid fusion [43] schemes are developed to fuse themodalities at data, feature, decision, and intermediate mixed levels [44].”
BackgroundVisualizing Deep Convolutional Neural Networks Using Natural Pre-images
Mahendran, A., & Vedaldi, A. (2016) · International Journal of Computer Vision · doi:10.1007/s11263-016-0911-8 · Cited by 496 · cited 1×
“[162] Activation maximization Activation maximization activates few neurons over the others that are vital for making predictions.”
BackgroundMaking the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
Goyal, Y., Khot, T., Agrawal, A., Summers-Stay, D., Batra, D., & Parikh, D. (2019) · International Journal of Computer Vision · doi:10.1007/s11263-018-1116-0 · Cited by 487 · cited 1×
“To exploit the complementary assistive and explanatory information and improved predictive power from multiple modalities, a deep understanding of the model’s working, predictions, and flaw detection are of utmost importance by using the respective modalities [22].”
BackgroundInferring and Executing Programs for Visual Reasoning
Johnson, J., Hariharan, B., Van Der Maaten, L., Hoffman, J., Fei-Fei, L., Zitnick, C. L., & Girshick, R. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.325 · Cited by 467 · cited 1×
“Modular approaches make the system interpretable by design [19], [20] but are majorly tested on synthetic datasets such as CLEVR [21].”
BackgroundHandbook of Multisensor Data Fusion
(2017) · CRC Press eBooks · doi:10.1201/9781420053098 · Cited by 452 · cited 1×
“The second method is to combine data at its lower-dimensional latent subspace [99].”
Explainable Deep Learning: A Field Guide for the Uninitiated
Ras, G., Xie, N., Van Gerven, M., & Doran, D. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13200 · Cited by 446
Data fusion in intelligent transportation systems: Progress and challenges – A survey
Faouzi, N.-E. E., Leung, H., & Kurian, A. (2011) · Information Fusion · doi:10.1016/j.inffus.2010.06.001 · Cited by 444 · cited 1×
“Transportatio n Systems [68] various modalities Data fusion techniques applied to various transportation systems”
MethodVisual question answering: A survey of methods and datasets
Wu, Q., Teney, D., Wang, P., Shen, C., Dick, A., & van den Hengel, A. (2017) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2017.05.001 · Cited by 438
Multimodal Explanations: Justifying Decisions and Pointing to the Evidence
Park, D. H., Hendricks, L. A., Akata, Z., Rohrbach, A., Schiele, B., Darrell, T., & Rohrbach, M. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00915 · Cited by 418
Counterfactual VQA: A Cause-Effect Look at Language Bias
Niu, Y., Tang, K., Zhang, H., Lu, Z., Hua, X.-S., & Wen, J.-R. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01251 · Cited by 418 · cited 2×
“Such methods make multimodal frameworks more reliable, interpretable by allowing the models to provide reasonable predictions for the right reasons rather than looking for mere performance, looking at cause and effect aspects [185].”
BackgroundMethodCounterfactual Visual ExplanationsPreprint
Goyal, Y., Wu, Z., Ernst, J., Batra, D., Parikh, D., & Lee, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.07451 · Cited by 397 · cited 1×
“In [184], a counterfactual visual explanation is generated based on the paper by Hendricks et al.”
MethodSentiment analysis on product reviews based on weighted word embeddings and deep neural networks
Onan, A. (2021) · Concurrency and Computation: Practice and Experience · doi:10.1002/cpe.5909 · Cited by 396 · cited 1×
“Multiple deep learning and embedding approaches are reported for NLP tasks such as text mining [35],[36], sentiment analysis [37],[38],sarcasm identification [39],[40], information extraction [41],[42],[43].”
BackgroundNweke, H. F., Teh, Y. W., Mujtaba, G., & Al-garadi, M. A. (2019) · Information Fusion · doi:10.1016/j.inffus.2018.06.002 · Cited by 394 · cited 1×
“[67] Accelerometer and other sensors Predicting the movement of a person based on the sensor data”
BackgroundONAN, A. (2021) · Computer Applications in Engineering Education · doi:10.1002/cae.22253 · Cited by 382 · cited 1×
“Multiple deep learning and embedding approaches are reported for NLP tasks such as text mining [35],[36], sentiment analysis [37],[38],sarcasm identification [39],[40], information extraction [41],[42],[43].”
BackgroundInterpretable Learning for Self-Driving Cars by Visualizing Causal Attention
Kim, J., & Canny, J. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.320 · Cited by 366
Verma, G. K., & Tiwary, U. S. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2013.11.007 · Cited by 354 · cited 1×
“Optimal integration of modalities, involve rules such as the Bayes rule [107],max-fusion [108], averagefusion [109], a majority vote for this approach.”
Method(2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01234-2 · Cited by 353
Holzinger, A., Malle, B., Saranti, A., & Pfeifer, B. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.01.008 · Cited by 352 · cited 1×
“Constructing a multimodal feature representation space spanning diverse modalities from a causal and counterfactual perspective in the medical domain is detailed [36].”
BackgroundOnan, A., & Tocoglu, M. A. (2021) · IEEE Access · doi:10.1109/access.2021.3049734 · Cited by 347 · cited 2×
“Multiple deep learning and embedding approaches are reported for NLP tasks such as text mining [35],[36], sentiment analysis [37],[38],sarcasm identification [39],[40], information extraction [41],[42],[43].”
BackgroundA Joint Sequence Fusion Model for Video Question Answering and Retrieval
Yu, Y., Kim, J., & Kim, G. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01234-2_29 · Cited by 343
Textual Explanations for Self-Driving Vehicles
Kim, J., Rohrbach, A., Darrell, T., Canny, J., & Akata, Z. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01216-8_35 · Cited by 337
Multimodal end-to-end autonomous driving
Xiao, Y., Codevilla, F., Gurram, A., Urfalioglu, O., & Lopez, A. M. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.3013234 · Cited by 319
What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use
Tonekaboni, S., Joshi, S., McCradden, M. D., & Goldenberg, A. (2019) · arXiv · doi:10.48550/arxiv.1905.05134 · Cited by 318 · cited 1×
“[83] Paper surveys and evaluates explanation from clinician trust wise in the medical domain.”
BackgroundGrad-CAM: Why did you say that?
Selvaraju, R. R., Das, A., Vedantam, R., Cogswell, M., Parikh, D., & Batra, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.07450 · Cited by 309 · cited 3×
“Grad-CAM [14] captures class-specific gradient information in layers to produce a localization map of important features.”
BackgroundMethodDeep Multimodal Fusion by Channel ExchangingPreprint
Wang, Y., Huang, W., Sun, F., Xu, T., Rong, Y., & Huang, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.05005 · Cited by 292 · cited 1×
“Channel exchanging fusion that can model the inter and intra model tradeoff during fusion with a parameter-free dynamic approach and sub-network channel exchange [124] are proposed.”
BackgroundChallenges and Opportunities of Multimodality and Data Fusion in Remote Sensing
Dalla Mura, M., Prasad, S., Pacifici, F., Gamba, P., Chanussot, J., & Benediktsson, J. A. (2015) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2462751 · Cited by 286 · cited 1×
“language features is necessary to perform various multimodal integrated tasks [113].”
BackgroundChen, X., Chen, H., Xu, H., Zhang, Y., Cao, Y., Qin, Z., & Zha, H. (2019) · Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3331184.3331254 · Cited by 284
UNITER: Learning UNiversal Image-TExt Representations
Chen, Y.-C., Li, L., Yu, L., Kholy, A. E., Ahmed, F., Gan, Z., Cheng, Y., & Liu, J. (2019) · Cited by 280
A Survey of Deep Learning for Scientific DiscoveryPreprint
Raghu, M., & Schmidt, E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.11755 · Cited by 275 · cited 1×
“Artificial Intelligence systems powered by deep neural nets have achieved state-of-theart results in various domains including, Computer Vision, Natural Language Processing, and Speech recognition [129].”
BackgroundAlgorithmic recourse: From counterfactual explanations to interventions
Karimi, A.-H., Schölkopf, B., & Valera, I. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445899 · Cited by 274 · cited 1×
“Multimodal explanations based on counterfactuality provide recommendations that provide actionable insights and recourse [181].”
Background(2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46448-0 · Cited by 272
Mining opinions from instructor evaluation reviews: a deep learning approach
Onan, A. (2020) · Computer Applications in Engineering Education · doi:10.1002/cae.22179 · Cited by 270 · cited 1×
“Multiple deep learning and embedding approaches are reported for NLP tasks such as text mining [35],[36], sentiment analysis [37],[38],sarcasm identification [39],[40], information extraction [41],[42],[43].”
BackgroundExplainable Artificial Intelligence: a Systematic Review
Vilone, G., & Longo, L. (2020) · arXiv · doi:10.48550/arxiv.2006.00093 · Cited by 269 · cited 2×
“This leads to deriving comprehensive global insights about the model’s design, working principle, decision-making process, flaw detection, and handling the bias and fairness issues [50].”
BackgroundCanonical Correlation Analysis for Data Fusion and Group Inferences
Correa, N., Adali, T., Li, Y.-O., & Calhoun, V. (2010) · IEEE Signal Processing Magazine · doi:10.1109/msp.2010.936725 · Cited by 264
Goebel, R., Chander, A., Holzinger, K., Lecue, F., Akata, Z., Stumpf, S., Kieseberg, P., & Holzinger, A. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-99740-7_21 · Cited by 259 · cited 1×
“Human visual explanations can help systems know where to attend human textual explanations and how they attended image regions to complement multimodal explanations [7].”
BackgroundOnan, A. (2019) · IEEE Access · doi:10.1109/access.2019.2945911 · Cited by 257 · cited 1×
“Multiple deep learning and embedding approaches are reported for NLP tasks such as text mining [35],[36], sentiment analysis [37],[38],sarcasm identification [39],[40], information extraction [41],[42],[43].”
BackgroundExplanation Methods in Deep Learning: Users, Values, Concerns and Challenges
Ras, G., van Gerven, M., & Haselager, P. (2018) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-319-98131-4_2 · Cited by 255
Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Grounded
Selvaraju, R. R., Lee, S., Shen, Y., Jin, H., Ghosh, S., Heck, L., Batra, D., & Parikh, D. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00268 · Cited by 251 · cited 2×
“Available: https://arxiv.org/abs/1806.07421 [150] R. R. Selvaraju, S. Lee, Y. Shen, H. Jin, S. Ghosh, L. Heck, D. Batra, and D. Parikh, ‘‘Taking a HINT: Leveraging explanations to make vision and languagemodels more grounded,’’ in Proc.”
MethodInfluentialCausal interpretability for machine learning-problems, methods and evaluation
Moraffah, R., Karami, M., Guo, R., Raglin, A., & Liu, H. (2020) · ACM SIGKDD Explorations Newsletter · doi:10.1145/3400051.3400058 · Cited by 238 · cited 3×
“In [35], a survey on interpretability methods in machine learning from a causal perspective is presented.”
BackgroundInfluential(2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46454-1 · Cited by 236
CNN-based sensor fusion techniques for multimodal human activity recognition
Münzner, S., Schmidt, P., Reiss, A., Hanselmann, M., Stiefelhagen, R., & Dürichen, R. (2017) · Proceedings of the 2017 ACM International Symposium on Wearable Computers · doi:10.1145/3123021.3123046 · Cited by 235
Interpretable Machine Learning
Boehmke, B., & Greenwell, B. (2019) · Hands-On Machine Learning with R · doi:10.1201/9780367816377-16 · Cited by 230 · cited 2×
“Gaining meaningful knowledge and understanding of how and why the model arrived at a particular decision or outcome is crucial inmodel explainability, making it one of the important evaluation metrics [6].”
BackgroundExplainability of Deep Vision-Based Autonomous Driving Systems: Review and Challenges
Zablocki, É., Ben-Younes, H., Pérez, P., & Cord, M. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01657-x · Cited by 227
Explainability of deep vision-based autonomous driving systems: Review and challenges
Zablocki, É., Ben-Younes, H., Pérez, P., & Cord, M. (2021) · arXiv · doi:10.48550/arxiv.2101.05307 · Cited by 227 · cited 1×
“[76] Visual, Sensory Explainability for visionbased autonomous driving systems is detailed out.”
BackgroundOn the early diagnosis of Alzheimer's Disease from multimodal signals: A survey
Alberdi, A., Aztiria, A., & Basarab, A. (2016) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2016.06.003 · Cited by 221 · cited 1×
“[86] [87] [30] Healthcare assistive technologies”
Mishra, S., Sarkar, U., Taraphder, S., Datta, S., Swain, D., Saikhom, R., Panda, S., & Laishram, M. (2017) · International Journal of Livestock Research · doi:10.5455/ijlr.20170415115235 · Cited by 218
Explainable Artificial Intelligence (XAI)
Mohaghegh, S. D. (2024) · Artificial Intelligence for Science and Engineering Applications · doi:10.1201/9781003369356-8 · Cited by 215
Beyond Sparsity: Tree Regularization of Deep Models for Interpretability
Wu, M., Hughes, M., Parbhoo, S., Zazzi, M., Roth, V., & Doshi-Velez, F. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11501 · Cited by 213
Independent component analysis: An introduction
Tharwat, A. (2021) · Applied Computing and Informatics · doi:10.1016/j.aci.2018.08.006 · Cited by 211
Schramowski, P., Stammer, W., Teso, S., Brugger, A., Herbert, F., Shao, X., Luigs, H.-G., Mahlein, A.-K., & Kersting, K. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-0212-3 · Cited by 207 · cited 1×
“, to be right for the right reason [16].”
BackgroundA Survey of the State of Explainable AI for Natural Language Processing
Danilevsky, M., Qian, K., Aharonov, R., Katsis, Y., Kawas, B., & Sen, P. (2020) · Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing · doi:10.18653/v1/2020.aacl-main.46 · Cited by 203 · cited 1×
“Reference [37] presents an overview of the Explainable AI (XAI) approaches for theNatural Language Processing domain.”
BackgroundTowards Robust Pattern Recognition: A Review
Zhang, X.-Y., Liu, C., & Suen, C. Y. (2020) · Proceedings of the IEEE · doi:10.1109/jproc.2020.2989782 · Cited by 200 · cited 1×
“Deep learning-based multimodal data fusion results can be optimized by including the model explainability, interpretability, justification, and reasoning capabilities for better and improved performance and predictions [128].”
BackgroundDurrant-Whyte, H., & Henderson, T. C. (2016) · Springer Handbooks · doi:10.1007/978-3-319-32552-1_35 · Cited by 199
Explainable and Interpretable Models in Computer Vision and Machine Learning
(2018) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-319-98131-4 · Cited by 199
RISE: Randomized Input Sampling for Explanation of Black-box Models
Petsiuk, V., Das, A., & Saenko, K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.07421 · Cited by 196 · cited 3×
“Perturbations-based Methods: Visualize feature relevance by comparing output between inputs and altered or changed copy of the input like Occlusion sensitivity, RISE [149], LIME [143], etc. to identify sensitive features for prediction are proposed.”
MethodRUBi: Reducing Unimodal Biases in Visual Question Answering
Cadène, R., Dancette, C., Ben-younes, H., Cord, M., & Parikh, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.10169 · Cited by 194 · cited 1×
“Such biases often lead to massive drops in performance when confronted with data outside training distributions [225].”
BackgroundMFAS: Multimodal Fusion Architecture Search
Perez-Rua, J.-M., Vielzeuf, V., Pateux, S., Baccouche, M., & Jurie, F. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00713 · Cited by 193 · cited 1×
“The multimodal fusion task has also been modeled as a neural architecture search algorithm to find an appropriate search space and a suitable architecture to fuse the modalities [125].”
MethodSokol, K., & Flach, P. (2020) · KI - Künstliche Intelligenz · doi:10.1007/s13218-020-00637-y · Cited by 186 · cited 1×
“An interactive approach to explanation leverages transparency in machine learning systems [188].”
BackgroundDeep Multimodal Subspace Clustering Networks
Abavisani, M., & Patel, V. M. (2018) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2018.2875385 · Cited by 181 · cited 1×
“It is the most widely used approach [110].”
MethodManica, M., Oskooei, A., Born, J., Subramanian, V., Sáez-Rodríguez, J., & Martínez, M. R. (2019) · Molecular Pharmaceutics · doi:10.1021/acs.molpharmaceut.9b00520 · Cited by 178 · cited 1×
“[4],[184],[178], [210],[177], [181],[79],[176] ,[188],[211], [212],[19],[123] ,[213],[214], [215] Attentionbased approach Explanations based on attention to capture the important visual and textual regions that contribute to a prediction.”
BackgroundEfficient Large-Scale Multi-Modal Classification
Kiela, D., Grave, E., Joulin, A., & Mikolov, T. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11945 · Cited by 177
RAVEN: A Dataset for Relational and Analogical Visual REasoNing
Zhang, C., Gao, F., Jia, B., Zhu, Y., & Zhu, S.-C. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00546 · Cited by 175 · cited 2×
“In [164], a new dataset is proposed based on Raven’s Progressive Matrices (RPM) for the task of visual recognition reasoning, comprising images and related RPM problems, with tree-structured annotations.”
BackgroundMethodFusion of Detected Objects in Text for Visual Question Answering
Alberti, C., Ling, J., Collins, M., & Reitter, D. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1219 · Cited by 175 · cited 1×
“BERT Transformer (B2T2) [98], UnicodeText Unidirectional long short term memory(LSTM)[104], Bidirectional LSTM (BiLSTM), Unidirectional Gated Recurrent Units (GRU)[105],bidirectional GRU (BiGRU) VL [99], VLBERT[100], UNITER [101], ViLBert [102], LXMERT [103]”
BackgroundLayer-Wise Relevance Propagation for Deep Neural Network Architectures
Binder, A., Bach, S., Montavon, G., Müller, K.-R., & Samek, W. (2016) · Lecture Notes in Electrical Engineering · doi:10.1007/978-981-10-0557-2_87 · Cited by 163 · cited 3×
“Layer wise-Relevance Propagation (LRP) [153] computes backward relevance propagation to highlight the contributing features.”
BackgroundMethodDurrant-Whyte, H., & Henderson, T. C. (2008) · Springer Handbook of Robotics · doi:10.1007/978-3-540-30301-5_26 · Cited by 162
TED: Teaching AI to Explain its Decisions
Hind, M., Wei, D., Campbell, M. S., Codella, N., Dhurandhar, A., Mojsilović, A., Ramamurthy, K. N., & Varshney, K. R. (2019) · Proceedings of the AAAI/ACM Conference on AI Ethics and Society · doi:10.1145/3306618.3314273 · Cited by 160
Explainable Artificial Intelligence for Neuroscience: Behavioral Neurostimulation
Fellous, J.-M., Sapiro, G., Rossi, A., Mayberg, H., & Ferrante, M. (2019) · Frontiers in Neuroscience · doi:10.3389/fnins.2019.01346 · Cited by 155 · cited 1×
“[82] The paper focuses on studying and mapping”
BackgroundMultimodal research in vision and language: A review of current and emerging trends
Uppal, S., Bhagat, S., Hazarika, D., Majumder, N., Poria, S., Zimmermann, R., & Zadeh, A. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.07.009 · Cited by 146 · cited 1×
“Hence there is an urgent need to understand the approaches, methods, and techniques for multimodal data fusion and build an integrative framework for developing tools and applications in various disciplines [85].”
Learning Multimodal Latent Attributes
Fu, Y., Hospedales, T. M., Xiang, T., & Gong, S. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.128 · Cited by 139
Gaur, M., Faldu, K., & Sheth, A. (2021) · IEEE Internet Computing · doi:10.1109/mic.2020.3031769 · Cited by 139 · cited 1×
“knowledge base infusion in healthcare and educational sectors [195].”
BackgroundLearn to combine modalities in multimodal deep learning
Liu, K., Li, Y., Xu, N., & Natarajan, P. (2018) · arXiv · doi:10.48550/arxiv.1805.11730 · Cited by 136 · cited 1×
“as early [11], late [42], and hybrid fusion [43] schemes are developed to fuse themodalities at data, feature, decision, and intermediate mixed levels [44].”
BackgroundCan we do better explanations? A proposal of user-centered explainable AI
Ribera, M., & Lapedriza, À. (2019) · IUI Workshops · Cited by 133
An Evaluation of the Human-Interpretability of Explanation
Lage, I., Chen, E., He, J., Narayanan, M., Kim, B., Gershman, S., & Doshi-Velez, F. (2019) · arXiv · doi:10.48550/arxiv.1902.00006 · Cited by 125
VQA-E: Explaining, Elaborating, and Enhancing Your Answers for Visual Questions
Li, Q., Tao, Q., Joty, S., Cai, J., & Luo, J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01234-2_34 · Cited by 123
“Let me explain!”: exploring the potential of virtual agents in explainable AI interaction design
Weitz, K., Schiller, D., Schlagowski, R., Huber, T., & André, E. (2021) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-020-00332-0 · Cited by 119 · cited 3×
“simultaneously, and show the interplay between multimodal interactions required for building interactive explanation interfaces followed with feedback mechanism able to incorporate change in the system [89].”
BackgroundInfluentialImage Captioning: A Comprehensive Survey
Sharma, H., Agrahari, M., Singh, S. K., Firoj, M., & Mishra, R. K. (2020) · International Conference on Power Electronics & IoT Applications in Renewable Energy and its Control (PARC) · doi:10.1109/parc49193.2020.236619 · Cited by 113 · cited 1×
“Image Captioning [59] image ,text Deep learning-based methods for generating”
MethodFooling Vision and Language Models Despite Localization and Attention Mechanism
Xu, X., Chen, X., Liu, C., Rohrbach, A., Darrell, T., & Song, D. X. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00520 · Cited by 111 · cited 1×
“examples shows that models can be fooled through adversarial attacks despite attention and localization, showcasing the need to establish strong defense and mitigation mechanisms against adversarial attacks [227].”
BackgroundMemory Fusion Network for Multi-view Sequential Learning
Zadeh, A., Liang, P. P., Mazumder, N., Poria, S., Cambria, E., & Morency, L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.00927 · Cited by 111 · cited 1×
“cross-modal interactions for multi-view datasets in [121].”
Self-Critical Reasoning for Robust Visual Question Answering
Wu, J., & Mooney, R. J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.09998 · Cited by 110 · cited 1×
“In [175], authors have introduced a ‘‘self-critical training’’ method that ensures that image explanations of correct answers map more”
BackgroundMultimodal deep learning for finance: integrating and forecasting international stock markets
Lee, S. I., & Yoo, S. J. (2019) · The Journal of Supercomputing · doi:10.1007/s11227-019-03101-3 · Cited by 106
Beyond Sparsity: Tree Regularization of Deep Models for InterpretabilityPreprint
Wu, M., Hughes, M. C., Parbhoo, S., Zazzi, M., Röth, V., & Doshi‐Velez, F. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.06178 · Cited by 106 · cited 1×
“Hybrid models such as deep KNN, Contextual Explanation Network [135], Self-Explaining Neural Network [136], joint prediction and explanation, and architectural adjustments through regularization techniques [137] are examples.”
MethodVisualBackProp: Efficient Visualization of CNNs for Autonomous Driving
Bojarski, M., Choromanska, A., Choromański, K., Firner, B., Ackel, L. J., Müller, U., Yeres, P., & Zieba, K. (2018) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2018.8461053 · Cited by 105 · cited 1×
“[77] Visual, Sensory The visualBackProp method is proposed for visualizing the necessary image pixels for interpreting the predictions of the CNN.”
MethodDeep Multimodal Representation Learning from Temporal Data
Yang, X., Ramesh, P., Chitta, R., Madhvanath, S., Bernal, E. A., & Luo, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.538 · Cited by 104 · cited 1×
“Individual modalities are mapped onto a common shared representation vector space either through joint or coordinated representation [49].”
BackgroundVisual to Text: Survey of Image and Video Captioning
Li, S., Tao, Z., Li, K., & Fu, Y. (2019) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2019.2892755 · Cited by 103 · cited 1×
“image captions for control systems and IoT devices Video captioning [60] audio ,video ,text Survey and evaluation of the deep learning-based image and video captioning”
BackgroundKaya, H., Gürpınar, F., & Salah, A. A. (2017) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.210 · Cited by 94
Futia, G., & Vetrò, A. (2020) · Information · doi:10.3390/info11020122 · Cited by 94 · cited 1×
“In [196], a comprehensive view on the neuro symbolic AI perceptive is provided and integration of knowledge”
Backgrounddel Val, J. A. L., & Ágreda, J. (2004) · SAGE Publications Ltd eBooks · doi:10.4135/9780857020123.n440 · Cited by 91
Contextual Explanation NetworksPreprint
Al-Shedivat, M., Dubey, A., & Xing, E. P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.10301 · Cited by 90 · cited 1×
“Hybrid models such as deep KNN, Contextual Explanation Network [135], Self-Explaining Neural Network [136], joint prediction and explanation, and architectural adjustments through regularization techniques [137] are examples.”
MethodExplainable AI and Multi-Modal Causability in Medicine
Holzinger, A. (2021) · i-com · doi:10.1515/icom-2020-0024 · Cited by 87
(2019) · neural information processing systems · Cited by 83
Evaluation of different techniques for data fusion of LC/MS and 1H-NMR
Forshed, J., Idborg, H., & Jacobsson, S. P. (2006) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/j.chemolab.2006.05.002 · Cited by 78 · cited 1×
“Features are fused using simple concatenation [104], pooling, and gated units [105].”
MethodAlgorithmic Recourse: from Counterfactual Explanations to Interventions
Karimi, A.-H., Schölkopf, B., & Valera, I. (2020) · arXiv · doi:10.48550/arxiv.2002.06278 · Cited by 76 · cited 1×
“Multimodal explanations based on counterfactuality provide recommendations that provide actionable insights and recourse [181].”
BackgroundFaithful Multimodal Explanation for Visual Question Answering
Wu, J., & Mooney, R. (2019) · Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w19-4812 · Cited by 73 · cited 2×
“In [170], authors proposed a faithful multimodal explanation framework with a ‘‘bottom-up and top-down attention mechanism’’ to have consistent visual and textual explanations by segmenting the image for precise localization, demonstrating rationales improve human understanding and quality of explanations.”
BackgroundMethodDeep Multimodal Fusion: Combining Discrete Events and Continuous Signals
Martínez, H. P., & Yannakakis, G. N. (2014) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2663204.2663236 · Cited by 72
Multimodal Data Fusion Based on Mutual Information
Bramon, R., Boada, I., Bardera, A., Rodriguez, J., Feixas, M., Puig, J., & Sbert, M. (2011) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2011.280 · Cited by 69 · cited 1×
“variation in mutual information of different channels [119].”
BackgroundGhosh, S., Burachas, G., Ray, A., & Ziskind, A. J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.05715 · Cited by 68
Bias in Multimodal AI: Testbed for Fair Automatic Recruitment
Pena, A., Serna, I., Morales, A., & Fierrez, J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00022 · Cited by 64 · cited 1×
“In [234], how biases affect automated recruitment systems is demonstrated.”
BackgroundOnan, A., & Toçoğlu, M. A. (2020) · Computer Applications in Engineering Education · doi:10.1002/cae.22252 · Cited by 63 · cited 1×
“Multiple deep learning and embedding approaches are reported for NLP tasks such as text mining [35],[36], sentiment analysis [37],[38],sarcasm identification [39],[40], information extraction [41],[42],[43].”
BackgroundRemoving Bias in Multi-modal Classifiers: Regularization by Maximizing Functional Entropies
Gat, I., Schwartz, I., Schwing, A., & Hazan, T. (2020) · arXiv · doi:10.48550/arxiv.2010.10802 · Cited by 63 · cited 1×
“To detect and mitigate the bias in [223], proposed a regularization approach based onmaximizing functional entropy and balancing modality contributions.”
BackgroundHase, P., Zhang, S., Xie, H., & Bansal, M. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.390 · Cited by 62
Information Fusion in Multimedia Information Retrieval
Kludas, J., Bruno, É., & Marchand‐Maillet, S. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-79860-6_12 · Cited by 61
Hoste, L., Dumas, B., & Signer, B. (2011) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2070481.2070500 · Cited by 60
Explaining Autonomous Driving by Learning End-to-End Visual Attention
Cultrera, L., Seidenari, L., Becattini, F., Pala, P., & Del Bimbo, A. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00178 · Cited by 60
Make Up Your Mind! Adversarial Generation of Inconsistent Natural Language Explanations
Camburu, O.-M., Shillingford, B., Minervini, P., Lukasiewicz, T., & Blunsom, P. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.382 · Cited by 59
Do explanations make VQA models more predictable to a human?
Chandrasekaran, A., Prabhu, V., Yadav, D., Chattopadhyay, P., & Parikh, D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1128 · Cited by 55 · cited 2×
“Attention maps do not guarantee explanatory power [15] as the models and human perception does not concentrate on the same areas while providing the output.”
BackgroundSQuINTing at VQA Models: Introspecting VQA Models With Sub-Questions
Selvaraju, R. R., Tendulkar, P., Parikh, D., Horvitz, E., Ribeiro, M. T., Nushi, B., & Kamar, E. S. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01002 · Cited by 55 · cited 1×
“[172] discusses the state-of-the-art VQA models perform well on perception and reasoning questions but provide inconsistent explanations modeling a task.”
Background
Research works printed but not linked: 7
- vidya sagar appaji, 2020, Maximizing joint probability in visual question answering models, Int J Adv Sci Technol, 29, 3914
- lin, 2019, TAB-VCR: Tags and attributes based VCR baselines, Proc NeurIPS
- tharwat, 2020, Appl Comput Inform
- vedantam, 2015, METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments, Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit, 4566
- sharma, 2018, Attend and attack?: Attention guided adversarial attacks on visual question answering models, Proc 32nd Conf Neural Inf Process Syst (NeurIPS), 1
- biran, 2017, Explanation and justification in machine learning: A survey, Proc Workshop Explainable AI (XAI), 8, 8
- alvarez-melis, 2018, Self-Explaining Neural Networks
Cited by
Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities
Saeed, W., & Omlin, C. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110273 · Cited by 783
Nauta, M., Trienes, J., Pathak, S., Nguyen, E., Peters, M., Schmitt, Y., Schlötterer, J., van Keulen, M., & Seifert, C. (2023) · ACM Computing Surveys · doi:10.1145/3583558 · Cited by 590
“Some surveys discuss evaluation as part of a broader review of XAI methods [1, 29, 32, 59, 81, 120, 170, 173, 175, 182, 303], or mainly discuss evaluation with user studies [52, 100, 136, 170].”
BackgroundA., S., & R., S. (2023) · Decision Analytics Journal · doi:10.1016/j.dajour.2023.100230 · Cited by 432
ŞAHiN, E., Arslan, N. N., & Özdemir, D. (2025) · Neural Computing and Applications · doi:10.1007/s00521-024-10437-2 · Cited by 278
Nazir, S., Dickson, D. M., & Akram, M. U. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.106668 · Cited by 283
“The black box and opaque nature of deep neural networks precludes their usefulness despite excellent performance of multimodal methods [288].”
Background
Metadata
- Authors
- Joshi, G., Walambe, R., & Kotecha, K.
- Affiliations
- Symbiosis International University (IN)
- Venue
- IEEE Access
- Published
- 2021
- Volume
- 9
- Pages
- 59800-59821
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W3148438775
- Semantic Scholar
- 6bda005a727969356a012e177cd600e7fdf80c96
- Cited by
- 204
- References
- 261
- Citations by year
- 2026: 26 · 2025: 49 · 2024: 45 · 2023: 50 · 2022: 25 · 2021: 9
- FWCI
- 11.40
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University) (cc-by)
- Funders
- Ministry of Human Resource Development (MHRD) Scheme for Promotion of Academic and Research Collaboration (SPARC) of the Government of India (P104)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- explainability, multimodal deep neural networks, multimodal fusion, model interpretability, vision-language tasks, black-box models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.