Multi-modal Knowledge-aware Hierarchical Attention Network for Explainable Medical Question Answering
Abstract
Online healthcare services can offer public ubiquitous access to the medical knowledge, especially with the emergence of medical question answering websites, where patients can get in touch with doctors without going to hospital. Explainability and accuracy are two main concerns for medical question answering. However, existing methods mainly focus on accuracy and cannot provide a good explanation for retrieved medical answers. This paper proposes a novelMulti-Modal Knowledge-aware Hierarchical Attention Network (MKHAN) to effectively exploit multi-modal knowledge graph (MKG) for explainable medical question answering. MKHAN can generate path representation by composing the structural, linguistics, and visual information of entities, and infer the underlying rationale of question-answer interactions by leveraging the sequential dependencies within a path from MKG. Furthermore, a novel hierarchical attention network is proposed to discriminate the salience of paths endowing our model with explainability. We build a large-scale multi-modal medical knowledge graph andtwo real-world medical question answering datasets, the experimental results demonstrate the superior performance on our approachcompared with the state-of-the-art methods.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In-KG applications include graph refinement (e.g., completion and error detection), triple/entity classification, and entity resolution. Out-of-KG applications include relation extraction, question answering, and recommendation, which are related to specific AIoT scenarios, such as network forensics analysis in smart security [224], smart assistants in smart healthcare and agriculture [39], [225],
“InKG applications include graph refinement (e.g., completion and error detection), triple/entity classification, and entity resolution. Out-of-KG applications include relation extraction, question answering, and recommendation, which are related to specific AIoT scenarios, such as network forensics analysis in smart security [224], smart assistants in smart healthcare and agriculture [39], [225],”
From Empowering things with intelligence: a survey of the progress, challenges, and opportunities in artificial intelligence of things (2021) · cites this paper as [39]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Shickel, B., Tighe, P. J., Bihorac, A., & Rashidi, P. (2018) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2017.2767063 · Cited by 1,688 · see citing works
IR evaluation methods for retrieving highly relevant documents
Järvelin, K., & Kekäläinen, J. (2017) · ACM SIGIR Forum · doi:10.1145/3130348.3130374 · Cited by 1,457 · see citing works
Proceedings of the 26th International Conference on Neural Information Processing Systems
Burges, C. J. C., Bottou, L., Welling, M., Zoubin Ghahramani, A., & Weinberger, K. Q. (2013) · Cited by 1,323 · see citing works
Hierarchical Feature Representation and Multimodal Fusion with Deep Learning for AD/MCI Diagnosis
Suk, H.-I., Lee, S.-W., & Shen, D. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2014.06.077 · Cited by 968 · see citing works
Proceedings of the 26th AAAI Conference on Artificial Intelligence (AAAI'12)
Hoffmann, J., & Selman, B. (2012) · HAL (Le Centre pour la Communication Scientifique Directe) · Cited by 718 · see citing works
Multimodal Neuroimaging Feature Learning for Multiclass Diagnosis of Alzheimer's Disease
Liu, S., Liu, S., Cai, W., Che, H., Pujol, S., Kikinis, R., Feng, D., Fulham, M. J., & ADNI (2015) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2014.2372011 · Cited by 637 · see citing works
GRAM: Graph-based Attention Model for Healthcare Representation Learning
Choi, E., Bahadori, M. T., Song, L., Stewart, W. F., & Sun, J. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098126 · Cited by 637 · see citing works
End-to-End Neural Ad-hoc Ranking with Kernel Pooling
Xiong, C., Dai, Z., Callan, J., Liu, Z., & Power, R. (2017) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3077136.3080809 · Cited by 554 · see citing works
Multi-layer Representation Learning for Medical Concepts
Choi, E., Bahadori, M. T., Searles, E., Coffey, C., Thompson, M., Bost, J., Tejedor-Sojo, J., & Sun, J. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939823 · Cited by 469 · see citing works
$\mathtt {Deepr}$: A Convolutional Net for Medical Records
Nguyen, P., Tran, T., Wickramasinghe, N. S., & Venkatesh, S. (2016) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2016.2633963 · Cited by 413 · see citing works
Recurrent knowledge graph embedding for effective recommendation
Sun, Z., Yang, J., Zhang, J., Bozzon, A., Huang, L.-K., & Xu, C. (2018) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3240323.3240361 · Cited by 382 · see citing works
Self-citationImage-embodied Knowledge Representation Learning
Xie, R., Liu, Z., Luan, H., & Sun, M. (2017) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2017/438 · Cited by 288 · see citing works
DeepCare: A Deep Dynamic Memory Model for Predictive Medicine
Pham, T., Tran, T., Phung, D., & Venkatesh, S. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-31750-2_3 · Cited by 269 · see citing works
Cui, W., Xiao, Y., Wang, H., Song, Y., Hwang, S.-W., & Wang, W. (2017) · Proceedings of the VLDB Endowment · doi:10.14778/3055540.3055549 · Cited by 244 · see citing works
MEANS: A medical question-answering system combining NLP techniques and semantic Web technologies
Abacha, A. B., & Zweigenbaum, P. (2015) · Information Processing & Management · doi:10.1016/j.ipm.2015.04.006 · Cited by 232 · see citing works
KAME: Knowledge-based Attention Model for Diagnosis Prediction in Healthcare
Ma, F., You, Q., Xiao, H., Chitta, R., Zhou, J., & Gao, J. (2018) · Proceedings of the 27th ACM International Conference on Information and Knowledge Management · doi:10.1145/3269206.3271701 · Cited by 213 · see citing works
Disease inference from health-related questions via sparse deep learning
Nie, L., Wang, M., Zhang, L., Yan, S., Zhang, B., & Chua, T.-S. (2015) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2015.2399298 · Cited by 195 · see citing works
Tran, T., Nguyen, T. D., Phung, D., & Venkatesh, S. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2015.01.012 · Cited by 172 · see citing works
Multi-Modal Event Topic Model for Social Event Analysis
Qian, S., Zhang, T., Xu, C., & Shao, J. (2015) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2015.2510329 · Cited by 150 · see citing works
Self-citationLearning to Rank Question Answer Pairs with Holographic Dual LSTM Architecture
Tay, Y., Phan, M. C., Tuan, L. A., & Hui, S. C. (2017) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3077136.3080790 · Cited by 120 · see citing works
A General Framework for Diagnosis Prediction via Incorporating Medical Code Descriptions
Ma, F., Wang, Y., Xiao, H., Yuan, Y., Chitta, R., Zhou, J. L., & Gao, J. (2018) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm.2018.8621395 · Cited by 31 · see citing works
Reliable Medical Diagnosis from Crowdsourcing
Li, Y., Du, N., Liu, C., Xie, Y., Fan, W., Li, Q., Gao, J., & Sun, H. (2017) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3018661.3018688 · Cited by 25 · see citing works
Knowledge Representations and Inference Techniques for Medical Question Answering
Goodwin, T. R., & Harabagiu, S. M. (2017) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3106745 · Cited by 25 · see citing works
Multi-Modal Knowledge Representation Learning via Webly-Supervised Relationships Mining
Nian, F., Bao, B., Li, T., & Xu, C. (2017) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3123266.3123443 · Cited by 17 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Zhang, J., & Tao, D. (2021) · IEEE Internet of Things Journal · doi:10.1109/jiot.2020.3039359 · Cited by 774 · see citing works
Biomedical Question Answering: A Survey of Approaches and Challenges
Jin, Q., Yuan, Z. P., Xiong, G., Yu, Q., Ying, H., Tan, C., Chen, M., Huang, S., Liu, X., & Yu, S. (2022) · ACM Computing Surveys · doi:10.1145/3490238 · Cited by 128 · see citing works
Knowledge-aware Multi-modal Adaptive Graph Convolutional Networks for Fake News Detection
Qian, S., Hu, J., Fang, Q., & Xu, C. (2021) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3451215 · Cited by 100 · see citing works
Self-citationA Survey of Multi-modal Knowledge Graphs: Technologies and Trends
Liang, W., Meo, P. D., Tang, Y. Z., & Zhu, J. (2024) · ACM Computing Surveys · doi:10.1145/3656579 · Cited by 99 · see citing works
Multimodal Disentangled Domain Adaption for Social Media Event Rumor Detection
Zhang, H., Qian, S., Fang, Q., & Xu, C. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3042055 · Cited by 69 · see citing works
Self-citationHyper-node Relational Graph Attention Network for Multi-modal Knowledge Graph Completion
Liang, S., Zhu, A., Zhang, J., & Shao, J. (2022) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3545573 · Cited by 66 · see citing works
Computational Technologies for Fashion Recommendation: A Survey
Ding, Y., Lai, Z., Mok, P. Y., & Chua, T. (2023) · ACM Computing Surveys · doi:10.1145/3627100 · Cited by 49 · see citing works
A Deep Multi-Modal Explanation Model for Zero-Shot Learning
Liu, Y., & Tuytelaars, T. (2020) · IEEE Transactions on Image Processing · doi:10.1109/tip.2020.2975980 · Cited by 31 · see citing works
External features enriched model for biomedical question answering
Xu, G., Rong, W., Wang, Y., Ouyang, Y., & Xiong, Z. (2021) · BMC Bioinformatics · doi:10.1186/s12859-021-04176-7 · Cited by 30 · see citing works
Shen, J., Pan, T., Xu, M., Gan, D., & An, B. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103322 · Cited by 25 · see citing works
Knowledge Graph Embedding Based on Graph Neural Network
Liang, S. (2023) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde55515.2023.00379 · Cited by 24 · see citing works
Zhang, Y., Fang, Q., Qian, S., & Xu, C. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413736 · Cited by 21 · see citing works
Self-citationMulti-modal Attentive Graph Pooling Model for Community Question Answer Matching
Hu, J., Fang, Q., Qian, S., & Xu, C. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413711 · Cited by 14 · see citing works
Self-citationKou, Z., Zhang, Y., Zhang, D., & Wang, D. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3555178 · Cited by 12 · see citing works
Self-citationConversational User Interfaces and Agents
Biancofiore, G. M., Palma, D. D., Pomo, C., Narducci, F., & Noia, T. D. (2025) · Human-computer interaction series · doi:10.1007/978-3-031-61375-3_4 · Cited by 12 · see citing works
Hu, J., Qian, S., Fang, Q., & Xu, C. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2020.3009491 · Cited by 11 · see citing works
Self-citationConversational Text Extraction with Large Language Models Using Retrieval-Augmented Systems
Roy, S., Goswami, M., Nargund, N., Mohanty, S., & Pattnaik, P. K. (2024) · International Conference on Computational Intelligence and Networks (CINE) · doi:10.1109/cine63708.2024.10881808 · Cited by 10 · see citing works
Adversarial Knowledge Distillation Based Biomedical Factoid Question Answering
Bai, J., Yin, C., Zhang, J., Wang, Y., Dong, Y., Rong, W., & Xiong, Z. (2022) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2022.3161032 · Cited by 12 · see citing works
Multimodal Knowledge Graph Embedding With Self-Attention Graph Neural Network
Zhang, W., Liu, T., Li, Z., Dai, Y., & Zhou, X. (2025) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2025.3556073 · Cited by 9 · see citing works
How Context or Knowledge Can Benefit Healthcare Question Answering
Wang, X., Luo, F., Wu, Q., & Bao, Z. (2021) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3090253 · Cited by 9 · see citing works
Biomedical Question Answering: A Comprehensive Review.Preprint
Jin, Q., Yuan, Z. P., Xiong, G., Yu, Q., Tan, C., Chen, M., Huang, S., Liu, X., & Yu, S. (2021) · arXiv (Cornell University) · Cited by 9 · see citing works
MAGIC: Rethinking Dynamic Convolution Design for Medical Image Segmentation
Li, S., Tu, Y., Xiang, Q., & Li, Z. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3680754 · Cited by 8 · see citing works
Mitigating Modality Bias in Multi-modal Entity Alignment from a Causal Perspective
Su, T., Sheng, J., Ma, D., Li, X., Yue, J., Song, M., Tang, Y., & Liu, T. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730037 · Cited by 7 · see citing works
Knowledge-aware Attentive Wasserstein Adversarial Dialogue Response Generation
Zhang, Y., Fang, Q., Qian, S., & Xu, C. (2020) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3384675 · Cited by 6 · see citing works
Self-citationPan, T., Shen, J., & Xu, M. (2025) · Frontiers in Neuroscience · doi:10.3389/fnins.2025.1606038 · Cited by 5 · see citing works
Sun, W., Li, M., Sileo, D., Davis, J. J., & Moens, M. (2025) · ACM Transactions on Computing for Healthcare · doi:10.1145/3712296 · Cited by 6 · see citing works
Learning Sensor Interdependencies for IMU-to-Segment Assignment
Kaichi, T., Maruyama, T., Tada, M., & Saitô, H. (2021) · IEEE Access · doi:10.1109/access.2021.3105801 · Cited by 2 · see citing works
Boosting Generic Visual-Linguistic Representation With Dynamic Contexts
Ma, G., Bai, Y., Zhang, W., Yao, T., Shihada, B., & Mei, T. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3237164 · Cited by 2 · see citing works
Systematic Literature Review of the Development of Data Science Applications in Healthcare
Rangaraj, A. Y., Langdon, T. P., & Said, H. (2025) · Issues in Informing Science and Information Technology · doi:10.28945/5503 · Cited by 1 · see citing works
MLFormer: Unleashing Efficiency Without Attention for Multimodal Knowledge Graph Embedding
Wang, M., Li, C., Chen, F., Shao, J., Qin, K., & Liang, S. (2025) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2025.3620089 · Cited by 1 · see citing works
Powering COVID-19 community Q&A with Curated Side InformationPreprint
Verma, M., Thadani, K., & Mishra, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2101.11556 · Cited by 1 · see citing works
Future Directions of Question Answering
Momtazi, S., & Abbasiantaeb, Z. (2022) · Springer eBooks · doi:10.1007/978-3-031-16552-8_9
Zhu, N., Huang, J., Cao, J., Hu, L., & Zhu, S. (2024) · Complex & Intelligent Systems · doi:10.1007/s40747-024-01629-3
- Authors
- Zhang, Y., Qian, S., Fang, Q., & Xu, C.
- Affiliations
- Institute of Automation (CN)
- Chinese Academy of Sciences (CN)
- Venue
- Proceedings of the 27th ACM International Conference on Multimedia
- Publisher
- ACM
- Published
- 2019-10-15
- Pages
- 1089-1097
- OpenAlex
- W2982244969
- Cited by
- 33
- References
- 25
- Citations by year
- 2025: 7 · 2024: 4 · 2023: 4 · 2022: 5 · 2021: 6 · 2020: 7
- FWCI
- 1.47
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM International Conference on Multimedia (ACM MM) (publishedVersion)
- Funders
- the National Key Research and Development Program of China (No. 2017YFB1002804)
- the K.C.Wong Education Foundation
- the Beijing Municipal Science $&$ Technology Commission (No. Z181100008918012)
- the National Natural Science Foundation of China under Grants (614320196172010600661572503618024056187242461702509, 61832002)
- the Key Research Program of Frontier Sciences CAS Grant (NO. QYZDJ-SSW-JSC039)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Keywords
- explainable question answering, medical question answering, multimodal knowledge graphs, hierarchical attention network, path representation, knowledge graph reasoning, rationale generation, online healthcare services
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
