KG-Rank: Enhancing Large Language Models for Medical QA with Knowledge Graphs and Ranking Techniques
Open accessAbstract
Large language models (LLMs) have demonstrated impressive generative capabilities with the potential to innovate in medicine. However, the application of LLMs in real clinical settings remains challenging due to the lack of factual consistency in the generated content. In this work, we develop an augmented LLM framework, KG-Rank, which leverages a medical knowledge graph (KG) along with ranking and re-ranking techniques, to improve the factuality of long-form question answering (QA) in the medical domain. Specifically, when receiving a question, KG-Rank automatically identifies medical entities within the question and retrieves the related triples from the medical KG to gather factual information. Subsequently, KG-Rank innovatively applies multiple ranking techniques to refine the ordering of these triples, providing more relevant and precise information for LLM inference. To the best of our knowledge, KG-Rank is the first application of KG combined with ranking models in medical QA specifically for generating long answers. Evaluation on four selected medical QA datasets demonstrates that KG-Rank achieves an improvement of over 18% in ROUGE-L score. Additionally, we extend KG-Rank to open domains, including law, business, music, and history, where it realizes a 14% improvement in ROUGE-L score, indicating the effectiveness and great potential of KG-Rank.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs have advanced text generation capability and extensive domain-specific knowledge.
“LLMs have advanced text generation capability and extensive domain-specific knowledge [8,9].”
From Mitigating cognitive biases in clinical decision-making through multi-agent conversations using large language models: simulation study (2024) · cites this paper as [9]
The Introduction says LLMs have powerful generative capabilities, supporting the text-generation part of the claim. It does not establish that they have extensive domain-specific knowledge; instead, it notes limited training on medical data and describes supplementing LLMs with external medical knowledge. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865 · see citing works
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · see citing works
BLEURT: Learning Robust Metrics for Text Generation
Sellam, T., Das, D., & Parikh, A. P. (2020) · arXiv · doi:10.48550/arxiv.2004.04696 · Cited by 1,005
MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover DistancePreprint
Zhao, W., Peyrard, M., Liu, F., Gao, Y., Meyer, C. M., & Eger, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.02622 · Cited by 71 · see citing works
Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat DataPreprint
Xu, C., Guo, D., Duan, N., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.01196 · Cited by 7 · see citing works
Retrieval-Augmented Generation for Generative Artificial Intelligence in MedicinePreprint
Yang, R., Ning, Y., Keppo, E., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2406.12449 · Cited by 6 · see citing works
Self-citationLeveraging Large Language Models for Concept Graph Recovery and Question Answering in NLP EducationPreprint
Yang, R., Yang, B., Ouyang, S., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.14293 · Cited by 3 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 11
- Asma Ben Abacha, Eugene Agichtein, Yuval Pinter, and Dina Demner-Fushman. 2017. Overview of the medical question answering task at trec 2017 liveqa. In TREC, pages 1-12.
- Asma Ben Abacha, Yassine Mrabet, Mark Sharp, Travis R Goodwin, Sonya E Shooshan, and Dina Demner-Fushman. 2019. Bridging the gap between consumers' medication questions and trusted an- swers. In MedInfo, pages 25-29.
- Jaime G. Carbonell and Jade Goldstein-Stewart. 1998. The use of mmr, diversity-based reranking for re- ordering documents and producing summaries. In Annual International ACM SIGIR Conference on Re- search and Development in Information Retrieval.
- Aosong Feng, Irene Li, Yuang Jiang, and Rex Ying. 2023. Diffuser: efficient transformers with multi-hop attention diffusion for long sequences. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 37, pages 12772-12780.
- Fan Gao, Hang Jiang, Moritz Blum, Jinghui Lu, Yuang Jiang, and Irene Li. 2023. Large language models on wikipedia-style survey generation: an evaluation in nlp concepts. arXiv preprint arXiv:2308.10410. · arXiv
- Irene Li, Jessica Pan, Jeremy Goldwasser, Neha Verma, Wai Pan Wong, Muhammed Yavuz Nuzumlalı, Ben- jamin Rosand, Yixin Li, Matthew Zhang, David Chang, et al. 2022a. Neural natural language process- ing for unstructured data in electronic health records: A review. Computer Science Review, 46:100511.
- Irene Li, Linfeng Song, Kun Xu, and Dong Yu. 2022b. Variational graph autoencoding as cheap supervision for amr coreference resolution. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 2790-2800.
- Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
- Cai Long, Deepak Subburam, Kayle Lowe, André dos Santos, Jessica Zhang, Sang Hwang, Neil Saduka, Yoav Horev, Tao Su, David Cote, et al. 2023. Cha- tent: Augmented large language model for expert knowledge retrieval in otolaryngology-head and neck surgery. medRxiv, pages 2023-08.
- Boming Yang, Dairui Liu, Toyotaro Suzumura, Rui- hai Dong, and Irene Li. 2023a. Going beyond local: Global graph-enhanced personalized news recom- mendations. In Proceedings of the 17th ACM Confer- ence on Recommender Systems, pages 24-34.
- Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36.
Cited by
Sorted by citations
Ke, Y., Yang, R., Lie, S. A., Lim, T. X. Y., Ning, Y., Li, I., Abdullah, H. R., Ting, D. S. W., & Liu, N. (2024) · Journal of Medical Internet Research · doi:10.2196/59439 · Cited by 64 · see citing works
“LLMs have advanced text generation capability and extensive domain-specific knowledge [8,9].”
Self-citationYang, R., Zeng, Q., You, K., Qiao, Y., Huang, L., Hsieh, C.-C., Rosand, B., Goldwasser, J., Dave, A., Keenan, T., et al. (2024) · Journal of Medical Internet Research · doi:10.2196/60601 · Cited by 16 · see citing works
Self-citationGraph-constrained Reasoning: Faithful Reasoning on Knowledge Graphs with Large Language Models
Luo, L., Zhao, Z., Haffari, G., Li, Y.-F., Gong, C., & Pan, S. (2024) · arXiv · doi:10.48550/arxiv.2410.13080 · Cited by 8 · see citing works
- Authors
- Yang, R., Liu, H., Marrese-Taylor, E., Zeng, Q., Ke, Y. H., Li, W., Cheng, L., Chen, Q., Caverlee, J., Matsuo, Y., & Li, I.
- Affiliations
- Duke-NUS Medical School (SG)
- Texas A&M University (US)
- The University of Tokyo (JP)
- Northwestern University (US)
- Zhejiang University (CN)
- Zhejiang Lab (CN)
- National Institutes of Health (US)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2403.05881
- OpenAlex
- W4392736985
- Cited by
- 5
- References
- 7
- Citations by year
- 2025: 2 · 2024: 3
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- large language models, medical question answering, knowledge graphs, answer ranking, factual consistency, entity identification, open-domain question answering, factual grounding
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
