Biomedical knowledge graph-optimized prompt generation for large language models
Open accessThe DOI matches its Crossref record; 56 of the 56 references it cites were found in a registry or an index; 8 of the 10 sources its claims cite were checked for support.
Abstract
MOTIVATION: Large language models (LLMs) are being adopted at an unprecedented rate, yet still face challenges in knowledge-intensive domains such as biomedicine. Solutions such as pretraining and domain-specific fine-tuning add substantial computational overhead, requiring further domain-expertise. Here, we introduce a token-optimized and robust Knowledge Graph-based Retrieval Augmented Generation (KG-RAG) framework by leveraging a massive biomedical KG (SPOKE) with LLMs such as Llama-2-13b, GPT-3.5-Turbo, and GPT-4, to generate meaningful biomedical text rooted in established knowledge. RESULTS: Compared to the existing RAG technique for Knowledge Graphs, the proposed method utilizes minimal graph schema for context extraction and uses embedding methods for context pruning. This optimization in context extraction results in more than 50% reduction in token consumption without compromising the accuracy, making a cost-effective and robust RAG implementation on proprietary LLMs. KG-RAG consistently enhanced the performance of LLMs across diverse biomedical prompts by generating responses rooted in established knowledge, accompanied by accurate provenance and statistical evidence (if available) to substantiate the claims.
Overview
CoChat Summary
## TL;DR
This paper introduces KG-RAG, a token-optimized retrieval-augmented generation framework that uses the SPOKE biomedical knowledge graph to ground LLM outputs in established biomedical knowledge. Across benchmark tasks, it reduced token use by more than 50% versus prior KG-RAG approaches and improved Llama-2 performance on a challenging MCQ dataset by 71%.
## Key findings
- On a 100-question retrieval benchmark, KG-RAG achieved 97% retrieval accuracy, compared with 75% for Cypher-RAG and 61% for full-text index retrieval.
- Under prompt perturbation (lowercasing entity names), Cypher-RAG retrieval accuracy dropped to 0%, while KG-RAG maintained 97% and full-text index dropped slightly to 58% from 61%.
- KG-RAG used an average of 3,693 tokens per response, versus 8,006 for Cypher-RAG and 10,590 for full-text index, corresponding to 53.9% and 65.1% token reductions, respectively.
- On the MCQ dataset, KG-RAG increased Llama-2 accuracy from 0.31 ± 0.03 to 0.53 ± 0.03, described as approximately a 71% boost; on the same KG-RAG setup, GPT-3.5-Turbo outperformed GPT-4 on MCQ (0.79 ± 0.02 vs 0.74 ± 0.03, P < 0.0001).
## Methods
- Framework design: disease entity extraction from user prompts, disease-node matching in SPOKE, neighborhood/context retrieval from the graph, context pruning with sentence-transformer embeddings and cosine-similarity thresholds, then prompt assembly and LLM generation.
- Data/sample: SPOKE, a biomedical knowledge graph with 42 million nodes, 160 million edges, and 41 integrated biomedical databases; benchmark datasets included 311 True/False questions, 306 MCQs, and 100 retrieval-comparison questions, plus 165 validation questions for hyperparameter tuning.
- Analysis: compared KG-RAG with prompt-only, Cypher-RAG, and full-text index approaches using accuracy, Jaccard similarity, robustness to prompt perturbation, and token usage; LLMs tested were Llama-2-13b, GPT-3.5-Turbo, and GPT-4 with temperature set to 0.
## Limitations
- The paper states the approach is currently limited to biomedical questions centered on diseases, though it includes a proof of concept extending to more than 87,000 non-disease nodes; this is an explicit limitation.
- The paper says it was developed on SPOKE only and future work will extend it to other biomedical KGs and domains; this is an explicit limitation.
- The paper notes that context quality depends on the underlying graph and that SPOKE is “not entirely error free” or ready for clinical use; this is an explicit limitation.
- Evident from the text: the evaluation focuses on curated benchmark questions and retrieval tasks rather than a formal clinical validation of the KG-RAG system in real-world biomedical use.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
4 Valid · 2 Not valid · 0 Not checked · 50 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Thus, while SPOKE’s reliability has been demonstrated through its successful application in various biomedical contexts (Supplementary S1 Text; Himmelstein and Baranzini, 2015, Himmelstein et al. 2017, Nelson et al. 2019, Nelson et al. 2021a,b, Baranzini et al. 2022, Morris et al. 2023, Soman et al. 2023a,b, Tang et al. 2024)”
§ Biomedical knowledge graph-optimized prompt generation for large language models > 3 Results > 3.4 Performance on True/False and MCQ datasets
What the sources say
- Partly supportedcites Himmelstein and Baranzini, 2015 sourceExperimental — accuracy not yet measured
The Discussion reports that this paper’s heterogeneous-network predictions successfully prioritized disease–gene associations across 29 diseases and helped identify three multiple-sclerosis susceptibility genes. That supports a successful biomedical application of a related network approach, but our copy does not identify the network as SPOKE or demonstrate SPOKE’s reliability across various contexts. (LLM, conf=0.83)
- Could not verifycites Himmelstein et al. 2017 sourceExperimental — accuracy not yet measured
The Introduction and Results > Hetionet v1.0 describe Hetionet and its use in drug repurposing and prototype biomedical queries, not SPOKE. Searches for SPOKE and its reliability found no passage in our copy addressing whether SPOKE was successfully applied across biomedical contexts; page numbers were unavailable. (LLM, conf=0.76)
- Partly supportedcites Morris et al. 2023 sourceExperimental — accuracy not yet measured
The Discussion reports that SPOKE has been used in several biomedical applications and that one approach predicted multiple sclerosis diagnoses with up to 83% accuracy. It does not establish SPOKE’s general reliability across those varied contexts; the documented success is specific to that prediction application. (LLM, conf=0.80)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Nevertheless, the “implicit representation” of knowledge in LLM has also been shown to generate non-factual information despite linguistically coherent answers (i.e. “hallucination”) as a response to the input prompt (Maynez et al. 2020, Raunak et al. 2021, Ji et al. 2023).”
§ Biomedical knowledge graph-optimized prompt generation for large language models > 1 Introduction
What the sources say
- Partly supportedcites Maynez et al. 2020 sourceExperimental — accuracy not yet measured
Section 5.3, “Assessment of Factual Hallucinations” (page unavailable), reports that over 90% of BERTS2S hallucinations were erroneous, supporting the point that generated text can be non-factual. But the study concerns abstractive summaries of documents, not LLM answers to prompts, and it does not establish that an implicit representation of knowledge causes the errors; it says pretraining helps generate more factual summaries. (LLM, conf=0.86)
- Partly supportedcites Raunak et al. 2021 sourceExperimental — accuracy not yet measured
The Introduction describes NMT hallucinations as outputs decoupled from the source sequence despite being fluent in the target language, and the Conclusion reports experiments on when such hallucinations arise. This supports the claim’s general point about fluent but unsupported output. The paper studies neural machine translation, however, not LLM answers to prompts or an ‘implicit representation’ of knowledge. (LLM, conf=0.88)
- Not matchedcites Ji et al. 2023
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“To achieve this, we make use of the biomedical KG called SPOKE (Morris et al. 2023) that integrates more than 40 publicly available biomedical knowledge sources of separate domains where each source is centered around a biomedical concept, such as genes, proteins, drugs, compounds, diseases, and one or more of their known relationships.”
§ Biomedical knowledge graph-optimized prompt generation for large language models > 1 Introduction
What the sources say
- Supportedcites Morris et al. 2023 source
The Introduction describes SPOKE as connecting 41 specialized databases across biomedical fields, using distinct node and relationship types. Table 1 in Section 2.1 lists sources for concepts including genes, proteins, compounds and diseases; the Results describe the relationships connecting them. Page numbers are not provided in our copy. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, we demonstrated that this method can be sensitive to how the prompt is formulated. Even slight perturbations to the prompt can lead to incorrect Cypher queries and subsequently impact downstream generative processes.”
§ Biomedical knowledge graph-optimized prompt generation for large language models > 2 Materials and methods > 2.8 Cypher-RAG
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Furthermore, KG-RAG enhanced the performance of proprietary GPT models, such as GPT-3.5 and GPT-4.”
§ Biomedical knowledge graph-optimized prompt generation for large language models > Abstract > Results
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: unclear — It is unclear why the paper says it holds.
25 more claims
Create a free account to see all 56 claims and every source verdict.
- Probably not a substantial claim. 7
- It may state two separate claims as one. 21
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Empirical evidence supporting the utility of LLMs in neurodegenerative disease characterization continues to emerge.”
From Review of Artificial Intelligence for Clinical Use in Alzheimer's Disease and Related Dementias (2026) · cites this paper as [21]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“provided neuropathologic differential diagnoses with high correlation to experts using human-curated clinical summaries.21”
From Review of Artificial Intelligence for Clinical Use in Alzheimer's Disease and Related Dementias (2026) · cites this paper as [21]
Experimental — accuracy not yet measured
Citations
Cites
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · cited 1×
“Nevertheless, the “implicit representation” of knowledge in LLM has also been shown to generate non-factual information despite linguistically coherent answers (i.e. “hallucination”) as a response to the input prompt (Maynez et al. 2020, Raunak et al. 2021, Ji et al. 2023).”
BackgroundChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · cited 1×
“This approach has led to the creation of models such as PubMedBERT (Lee et al. 2020, Gu et al. 2021) BlueBERT (Peng et al. 2019), SciBERT (Beltagy et al. 2019), ClinicalBERT (Huang et al. 2019), BioGPT (Luo et al. 2022), Med-PaLM (Singhal et al. 2022), and BioMedGPT (Luo et al. 2023).”
MethodRetrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101
Piñero, J., Bravo, À., Queralt-Rosinach, N., Gutiérrez-Sacristán, A., Deu-Pons, J., Centeno, E., García-García, J., Sanz, F., & Furlong, L. I. (2017) · Nucleic Acids Research · doi:10.1093/nar/gkw943 · Cited by 2,680
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604
Domain-specific language model pretraining for biomedical natural language processing
Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., & Poon, H. (2022) · ACM Transactions on Computing for Healthcare · doi:10.1145/3458754 · Cited by 2,170 · cited 8×
“Second, it produced lightweight embeddings (384 dimensions) compared to other sentence transformers like the PubmedBERT model (Gu et al. 2021) (768 dimensions), making it more memory efficient.”
BackgroundMethodInfluentialA Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.18223 · Cited by 2,090
Large Language Models are Zero-Shot Reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11916 · Cited by 2,013
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013
Emergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264
On Faithfulness and Factuality in Abstractive Summarization
Maynez, J., Narayan, S., Bohnet, B., & McDonald, R. (2020) · arXiv · doi:10.48550/arxiv.2005.00661 · Cited by 1,185 · cited 1×
“Nevertheless, the “implicit representation” of knowledge in LLM has also been shown to generate non-factual information despite linguistically coherent answers (i.e. “hallucination”) as a response to the input prompt (Maynez et al. 2020, Raunak et al. 2021, Ji et al. 2023).”
BackgroundBioGPT: generative pre-trained transformer for biomedical text generation and mining
Luo, R., Sun, L., Xia, Y., Qin, T., Zhang, S., Poon, H., & Liu, T.-Y. (2022) · Briefings in Bioinformatics · doi:10.1093/bib/bbac409 · Cited by 1,150
Finetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101 · cited 1×
“Similar to the transfer learning ap - proach commonly used in deep neural networks, this implicit knowledge can be refined through supervised training to excel in a range of domain-specific tasks (Luo et al. 2023).”
BackgroundQA-GNN: Reasoning with Language Models and Knowledge Graphs for Question AnsweringPreprint
Yasunaga, M., Ren, H., Bosselut, A., Liang, P., & Leskovec, J. (2021) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2104.06378 · Cited by 939
Peng, Y., Yan, S., & Lu, Z. (2019) · Proceedings of the 18th BioNLP Workshop and Shared Task · doi:10.18653/v1/w19-5006 · Cited by 899
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., & Zettlemoyer, L. (2022) · arXiv · doi:10.48550/arxiv.2202.12837 · Cited by 752
Systematic integration of biomedical knowledge prioritizes drugs for repurposing
Himmelstein, D. S., Lizee, A., Hessler, C., Brueggeman, L., Chen, S. L., Hadley, D., Green, A., Khankhanian, P., & Baranzini, S. E. (2017) · eLife · doi:10.7554/elife.26726 · Cited by 666 · cited 1×
“…has been demonstrated through its successful application in various biomedical contexts (Supplementary S1 Text; Himmelstein and Baranzini, 2015, Himmelstein et al. 2017, Nelson et al. 2019, Nelson et al. 2021a,b, Baranzini et al. 2022, Morris et al. 2023, Soman et al. 2023a,b, Tang et al.…”
BackgroundSelf-citationClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
Huang, K., Altosaar, J., & Ranganath, R. (2019) · arXiv · doi:10.48550/arxiv.1904.05342 · Cited by 618 · cited 1×
“This approach has led to the creation of models such as PubMedBERT (Lee et al. 2020, Gu et al. 2021) BlueBERT (Peng et al. 2019), SciBERT (Beltagy et al. 2019), ClinicalBERT (Huang et al. 2019), BioGPT (Luo et al. 2022), Med-PaLM (Singhal et al. 2022), and BioMedGPT (Luo et al. 2023).”
MethodHow is ChatGPT's behavior changing over time?
Chen, L., Zaharia, M., & Zou, J. (2023) · arXiv · doi:10.48550/arxiv.2307.09009 · Cited by 507
Mungall, C. J., McMurry, J. A., Köhler, S., Balhoff, J. P., Borromeo, C., Brush, M., Carbon, S., Conlin, T., Dunn, N., Engelstad, M., et al. (2017) · Nucleic Acids Research · doi:10.1093/nar/gkw1128 · Cited by 441 · cited 1×
“MCQ, comprising five choices with a single correct answer for each question, were created using data from the Monarch Initiative (Mungall et al. 2017) and ROBOKOP (Reasoning Over Biomedical Objects linked in Knowledge-Oriented Pathways; Bizon et al. 2019) (Supplementary S1 Text).”
MethodSemMedDB: a PubMed-scale repository of biomedical semantic predications
Kilicoglu, H., Shin, D., Fiszman, M., Rosemblat, G., & Rindflesch, T. C. (2012) · Bioinformatics · doi:10.1093/bioinformatics/bts591 · Cited by 372
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · arXiv · doi:10.48550/arxiv.1908.10084 · Cited by 343
Deep Bidirectional Language-Knowledge Graph PretrainingPreprint
Yasunaga, M., Bosselut, A., Ren, H., Zhang, X., Manning, C. D., Liang, P., & Leskovec, J. (2022) · Neural Information Processing Systems · doi:10.48550/arxiv.2210.09338 · Cited by 300
Graph-Based Reasoning over Heterogeneous External Knowledge for Commonsense Question Answering
Lv, S., Guo, D., Xu, J., Tang, D., Duan, N., Gong, M., Shou, L., Jiang, D., Cao, G., & Hu, S. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6364 · Cited by 183
Himmelstein, D. S., & Baranzini, S. E. (2015) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1004259 · Cited by 172
Self-citationEnhancing Pre-Trained Language Representations with Rich Knowledge for Machine Reading Comprehension
Yang, A., Wang, Q., Liu, J., Liu, K., Lyu, Y., Wu, H., She, Q., & Li, S. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1226 · Cited by 155
Improving Natural Language Inference Using External Knowledge in the Science Questions Domain
Wang, X., Kapanipathi, P., Musa, R., Yu, M., Talamadupula, K., Abdelaziz, I., Chang, M., Fokoue, A., Makni, B., Mattei, N., et al. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33017208 · Cited by 148
The Curious Case of Hallucinations in Neural Machine Translation
Raunak, V., Menezes, A., & Junczys-Dowmunt, M. (2021) · arXiv · doi:10.48550/arxiv.2104.06683 · Cited by 142
Mondo: Unifying diseases for the world, by the world
Vasilevsky, N. A., Matentzoglu, N. A., Toro, S., Flack, J. E., Hegde, H., Unni, D. R., Alyea, G. F., Amberger, J. S., Babb, L., Balhoff, J. P., et al. (2022) · doi:10.1101/2022.04.13.22273750 · Cited by 107
Deep Bidirectional Language-Knowledge Graph Pretraining
Yasunaga, M., Bosselut, A., Ren, H., Zhang, X., Manning, C. D., Liang, P., & Leskovec, J. (2022) · neural information processing systems · doi:10.52202/068431-2704 · Cited by 103
Morris, J. H., Soman, K., Akbas, R. E., Zhou, X., Smith, B., Meng, E. C., Huang, C. C., Cerono, G., Schenk, G., Rizk-Jackson, A., et al. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad080 · Cited by 99 · cited 5×
“To achieve this, we make use of the biomedical KG called SPOKE (Morris et al. 2023) that integrates more than 40 publicly available biomedical knowledge sources of separate domains where each source is centered around a biomedical concept, such as genes, proteins, drugs, compounds, diseases, and…”
BackgroundMethodInfluentialSelf-citationTang, A. S., Rankin, K. P., Cerono, G., Miramontes, S., Mills, H., Roger, J., Zeng, B., Nelson, C., Soman, K., Woldemariam, S., et al. (2024) · Nature Aging · doi:10.1038/s43587-024-00573-8 · Cited by 86
Self-citationNelson, C. A., Butte, A. J., & Baranzini, S. E. (2019) · Nature Communications · doi:10.1038/s41467-019-11069-0 · Cited by 85
Self-citationROBOKOP KG and KGB: Integrated Knowledge Graphs from Federated Sources
Bizon, C., Cox, S., Balhoff, J. P., Kebede, Y., Wang, P., Morton, K. D., Fecho, K., & Tropsha, A. (2019) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.9b00683 · Cited by 76
Peng, Y., Yan, S., & Lu, Z. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.05474 · Cited by 72
Nelson, C. A., Bove, R., Butte, A. J., & Baranzini, S. E. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab270 · Cited by 50
Self-citationImproved Methods to Aid Unsupervised Evidence-Based Fact Checking for Online Heath News
Deka, P., Jurek-Loughrey, A., & P., D. (2022) · Journal of Data Intelligence · doi:10.26421/jdi3.4-5 · Cited by 49
SciBERT: A Pretrained Language Model for Scientific Text
Beltagy, I., Lo, K., & Cohan, A. (2019) · arXiv · doi:10.48550/arxiv.1903.10676 · Cited by 41
Soman, K., Nelson, C., Cerono, G., Goldman, S. M., Baranzini, S. E., & Brown, E. G. (2023) · Frontiers in Medicine · doi:10.3389/fmed.2023.1081087 · Cited by 34
Self-citationKagNet: Knowledge-Aware Graph Networks for Commonsense ReasoningPreprint
Lin, B. Y., Chen, X., Chen, J., & Ren, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.02151 · Cited by 32
BioMedGPT: Open Multimodal Generative Pre-trained Transformer for BioMedicinePreprint
Luo, Y., Zhang, J., Fan, S., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.09442 · Cited by 29
Nelson, C., Acuna, A. U., Paul, A. M., Scott, R. T., Butte, A. J., Cekanaviciute, E., Baranzini, S. E., & Costes, S. V. (2021) · Life · doi:10.3390/life11010042 · Cited by 28
Self-citationScalable Multi-Hop Relational Reasoning for Knowledge-Aware Question AnsweringPreprint
Feng, Y., Chen, X., Lin, B. Y., Wang, P., Yan, J., & Ren, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.00646 · Cited by 25
Graph-Based Reasoning over Heterogeneous External Knowledge for Commonsense Question AnsweringPreprint
Lv, S., Guo, D., Xu, J., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.05311 · Cited by 21
A Biomedical Open Knowledge Network Harnesses the Power of AI to Understand Deep Human Biology
Baranzini, S. E., Börner, K., Morris, J. P., Nelson, C., Soman, K., Schleimer, E., Keiser, M. J., Musen, M. A., Pearce, R., Reza, T. A., et al. (2022) · AI Magazine · doi:10.1609/aimag.v43i1.19125 · Cited by 12
Self-citationAre Emergent Abilities in Large Language Models just In-Context Learning?
Lu, S., Bigoulaeva, I., Sachdeva, R., Madabushi, H. T., & Gurevych, I. (2023) · arXiv · doi:10.48550/arxiv.2309.01809 · Cited by 12
SKILL: Structured Knowledge Infusion for Large Language ModelsPreprint
Moiseev, F., Dong, Z., Alfonseca, E., et al. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2205.08184 · Cited by 8
Time-aware Embeddings of Clinical Data using a Knowledge Graph
Soman, K., Nelson, C., Cerono, G., & Baranzini, S. E. (2022) · Pacific Symposium on Biocomputing (PSB) · doi:10.1142/9789811270611_0010 · Cited by 6
Self-citationLarge Language Models and Knowledge Graphs: Opportunities and Challenges
Pan, J. Z., Razniewski, S., Kalo, J.-C., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.06374
A biomedical open knowledge network harnesses the power of AI to understand deep human biology
Baranzini, S. E., Börner, K., Morris, J., Nelson, C. A., Soman, K., Schleimer, E., Keiser, M., Musen, M., Pearce, R., Reza, T., et al. (2022) · AI Magazine · doi:10.1002/aaai.12037
Self-citation
Cited by
Retrieval augmented generation for large language models in healthcare: A systematic review
Amugongo, L. M., Mascheroni, P., Brooks, S., Doering, S., & Seidel, J. (2025) · PLOS Digital Health · doi:10.1371/journal.pdig.0000877 · Cited by 267
A survey on knowledge-oriented retrieval-augmented generation
Cheng, M., Luo, Y., Ouyang, J., Liu, Q., Liu, H., Li, L., Yu, S., Zhang, B., Cao, J., Ma, J., et al. (2025) · arXiv · doi:10.48550/arxiv.2503.10677 · Cited by 140
Zhao, X., Liu, S., Yang, S.-Y., & Miao, C. (2025) · Proceedings of the ACM on Web Conference 2025 · doi:10.1145/3696410.3714782 · Cited by 116
Prompt Engineering in Clinical Practice: Tutorial for Clinicians
Liu, J., Liu, F., Wang, C., & Liu, S. (2025) · Journal of Medical Internet Research · doi:10.2196/72644 · Cited by 76
A comprehensive survey on integrating large language models with knowledge-based methods
Yang, W., Some, L., Bain, M., & Kang, B. H. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113503 · Cited by 154
Metadata
- Authors
- Soman, K., Rose, P. W., Morris, J. H., Akbas, R. E., Smith, B., Peetoom, B., Villouta-Reyes, C., Cerono, G., Shi, Y., Rizk-Jackson, A., Israni, S., Nelson, C. A., Huang, S., & Baranzini, S. E.
- Affiliations
- University of California, San Francisco (US)
- San Diego Supercomputer Center (US)
- Institute for Systems Biology (US)
- Bakar Computational Health Sciences Institute, University of California , San Francisco, CA 94158,
- Mate Bioservices, Inc. Swallowtail Ct. , Brisbane, CA 94005,
- Corresponding author
- Baranzini, Sergio E
- Venue
- Bioinformatics
- Publisher
- Oxford University Press (OUP)
- Published
- 2024-09
- Volume
- 40
- Issue
- 9
- ISSN
- 1367-4803 (ISSN-L)
- 1367-4811
- PMC
- PMC11441322
- OpenAlex
- W4402567773
- Cited by
- 135
- References
- 35
- Citations by year
- 2026: 61 · 2025: 63 · 2024: 9
- FWCI
- 10.72
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Bioinformatics (publishedVersion, cc-by)
- PubMed (publishedVersion)
- eScholarship (California Digital Library) (submittedVersion, cc-by)
- PubMed Central (submittedVersion, cc-by)
- Funders
- National Science Foundation (NSF_2033569)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Graph Neural Networks
- Keywords
- knowledge graph retrieval-augmented generation, prompt generation, large language models, biomedical knowledge graphs, context extraction, Llama 2, GPT-4, GPT-3.5-turbo
- MeSH terms
- Algorithms, Humans, Natural Language Processing, Computational Biology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.