Leveraging long context in retrieval augmented language models for medical question answering
Open accessAbstract
While holding great promise for improving and facilitating healthcare through applications of medical literature summarization, large language models (LLMs) struggle to produce up-to-date responses on evolving topics due to outdated knowledge or hallucination. Retrieval-augmented generation (RAG) is a pivotal innovation that improves the accuracy and relevance of LLM responses by integrating LLMs with a search engine and external sources of knowledge. However, the quality of RAG responses can be largely impacted by the rank and density of key information in the retrieval results, such as the "lost-in-the-middle" problem. In this work, we aim to improve the robustness and reliability of the RAG workflow in the medical domain. Specifically, we propose a map-reduce strategy, BriefContext, to combat the "lost-in-the-middle" issue without modifying the model weights. We demonstrated the advantage of the workflow with various LLM backbones and on multiple QA datasets. This method promises to improve the safety and reliability of LLMs deployed in healthcare domains by reducing the risk of misinformation, ensuring critical clinical content is retained in generated responses, and enabling more trustworthy use of LLMs in critical tasks such as medical question answering, clinical decision support, and patient-facing applications.
AI snapshot
Written from the paper’s full text.
TL;DR
BriefContext is a map-reduce RAG workflow that partitions retrieved documents into shorter contexts and summarizes extracted information to mitigate the “lost-in-the-middle” problem without changing model weights. Across controlled and real-world medical QA evaluations, it improved accuracy over vanilla RAG for several LLM backbones and included a preflight check to predict when the issue occurs.
Key findings
- In controlled synthetic-ranking experiments, BriefContext improved average accuracy over vanilla RAG: with Mixtral-7x8b from 57.66 to 60.41 at top_k = 16; with GPT-3.5-turbo from 54.82 to 58.11 at top_k = 8 and from 52.12 to 58.51 at top_k = 16.
- In integration testing with real retrieval rankings, accuracy improved from 55.81 to 66.47 with Llama2-70B-chat, from 76.75 to 79.03 with Llama3-70B-instruct, from 70.52 to 72.20 with Mixtral-7x8b, and from 69.19 to 72.51 with GPT-3.5-turbo-0125.
- The preflight check predicted “lost-in-the-middle” occurrence with 50.18% precision, 92.61% recall, and 65.09% F1; 35.7% of true-negative cases were correctly filtered, estimated to reduce extra overhead by up to 35%.
- On 48 open-ended medical questions reviewed by three medical experts, BriefContext generated better answers than RAG for 29.2% of questions, worse answers for 12.5%, and the same responses for 58.3%.
Methods
- BriefContext consists of four modules: Retrieval, Preflight check, Context Map, and Context Reduce. It divides retrieved documents into partitions, prompts LLMs to extract relevant information from each partition, and summarizes the extracted responses into a final answer without modifying model weights.
- Evaluation used multiple-choice questions from the MIRAGE benchmark, focusing on PubMedQA, BioASQ-Y/N, and MedMCQA, plus 48 open-ended questions generated from StatPearls. The knowledge base included PubMed abstracts and 18 medical textbooks; MedCPT was the primary search engine and BM25 was used as a secondary ranking for the preflight check.
- Controlled studies synthesized rankings by placing key documents at 0th, 25th, 50th, 75th, and 100th percentiles; integration testing used real retrieval rankings. Backbone LLMs included Llama2-70B-chat, Llama3-70B-instruct, Mixtral-7x8b, and GPT-3.5-turbo(-0125), with accuracy as the main metric.
Limitations
- Stated: the authors lacked open-ended questions annotated with key documents, so they could not quantitatively evaluate the impact of key document positioning on open-ended QA responses; they addressed this with controlled multiple-choice experiments.
- Stated: they used off-the-shelf LLMs without modification, and future work could explore the map-reduce paradigm with fine-tuned or task-specific LLMs; their focus was medical QA, with other tasks and scientific domains left for future studies.
- Evident from the text: the preflight check has low precision (50.18%), so false positives can incur unnecessary BriefContext calls; LLMs also do not always resolve conflicting context correctly, and failures can occur even in oracle settings where only key documents are provided.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 38 cited sources checked · 0 supported · 0 flagged
- Findingown data
BriefContext and vanilla RAG are at the same level in terms of theoretical complexity (big-O).
“In our cost analysis, we adopted the big-O notation, proving that BriefContext and vanilla RAG are at the same level in terms of theoretical complexity.”
Quote in the paperNot a substantial claim - Findingown data
With LLama2-70B-chat, BriefContext improved accuracy from 55.81 to 66.47 in integration testing.
“With LLama2-70B-chat, the accuracy was improved from 55.81 to 66.47; with LLama3-70B-instruct, the accuracy was improved from 76.75 to 79.03; with Mixtral-7x8b, the accuracy was improved from 70.52 to 72.20; with GPT-3.5-turbo-0125, the accuracy was improved from 69.19 to 72.51.”
Quote in the paperSubstantial claim- Rests on an assumption: The comparison between RAG and BriefContext remains fair despite possible inclusion of the knowledge base or corpus in LLM pretraining, because the same backbone LLMs are used.
- Findingreasoning
The extra cost incurred by BriefContext invocations is not significant in scale.
“Since the lengths of instruction and output are much shorter than the context information, the extra cost incurred by BriefContext invocations is not significant in scale.”
Quote in the paperNot a substantial claim - Findingown data
The preflight check (IoU heuristic) predicts the occurrence of "lost-in-the-middle" with 50.18% precision, 92.61% recall, and 65.09% F1.
“The IoU heuristic achieved 50.18% precision, 92.61% recall, and 65.09% F1 (Supplementary Table 5).”
- Also stated in: § Introduction · plain
“We show that the preflight check can predict the issue occurrence with a recall of 92.61% but a precision of 50.18%.”
Quote in the paperSubstantial claim - Also stated in: § Introduction · plain
- Findingown data
About 35.7% of true-negative cases can be correctly filtered by the preflight check.
“Based on the confusion matrix, 35.7% of true-negative cases can be correctly filtered, which avoids unnecessary procedure calls of BriefContext.”
Quote in the paperSubstantial claim
- Findingown data
Accuracy is higher when key documents are mixed with random documents (experimental group) compared to relevant documents (control group), even though the key documents are placed right in the middle of the context.
“Figure 6 shows that the accuracy is higher when the key documents are mixed with random documents (experimental group) as compared to relevant documents (control group), even though the key documents are placed right in the middle of the context.”
Quote in the paperSubstantial claim - Findingown data
LLMs perform better at reasoning over shorter contexts than longer ones when the same key information is in the context.
“Here, we prove the hypothesis that with the same key information in the context, LLMs perform better at reasoning over shorter contexts than longer ones.”
- Also stated in: § Introduction · boosted
“This confirms that short context is utilized more effectively than long context in RAG.”
Quote in the paperSubstantial claim - Also stated in: § Introduction · boosted
- Findingown data
BriefContext also improves model performance when the key information is placed in spotlight positions.
“BriefContext also improves the model performance when the key information is placed in spotlight positions.”
Quote in the paperSubstantial claim- Rests on an assumption: The comparison between RAG and BriefContext remains fair despite possible inclusion of the knowledge base or corpus in LLM pretraining, because the same backbone LLMs are used.
- Findingown data
LLMs can correctly resolve 74.7% of cases with conflicting information in the context window.
“We find LLMs can correctly resolve 74.7% of cases with conflicting information in the context window.”
- Also stated in: § Discussion · plain
“We show LLMs can correctly resolve 74.7% of cases with conflicting information in the context window.”
Quote in the paperSubstantial claim - Also stated in: § Discussion · plain
- Findingreasoning
Dividing the long context into several subtasks improves model performance on biomedical QA tasks without adjusting model weights.
“Next, by dividing the long context into several subtasks, we improve the model performance on biomedical QA tasks without adjusting model weights.”
Quote in the paperSubstantial claim - Findingown data
Using GPT-3.5-turbo as the LLM backbone, BriefContext improved accuracy from 54.82 to 58.11 when top_k = 8 (p << 0.01).
“Using GPT-3.5-turbo43 as the LLM backbone, the accuracy was improved from 54.82 to 58.11 when top_k = 8 (p << 0.01), and 52.12 to 58.51 when top_k = 16 (p << 0.01).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [43] Brown, T. et al. Language models are few-shot learners. Adv. Neural Inf. Process. Syst. 33, 1877–1901 (2020).
Not checked in the free run.
- Rests on an assumption: The comparison between RAG and BriefContext remains fair despite possible inclusion of the knowledge base or corpus in LLM pretraining, because the same backbone LLMs are used.
- Findingown data
With GPT-3.5-turbo-0125, BriefContext improved accuracy from 69.19 to 72.51 in integration testing.
“With LLama2-70B-chat, the accuracy was improved from 55.81 to 66.47; with LLama3-70B-instruct, the accuracy was improved from 76.75 to 79.03; with Mixtral-7x8b, the accuracy was improved from 70.52 to 72.20; with GPT-3.5-turbo-0125, the accuracy was improved from 69.19 to 72.51.”
Quote in the paperSubstantial claim- Rests on an assumption: The comparison between RAG and BriefContext remains fair despite possible inclusion of the knowledge base or corpus in LLM pretraining, because the same backbone LLMs are used.
- Findingown data
BriefContext consistently demonstrates a higher win rate than lose rate in handling context with conflict information.
“Overall, BriefContext consistently demonstrates a higher win rate than lose rate, which indicates the advantages of BriefContext handling context with conflict information.”
Quote in the paperSubstantial claim - Findingown data
In integration testing with real-world rankings, BriefContext improved overall accuracy across different LLM backbones.
“The results of integration testing shown in Fig. 3 and Supplementary Table 3 demonstrate that BriefContext has improved the overall accuracy across different LLM backbones.”
Quote in the paperSubstantial claim- Rests on an assumption: The comparison between RAG and BriefContext remains fair despite possible inclusion of the knowledge base or corpus in LLM pretraining, because the same backbone LLMs are used.
- Findingown data
Mixtral-7x8b resolved 171 out of 217 cases with conflicting contextual information correctly.
“Overall, Mixtral-7x8b resolved 171 out of 217 cases with conflicting contextual information correctly; GPT-3.5-turbo resolved 225 out of 313 cases.”
Quote in the paperSubstantial claim - Findingown data
Model performance decreases as the number of documents in the context increases, even though the same key information is present.
“We find that model performance decreases as the number of documents increases, even though the same key information is present in the context.”
- Also stated in: § Do LLMs favor short context over long context in the ContextMap step? · plain
“With the same key document in the context, the accuracy decreases as the number of documents increases.”
Quote in the paperSubstantial claim - Also stated in: § Do LLMs favor short context over long context in the ContextMap step? · plain
- Findingown data
The advantages of BriefContext in handling context with conflict information are more manifested in the 25th, 50th, and 75th percentile of positions than the others.
“The advantages are more manifested in 25th, 50th, 75th percentile of positions than the others.”
Quote in the paperNot a substantial claim - Findingown data
With LLama3-70B-instruct, BriefContext improved accuracy from 76.75 to 79.03 in integration testing.
“With LLama2-70B-chat, the accuracy was improved from 55.81 to 66.47; with LLama3-70B-instruct, the accuracy was improved from 76.75 to 79.03; with Mixtral-7x8b, the accuracy was improved from 70.52 to 72.20; with GPT-3.5-turbo-0125, the accuracy was improved from 69.19 to 72.51.”
Quote in the paperSubstantial claim- Rests on an assumption: The comparison between RAG and BriefContext remains fair despite possible inclusion of the knowledge base or corpus in LLM pretraining, because the same backbone LLMs are used.
- Findingown data
Mixtral-7x8B and GPT-3.5-turbo benefit more from retrieval augmentation than other models.
“We used Mixtral-7x8B and GPT-3.5-turbo as LLM backbones since these two models benefit more from retrieval augmentation than others (Fig. 2).”
Quote in the paperSubstantial claim - Findingown data
LLMs do not always correctly resolve conflict information.
“However, LLMs do not always correctly resolve the conflict information.”
Quote in the paperNot a substantial claim - Findingown data
GPT-3.5-turbo resolved 225 out of 313 cases with conflicting contextual information correctly.
“Overall, Mixtral-7x8b resolved 171 out of 217 cases with conflicting contextual information correctly; GPT-3.5-turbo resolved 225 out of 313 cases.”
Quote in the paperSubstantial claim - Findingown data
With Mixtral-7x8b, BriefContext improved accuracy from 70.52 to 72.20 in integration testing.
“With LLama2-70B-chat, the accuracy was improved from 55.81 to 66.47; with LLama3-70B-instruct, the accuracy was improved from 76.75 to 79.03; with Mixtral-7x8b, the accuracy was improved from 70.52 to 72.20; with GPT-3.5-turbo-0125, the accuracy was improved from 69.19 to 72.51.”
Quote in the paperSubstantial claim- Rests on an assumption: The comparison between RAG and BriefContext remains fair despite possible inclusion of the knowledge base or corpus in LLM pretraining, because the same backbone LLMs are used.
- Findingown data
Using Mixtral-7x8b as the LLM backbone, BriefContext improved accuracy averaged over different positions from 57.66 to 60.41 when top_k = 16 (p << 0.05).
“Using Mixtral-7x8b42 as the LLM backbone, the accuracy averaged over different positions was improved from 57.66 to 60.41 when top_k = 16 (p << 0.05).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [42] Jiang, A. Q. et al. Mixtral of Experts. arXiv [cs.LG] (2024).
Not checked in the free run.
- Rests on an assumption: The comparison between RAG and BriefContext remains fair despite possible inclusion of the knowledge base or corpus in LLM pretraining, because the same backbone LLMs are used.
- Findingown data
Using GPT-3.5-turbo as the LLM backbone, BriefContext improved accuracy from 52.12 to 58.51 when top_k = 16 (p << 0.01).
“Using GPT-3.5-turbo43 as the LLM backbone, the accuracy was improved from 54.82 to 58.11 when top_k = 8 (p << 0.01), and 52.12 to 58.51 when top_k = 16 (p << 0.01).”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [43] Brown, T. et al. Language models are few-shot learners. Adv. Neural Inf. Process. Syst. 33, 1877–1901 (2020).
Not checked in the free run.
- Rests on an assumption: The comparison between RAG and BriefContext remains fair despite possible inclusion of the knowledge base or corpus in LLM pretraining, because the same backbone LLMs are used.
- Findingnone given
LLMs can better utilize short context than long context.
“First, we showed that LLMs can better utilize short context than long context.”
Quote in the paperNot a substantial claim - Findingown data
LLMs may still fail to answer questions correctly even in oracle settings, where only key documents are provided as the context.
“Second, although the RAG paradigm improves LLMs via external knowledge sources, we show that LLMs may still fail to answer questions correctly even in the oracle settings, where only key documents were provided as the context.”
Quote in the paperSubstantial claim - Premisecitation
If the key document is not ranked at the top, the key information may be neglected by the generative module, resulting in incomplete or inaccurate responses.
“If not ranked at the top, the key information may be neglected by the generative module, resulting in incomplete or inaccurate responses to the user queries24.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [24] Liu, N. F. et al. Lost in the middle: how language models use long contexts. Trans. Assoc. Comput. Linguist. 12, 157–173 (2023). source
Not checked in the free run.
- Premisecitation
The increased amount of content to reason over can hinder model performance.
“Even though recently released models can process an increasing number of tokens, the increased amount of content to reason over can still hinder model performance23.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [23] Li T, Zhang G, Do QD, Yue X, Chen W. Long-context LLMs struggle with long in-context learning. arXiv [csCL]. 2024.
Not checked in the free run.
- Premisecitation
With a retrieval module, the generative module can provide more accurate and factual answers without continual training or fine-tuning.
“With the help of the retrieval module, the generative module can provide more accurate and factual answers without the need for continual training or fine-tuning.”
Quote in the paperSubstantial claim0 of 2 sources checked- cites [14] Chen, J., Lin, H., Han, X. & Sun, L. Benchmarking large language models in retrieval-augmented generation. AAAI 38, 17754–17762 (2024). source
Not checked in the free run.
- cites [22] Ding, Y. et al. A Survey on RAG Meets LLMs: towards retrieval-augmented large language models. arXiv [cs.CL] (2024).
Not checked in the free run.
- Premisecitation
LLMs struggle to generate accurate and up-to-date responses on current topics due to outdated knowledge, lack of domain-specific expertise, or hallucination.
“LLMs struggle to generate accurate and up-to-date responses on current topics, due to outdated knowledge, lack of domain-specific expertise, or hallucination13–17.”
Quote in the paperSubstantial claim0 of 5 sources checked- cites [13] Cao, M., Dong, Y., Wu, J. & Cheung, J. C. K. Factual error correction for abstractive summarization models. In Proc. 2020 Conf. Empirical Methods in Natural Language Processing (EMNLP) 6251–6258 (Association for Computational Linguistics, 2020). https://doi.org/10.18653/v1/2020.emnlp-main.506. source
Not checked in the free run.
- cites [14] Chen, J., Lin, H., Han, X. & Sun, L. Benchmarking large language models in retrieval-augmented generation. AAAI 38, 17754–17762 (2024). source
Not checked in the free run.
- cites [15] Raunak, V., Menezes, A. & Junczys-Dowmunt, M. The curious case of hallucinations in neural machine translation. In Proc. 2021 Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) 1172–1183 (Association for Computational Linguistics, 2021). https://doi.org/10.18653/v1/2021.naacl-main.92. source
Not checked in the free run.
- cites [16] Ji, Z. et al. Survey of hallucination in natural language generation. ACM Comput. Surv. 55, 1–38 (2023). source
Not checked in the free run.
- cites [17] Zhang, G. et al. Leveraging generative AI for clinical evidence synthesis needs to ensure trustworthiness. J. Biomed. Inform. 153, 104640 (2024). source
Not checked in the free run.
- Premisecitation
The position of key information in the LLM context impacts the quality of model completions, known as the "lost-in-the-middle" issue.
“A recent study reports an issue of “lost-in-the-middle”, i.e., the position of key information in the LLM context impacts the quality of the model completions24.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [24] Liu, N. F. et al. Lost in the middle: how language models use long contexts. Trans. Assoc. Comput. Linguist. 12, 157–173 (2023). source
Not checked in the free run.
- Premisecitation
The quality of RAG completion depends on retrieval results, such as the density or positions of query-relevant information.
“The quality of RAG completion also depends on the retrieval results, such as the density or positions of query-relevant information14,22,24–26.”
Quote in the paperSubstantial claim0 of 5 sources checked- cites [14] Chen, J., Lin, H., Han, X. & Sun, L. Benchmarking large language models in retrieval-augmented generation. AAAI 38, 17754–17762 (2024). source
Not checked in the free run.
- cites [22] Ding, Y. et al. A Survey on RAG Meets LLMs: towards retrieval-augmented large language models. arXiv [cs.CL] (2024).
Not checked in the free run.
- cites [24] Liu, N. F. et al. Lost in the middle: how language models use long contexts. Trans. Assoc. Comput. Linguist. 12, 157–173 (2023). source
Not checked in the free run.
- cites [25] Jiang, H. et al. LongLLMLingua: Accelerating and enhancing LLMs in long context scenarios via prompt compression. In Proc. 62nd Annu. Meet. Assoc. Comput. Linguistics (ACL) 1658–1677 (Association for Computational Linguistics, 2024). https://doi.org/10.18653/v1/2024.acl-long.91. source
Not checked in the free run.
- cites [26] Xiong, G., Jin, Q., Lu, Z. & Zhang, A. Benchmarking retrieval-augmented generation for medicine. In Findings of the Association for Computational Linguistics ACL 2024 6233–6251 (2024). https://doi.org/10.18653/V1/2024.FINDINGS-ACL.372. source
Not checked in the free run.
- Study design and conductreasoning
They prioritize achieving high recall over high precision in the preflight check threshold.
“To better demonstrate the effectiveness of our methods, we prioritize achieving high recall over high precision.”
Quote in the paperSubstantial claim - Premisecitation
Adjusting model weights can lead to catastrophic forgetting, where overall LLM performance degrades upon adopting new information on a specific task.
“However, adjusting model weights can lead to catastrophic forgetting30,31, i.e., the overall performance of LLMs degrades upon adopting new information on a specific task.”
Quote in the paperSubstantial claim0 of 2 sources checked- cites [30] Kemker, R., McClure, M., Abitino, A., Hayes, T. & Kanan, C. Measuring catastrophic forgetting in neural networks. In Proceedings of the AAAI conference on artificial intelligence, Vol. 32, No. 1, 3390–3398 (2018). source
Not checked in the free run.
- cites [31] Kirkpatrick, J. et al. Overcoming catastrophic forgetting in neural networks. Proc. Natl. Acad. Sci. USA. 114, 3521–3526 (2017). source
Not checked in the free run.
- Interpretationown data
BriefContext can better utilize the key document than RAG, mainly when the context contains conflicting information.
“As such, the BriefContext can better utilize the key document than RAG, mainly when the context contains conflicting information.”
Quote in the paperSubstantial claim - Interpretationown data
Positional attention bias is overpowered by text-embedding-based attention when the key information is distinguishable from other documents in the context.
“These results prove that positional attention bias is overpowered by text-embedding-based attention when the key information is distinguishable from other documents in the context.”
Quote in the paperSubstantial claim - Interpretationreasoning
Search engines based on embedding representation or dense retrieval sometimes return irrelevant documents that manifest a high resemblance to the query vocabulary.
“Furthermore, this observation highlights a limitation of search engines based on embedding representation or dense retrieval. These search engines sometimes return irrelevant documents that manifest a high resemblance to the query vocabulary.”
Quote in the paperNot a substantial claim - Interpretationown data
The preflight check can help reduce extra overhead by up to 35%.
“Based on these numbers, we estimate the preflight check can help reduce extra overhead by up to 35%.”
Quote in the paperSubstantial claim - Interpretationown data
BriefContext improves the robustness of the RAG paradigm regarding the order of retrieved documents without adjusting model weights.
“Our experiments demonstrate that BriefContext improves the robustness regarding the order of retrieved documents in the RAG paradigm without adjusting model weights.”
Quote in the paperSubstantial claim- Rests on an assumption: The comparison between RAG and BriefContext remains fair despite possible inclusion of the knowledge base or corpus in LLM pretraining, because the same backbone LLMs are used.
- Interpretationown data
Key information is under-utilized by vanilla RAG, especially when the context contains conflicting information.
“This also highlights that the key information is under-utilized by the vanilla RAG, especially when the context contains conflicting information.”
Quote in the paperSubstantial claim - Interpretationown data
Positional attention bias is triggered when the key documents contain similar vocabulary to other documents in the context that do not provide supporting information to the user query.
“We show that positional attention bias is triggered when the key documents contain similar vocabulary to other documents in the context that do not provide supporting information to the user query.”
Quote in the paperSubstantial claim - Interpretationreasoning
The lack of extra information in the knowledge base makes the comparison between BriefContext and RAG fair.
“which makes a fair comparison between BriefContext and RAG.”
Quote in the paperSubstantial claim - Premisecitation
Model performance drops significantly when N (top N positions) becomes larger than 3.
“The threshold N is set to 3 since model performance drops significantly when N becomes larger than 3, according to earlier studies24,27,28.”
Quote in the paperSubstantial claim0 of 3 sources checked- cites [24] Liu, N. F. et al. Lost in the middle: how language models use long contexts. Trans. Assoc. Comput. Linguist. 12, 157–173 (2023). source
Not checked in the free run.
- cites [27] He, J. et al. Never lost in the middle: improving large language models via attention strengthening question answering. arXiv [cs.CL] (2023).
Not checked in the free run.
- cites [28] Hsieh, C.-Y. et al. Found in the middle: Calibrating positional attention bias improves long context utilization. In Findings of the Association for Computational Linguistics (ACL) 14982–14995 (Association for Computational Linguistics, 2024). https://doi.org/10.18653/v1/2024.findings-acl.890. source
Not checked in the free run.
- Study design and conductreasoning
The knowledge base used in integration testing contains no extra information compared to the MIRAGE benchmark corpus.
“The knowledge base is a subset of the corpus that was used in the MIRAGE benchmark results reported by Xiong et al.26. Our knowledge base thus contains no extra information as compared to theirs”
Quote in the paperSubstantial claim - Interpretationreasoning
BriefContext improves biomedical QA accuracy in RAG workflow by taking advantage of LLMs' conflict resolution capability and shorter context.
“BriefContext takes advantage of this capability of LLMs and shorter context, which explains how BriefContext improves biomedical QA accuracy in RAG workflow.”
Quote in the paperSubstantial claim
- Classified as probably not a substantial claim. 21
- May combine two claims. 7
- The quote only partly states this claim. 1
Probably not a substantial claim: may not pass S4 (based).
Findingnone givenCompared to closed-book settings, LLMs perform significantly better in Oracle settings.
“Compared to closed-book settings, LLMs perform significantly better in Oracle settings.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataIn vanilla RAG, the average numbers of input and output tokens per request are 3066.0 and 183.0, respectively.
“In vanilla RAG, these numbers are 3066.0 and 183.0.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataIn expert evaluation of 48 open-ended medical questions, BriefContext generated better answers than the RAG baseline for 29.2% of questions and worse answers for 12.5%, with the remaining 58.3% the same.
“Out of the 48 questions, our method generates better answers than the RAG baseline for 29.2% of questions and worse answers for 12.5% of questions. For the remaining 58.3% of questions, our method and the RAG baseline produced the same responses.”
Quote in the paperSubstantial claim- Rests on an assumption: The comparison between RAG and BriefContext remains fair despite possible inclusion of the knowledge base or corpus in LLM pretraining, because the same backbone LLMs are used.
The quote only partly states this claim.
Findingown dataBriefContext improves accuracy over baseline RAG in controlled studies with synthetic rankings when key information is placed in the middle.
“BriefContext utilizes the external information in the middle of the context more effectively than the baseline RAG workflow”
- Also stated in: § Introduction · boosted
“These experiments demonstrate that BriefContext consistently outperforms the RAG baseline by a substantial margin when the key information is placed in the middle.”
Quote in the paperSubstantial claim- Rests on an assumption: The comparison between RAG and BriefContext remains fair despite possible inclusion of the knowledge base or corpus in LLM pretraining, because the same backbone LLMs are used.
- Also stated in: § Introduction · boosted
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataIn BriefContext, the average numbers of input and output tokens per request are 5496.5 and 247.5, respectively.
“In BriefContext, the average numbers of input and output tokens per request are 5496.5 and 247.5, respectively.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
PremisecitationThe "lost-in-the-middle" issue is attributed to positional attention bias, where more attention weights are allocated to information at spotlight positions than others.
“Current studies attribute this issue to positional attention bias, i.e., more attention weights are allocated more to information at spotlight positions than others27–29.”
Quote in the paperSubstantial claim0 of 3 sources checked- cites [27] He, J. et al. Never lost in the middle: improving large language models via attention strengthening question answering. arXiv [cs.CL] (2023).
Not checked in the free run.
- cites [28] Hsieh, C.-Y. et al. Found in the middle: Calibrating positional attention bias improves long context utilization. In Findings of the Association for Computational Linguistics (ACL) 14982–14995 (Association for Computational Linguistics, 2024). https://doi.org/10.18653/v1/2024.findings-acl.890. source
Not checked in the free run.
- cites [29] Wu, T., Zhao, Y. & Zheng, Z. An efficient recipe for long context extension via middle-focused positional encoding. In Adv. Neural Inf. Process. Syst. 38 (NeurIPS, 2024).
Not checked in the free run.
May combine two claims.
PremisecitationRetrieval systems are imperfect, so it is inevitable to retrieve information irrelevant to the user query.
“As retrieval systems are still imperfect, it is inevitable to retrieve information irrelevant to the user query14.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [14] Chen, J., Lin, H., Han, X. & Sun, L. Benchmarking large language models in retrieval-augmented generation. AAAI 38, 17754–17762 (2024). source
Not checked in the free run.
May combine two claims.
PremisecitationRAG improves the accuracy and relevance of LLM responses by integrating LLMs with a search engine and external sources of knowledge.
“Retrieval-Augmented Generation (RAG) is a pivotal innovation to enhance the quality and relevance of responses in LLMs18–21.”
Quote in the paperSubstantial claim0 of 4 sources checked- cites [18] Guu, K., Lee, K., Tung, Z., Pasupat, P. & Chang, M. Retrieval augmented language model pre-training. In Proc. 37th International Conference on Machine Learning (eds. Iii, H. D. & Singh, A.) 3929–3938 (PMLR, 2020).
Not checked in the free run.
- cites [19] Lewis, P., Perez, E., Piktus, A. & Petroni, F. Retrieval-augmented generation for knowledge-intensive nlp tasks. Adv. Neural Inform. Process. Syst. 33, 9459–9474 (2020).
Not checked in the free run.
- cites [20] Borgeaud, S. et al. Improving language models by retrieving from trillions of tokens. In Proc. 39th International Conference on Machine Learning (eds. Chaudhuri, K. et al.)2206–2240 (PMLR, 2022).
Not checked in the free run.
- cites [21] Izacard, G. & Grave, E. Leveraging passage retrieval with generative models for open domain question answering. In Proc. 16th Conf. of the European Chapter of the Association for Computational Linguistics (EACL) 874–880 (Association for Computational Linguistics, 2021). https://doi.org/10.18653/v1/2021.eacl-main.74. source
Not checked in the free run.
- Rests on an assumption: Providing contextual information enhances the model’s ability to perform downstream tasks by augmenting LLMs with external domain-specific knowledge that is under-represented in their pretraining data.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationStatPearls summarizes up-to-date medical knowledge and practice across various specialties.
“We created these questions using StatPearls50, a source that summarizes up-to-date medical knowledge and practice across various specialties.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [50] StatPearls Publishing. (n.d.). StatPearls. Retrieved September 5th, 2024, from https://www.ncbi.nlm.nih.gov/books/NBK430685/.
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenThe generated MedQ QA pairs were reviewed by three specialties (dermatology, neurology, and endocrinology) to ensure their accuracy and relevance.
“The generated QA pairs were then reviewed by three specialties (dermatology, neurology, and endocrinology) to ensure their accuracy and relevance.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).
InterpretationreasoningBriefContext shows promise for generalizing to tasks requiring effective processing of long-context data, such as extracting pertinent data from lengthy, duplicative electronic health records, legal document analysis, historical research, or technical report summarization.
“While our proposed BriefContext framework was evaluated only within the biomedical question-answering in this study, it shows promise for generalizing to tasks that require effective processing of long-context data, such as extracting pertinent data from lengthy, duplicative electronic health records, legal document analysis, historical research, or technical report summarization.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationMedCPT was specifically pretrained on biomedical literature using user click information.
“MedCPT was specifically pretrained on biomedical literature using user click information2.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [2] Jin, Q. et al. MedCPT: contrastive pre-trained transformers with large-scale PubMed search logs for zero-shot biomedical information retrieval. Bioinformatics 39, btad651 (2023). source
Not checked in the free run.
Classified as probably not a substantial claim.
InterpretationreasoningThe explicit specifications of key documents in PubMedQA and BioASQ-Y/N allow analysis of the relationship between key document position and pipeline accuracy.
“This information allows us to perform a deep analysis the relationship between the key document position and the pipeline accuracy.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
InterpretationreasoningIn real-world scenarios, constant factors increase the actual costs of BriefContext.
“However, in real-world scenarios, constant factors do play a role. While these extra tokens do not impact the big-O analysis, they would still increase the actual costs.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Study design and conductown dataThree medical experts curated 48 open-ended question-answer pairs from their specialty domain.
“In the second integration testing, we invited three medical experts to help curate 48 open-ended question-answer pairs from their specialty domain and compare our method with the RAG baseline.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenTheir current focus is on QA tasks in the medical domain.
“Lastly, our current focus is on QA tasks in the medical domain.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
Study design and conductreasoningThey addressed the lack of open-ended questions by conducting a controlled experiment using multiple-choice questions with key documents placed at various positions.
“However, we addressed this by conducting a controlled experiment using multiple-choice questions where the key document was strategically placed at various positions within the prompt context.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductreasoningRetrieval augmented LLMs based on clinical data is out of the scope of this study due to the sensitive nature of clinical data.
“However, due to the sensitive nature, retrieval augmented LLMs based on clinical data is out of the scope of this study.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationTransformer architecture adopts self-attention where weight is calculated as an inner-product between each pair of embeddings.
“Recall that Transformer architecture adopts the self-attention mechanism, where the weight is calculated as an inner-product between each pair of embeddings44.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [44] Vaswani, A. et al. Attention is all you need. Adv. Neural Inf. Process. Syst. 30, 5998–6008 (2017).
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenIn their study, they decouple the impact of segment embeddings on attention weights from the impact of positional embeddings.
“In our study, we decouple the impact of segment embeddings on attention weights from the impact of positional embeddings.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductown dataDue to lack of open-ended questions annotated with key documents, they cannot quantitatively evaluate the impact of key document positioning on QA responses.
“Firstly, due to the lack of open-ended questions annotated with key documents, we cannot quantitatively evaluate the impact of key document positioning on QA responses.”
Quote in the paperNot a substantial claimMay combine two claims.
PremisecitationPubMedQA and BioASQ-Y/N contain explicit specifications of the key documents.
“two biomedical research QA subsets (PubMedQA41 and BioASQ-Y/N40)”
Quote in the paperSubstantial claim0 of 2 sources checked- cites [41] Jin, Q., Dhingra, B., Liu, Z., Cohen, W. & Lu, X. PubMedQA: a dataset for biomedical research question answering. In Proc. 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) 2567–2577 (Association for Computational Linguistics, 2019). source
Not checked in the free run.
- cites [40] Tsatsaronis, G. et al. An overview of the BIOASQ large-scale biomedical semantic indexing and question answering competition. BMC Bioinformatics 16, 138 (2015). source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
PremisecitationEach embedding consists of positional, token, and segment embeddings, which encode position and semantics.
“Each embedding consists of positional, token, and segment embeddings, which encode position and semantics, respectively20,45.”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites [20] Borgeaud, S. et al. Improving language models by retrieving from trillions of tokens. In Proc. 39th International Conference on Machine Learning (eds. Chaudhuri, K. et al.)2206–2240 (PMLR, 2022).
Not checked in the free run.
- cites [45] Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. 2019 Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) 4171–4186 (Association for Computational Linguistics, 2019). https://doi.org/10.18653/v1/n19-1423. source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenTheir choice of off-the-shelf LLMs without modifications is a limitation.
“Secondly, our choice of the off-the-shelf LLMs without any modifications presents another limitation.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
PremisereasoningDecoder-only models cannot attend to query tokens if the query is placed behind the contextual information.
“It has been pointed out that decoder-only models cannot attend to query tokens if the query is only placed behind the contextual information, since decoder-only models only attend to prior tokens by each timestamp.”
Quote in the paperSubstantial claimMay combine two claims.
InterpretationreasoningThe problem of reasoning over long context can be divided into multiple subtasks of reasoning over short context, and the correct answer can be more easily located in one of the subtasks.
“Furthermore, since LLMs perform better at reasoning over shorter contexts than longer ones, the problem of reasoning over long context can be divided into multiple subtasks of reasoning over short context, and the correct answer can be more easily located in one of the subtasks.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.
Interpretationnone givenIf the context is as short as a prompt instruction, the lost-in-the-middle issue would not appear, thus not requiring the BriefContext procedure.
“In case the context is as short as a prompt instruction, the lost-in-the-middle issue would not appear, thus not requiring the BriefContext procedure.”
Quote in the paperNot a substantial claim
The paper relies on this without support.
The comparison between RAG and BriefContext remains fair despite possible inclusion of the knowledge base or corpus in LLM pretraining, because the same backbone LLMs are used.
“While a portion of the knowledge base or corpus where the dataset was derived (e.g., PubMed abstracts or textbooks) is probably included in the pre-training of LLMs, we deem the comparison remains fair, since we used the same backbone LLMs for RAG and BriefContext.”
The paper relies on this without support.
Providing contextual information enhances the model’s ability to perform downstream tasks by augmenting LLMs with external domain-specific knowledge that is under-represented in their pretraining data.
“On the one hand, providing contextual information enhances the model’s ability to perform the downstream tasks by augmenting LLMs with external domain-specific knowledge that is under-represented in their pretraining data.”
The paper relies on this without support.
In real-world medical QA, questions arise without predefined options.
“In the real-world practice of medical QA, questions always arise without predefined options, reflecting the open-ended nature of real-world scenarios.”
The paper relies on this without support.
Scientific literature has limitations such as publication bias and lack of generalizability across different population groups.
“Scientific literature has certain limitations, such as publication bias and lack of generalizability across different population groups.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
GenAI and LLMs have fundamentally transformed global knowledge production.
“The rapid proliferation of Generative Artificial Intelligence (GenAI) and Large Language Models (LLMs) has fundamentally transformed global knowledge production (Zhang et al., 2025).”
From The Attention Mismatch: Mapping the Structural Academic Governance Deficit in the Age of Generative AI (2026) · cites this paper as Zhang et al., 2025
The Introduction discusses LLMs’ expanding use in healthcare, and the Abstract focuses on medical question answering. Neither addresses whether GenAI and LLMs have fundamentally transformed global knowledge production; searches for that claim and for knowledge creation found no supporting passage in our copy. (LLM, conf=0.75)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Discussion cites the wrong figure for document-count results
MinorThe discussion attributes the decrease in accuracy with more documents to Fig. 6, but the paper's Fig. 5 caption identifies that result. Fig. 6 instead compares real retrieval with randomly selected context documents.
“As shown in Fig. 6, the QA accuracy decreases as the number of documents is increased.”
“Figure: Fig. 5 Medical QA accuracy of LLMs with various numbers of documents as context information. We show the mean and standard deviation of accuracy with different number of documents in the context window.”
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Overcoming catastrophic forgetting in neural networks
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1611835114 · Cited by 7,899 · see citing works
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · see citing works
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083 · see citing works
Self-citationJin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2020) · Applied Sciences · doi:10.48550/arxiv.2009.13081 · Cited by 2,320 · see citing works
Lost in the Middle: How Language Models Use Long Contexts
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00638 · Cited by 1,612 · cited 4× · see citing works
“The quality of RAG completion also depends on the retrieval results, such as the density or positions of query-relevant information 14,22,24 – 26 .”
InfluentialDean, J., & Ghemawat, S. (2009) · Communications of the ACM · doi:10.1145/1629175.1629198 · Cited by 1,185 · see citing works
Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering
Izacard, G., & Grave, E. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.74 · Cited by 1,109 · see citing works
PubMedQA: A Dataset for Biomedical Research Question Answering
Jin, Q., Dhingra, B., Liu, Z., Cohen, W., & Lu, X. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1259 · Cited by 685 · see citing works
Self-citationMeasuring Catastrophic Forgetting in Neural Networks
Kemker, R., McClure, M., Abitino, A., Hayes, T., & Kanan, C. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11651 · Cited by 672 · see citing works
Tsatsaronis, G., Balikas, G., Malakasiotis, P., Partalas, I., Zschunke, M., Alvers, M. R., Weissenborn, D., Krithara, A., Petridis, S., Polychronopoulos, D., et al. (2015) · BMC Bioinformatics · doi:10.1186/s12859-015-0564-6 · Cited by 598 · see citing works
Benchmarking Large Language Models in Retrieval-Augmented Generation
Chen, J., Lin, H., Han, X., & Sun, L. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29728 · Cited by 454 · see citing works
Improving language models by retrieving from trillions of tokens
Borgeaud, S., Mensch, A., Hoffmann, J., Cai, T., Rutherford, E., Millican, K., Driessche, G. V. D., Lespiau, J.-B., Damoc, B., Clark, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.04426 · Cited by 382 · see citing works
Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Health
Tian, S., Jin, Q., Yeganova, L., Lai, P.-T., Zhu, Q., Chen, X., Yang, Y., Chen, Q., Kim, W., Comeau, D. C., et al. (2023) · Briefings in Bioinformatics · doi:10.1093/bib/bbad493 · Cited by 359 · see citing works
Self-citationAI-Generated Medical Advice—GPT and Beyond
Haupt, C. E., & Marks, M. (2023) · JAMA · doi:10.1001/jama.2023.5321 · Cited by 346 · see citing works
Can large language models reason about medical questions?
Liévin, V., Hother, C. E., Motzfeldt, A. G., & Winther, O. (2024) · Patterns · doi:10.1016/j.patter.2024.100943 · Cited by 343 · cited 1× · see citing works
“In particular, we evaluate general-purpose LLMs on answering medical QA questions that require domain knowledge in depth 1,26,38 – 41 .”
Benchmarking Retrieval-Augmented Generation for Medicine
Xiong, G., Jin, Q., Lu, Z., & Zhang, A. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.372 · Cited by 310 · see citing works
Self-citationMeasuring Massive Multitask Language Understanding
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2020) · arXiv · doi:10.48550/arxiv.2009.03300 · Cited by 291 · see citing works
Jin, Q., Kim, W., Chen, Q., Comeau, D. C., Yeganova, L., Wilbur, W. J., & Lu, Z. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad651 · Cited by 175 · see citing works
Self-citationIs Cosine-Similarity of Embeddings Really About Similarity?
Steck, H., Ekanadham, C., & Kallus, N. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589335.3651526 · Cited by 161 · see citing works
The Curious Case of Hallucinations in Neural Machine Translation
Raunak, V., Menezes, A., & Junczys-Dowmunt, M. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.92 · Cited by 142 · see citing works
Factual Error Correction for Abstractive Summarization Models
Cao, M., Dong, Y., Wu, J., & Cheung, J. C. K. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.506 · Cited by 132 · see citing works
LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
Jiang, H., Wu, Q., Luo, X., Li, D., Lin, C.-Y., Yang, Y., & Qiu, L. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.91 · Cited by 96 · see citing works
Zhang, G., Jin, Q., Zhou, Y., Wang, S., Idnay, B., Luo, Y., Park, E., Nestor, J. G., Spotnitz, M. E., Soroush, A., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01239-w · Cited by 87 · see citing works
Self-citationAI-generated text may have a role in evidence-based medicine
Peng, Y., Rousseau, J. F., Shortliffe, E. H., & Weng, C. (2023) · Nature Medicine · doi:10.1038/s41591-023-02366-9 · Cited by 64 · see citing works
Self-citationRare disease diagnosis using knowledge guided retrieval augmentation for ChatGPT
Zelin, C., Chung, W. K., Jeanne, M., Zhang, G., & Weng, C. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104702 · Cited by 47 · see citing works
Self-citationA Survey of Clinicians' Views of the Utility of Large Language Models
Spotnitz, M., Idnay, B., Gordon, E. R., Shyu, R., Zhang, G., Liu, C., Cimino, J. J., & Weng, C. (2024) · Applied Clinical Informatics · doi:10.1055/a-2281-7092 · Cited by 47 · see citing works
Self-citationPark, J., Fang, Y., Ta, C., Zhang, G., Idnay, B., Chen, F., Feng, D. D., Shyu, R., Gordon, E. R., Spotnitz, M., et al. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104649 · Cited by 42 · see citing works
Self-citationLeveraging generative AI for clinical evidence synthesis needs to ensure trustworthiness
Zhang, G., Jin, Q., Jered McInerney, D., Chen, Y., Wang, F., Cole, C. L., Yang, Q., Wang, Y., Malin, B. A., Peleg, M., et al. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104640 · Cited by 40 · see citing works
Self-citationFound in the middle: Calibrating Positional Attention Bias Improves Long Context Utilization
Hsieh, C.-Y., Chuang, Y.-S., Li, C.-L., Wang, Z., Le, L., Kumar, A., Glass, J., Ratner, A., Lee, C.-Y., Krishna, R., et al. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.890 · Cited by 31 · see citing works
Zhang, G., Zhou, Y., Hu, Y., Xu, H., Weng, C., & Peng, Y. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae065 · Cited by 17 · see citing works
Self-citationLongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
Jiang, H., Wu, Q., Luo, X., Li, D., Lin, C.-Y., Yang, Y., & Qiu, L. (2023) · arXiv · Cited by 12 · see citing works
MapReduce-based distributed tensor clustering algorithm
Zhang, H., Li, P., Meng, F., Fan, W., & Xue, Z. (2023) · Neural Computing and Applications · doi:10.1007/s00521-023-08415-1 · Cited by 8 · see citing works
Idnay, B., Zhang, G., Chen, F., Ta, C., Schelke, M. W., Marder, K. S., & Weng, C. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae274 · Cited by 6 · see citing works
Self-citationBergui, M., Hourri, S., Najah, S., & Nikolov, N. S. (2024) · Journal Of Big Data · doi:10.1186/s40537-024-00964-z · Cited by 5 · see citing works
Selvi, R. S., Sankari, V., Ramya, N., & Selvi, M. (2024) · Journal of Circuits Systems and Computers · doi:10.1142/s0218126624502025 · Cited by 2 · see citing works
Face Image Encryption Using Fuzzy K2DPCA and Chaotic MapReduce
Luo, Y., & Li, J. (2024) · Tehnicki vjesnik - Technical Gazette · doi:10.17559/tv-20230922000954 · Cited by 1 · see citing works
Survey on MapReduce Scheduler Algorithms in Hadoop Framework
Varadharajan, K. M., PremKumar, D. J., & Asokaraj, M. (2024) · International Journal of Innovative Research in Information Security · doi:10.26562/ijiris.2024.v1004.38 · Cited by 1 · see citing works
Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization
Hsieh, C.-Y., Chuang, Y.-S., Li, C.-L., Wang, Z., Le, L. T., Kumar, A., Glass, J., Ratner, A., Lee, C.-Y., Krishna, R., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.16008 · cited 4×
“Recent studies 27,28 pointed out that lost-in-middle-issue is attributed to the positional attention bias, i.e., models exhibit U-shaped attention patterns where documents at the beginning or end of the inputs receive higher attention values, regardless of their relevance.”
Influential
Flagged first, then by citations
Research works printed but not linked: 12
- Jin, Q. et al. Demystifying large language models for medicine: a primer. arXiv [cs.AI] (2024).
- Guu, K., Lee, K., Tung, Z., Pasupat, P. & Chang, M. Retrieval augmented language model pre-training. In Proc. 37th International Conference on Machine Learning (eds. Iii, H. D. & Singh, A.) 3929–3938 (PMLR, 2020).
- Lewis, P., Perez, E., Piktus, A. & Petroni, F. Retrieval-augmented generation for knowledge-intensive nlp tasks. Adv. Neural Inform. Process. Syst. 33, 9459–9474 (2020).
- Ding, Y. et al. A Survey on RAG Meets LLMs: towards retrieval-augmented large language models. arXiv [cs.CL] (2024).
- Li T, Zhang G, Do QD, Yue X, Chen W. Long-context LLMs struggle with long in-context learning. arXiv [csCL]. 2024.
- He, J. et al. Never lost in the middle: improving large language models via attention strengthening question answering. arXiv [cs.CL] (2023).
- Wu, T., Zhao, Y. & Zheng, Z. An efficient recipe for long context extension via middle-focused positional encoding. In Adv. Neural Inf. Process. Syst. 38 (NeurIPS, 2024).
- Jiang, A. Q. et al. Mixtral of Experts. arXiv [cs.LG] (2024).
- Radford, A., Narasimhan, K., Salimans, T. & Sutskever, I. Improving language understanding by generative pre-training. OpenAI. Preprint, pp.1–12, (2018).
- Pal, A., Umapathi, L. K. & Sankarasubbu, M. MedMCQA: a large-scale multi-subject multi-choice dataset for medical domain question answering. In Proc. Conference on Health, Inference, and Learning (eds. Flores, G., Chen, G. H., Pollard, T., Ho, J. C. & Naumann, T.) 248–260 (PMLR, 2022).
- StatPearls Publishing. (n.d.). StatPearls. Retrieved September 5th, 2024, from https://www.ncbi.nlm.nih.gov/books/NBK430685/.
- Touvron, H. et al. Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv [cs.CL] (2023).
Cited by
Sorted by citations
Context Length Alone Hurts LLM Performance Despite Perfect Retrieval
Du, Y., Tian, M., Ronanki, S., Rongali, S., Bodapati, S. B., Galstyan, A., Wells, A., Schwartz, R., Huerta, E. A., & Peng, H. (2025) · Findings of the Association for Computational Linguistics: EMNLP 2025 · doi:10.18653/v1/2025.findings-emnlp.1264 · Cited by 39 · see citing works
Siam, K., Varela, A., Faruk, M. J. H., Cheng, J. Q., Gu, H., Maruf, A. A., & Aung, Z. (2025) · Scientific Reports · doi:10.1038/s41598-025-31010-4 · Cited by 28 · see citing works
Wong, H. S., & Wong, T. K. (2026) · JMIR Medical Informatics · doi:10.2196/82026 · Cited by 16 · see citing works
Scaling medical AI across clinical contexts
Li, M. M., Reis, B. Y., Rodman, A., Cai, T.-X., Dagan, N., Balicer, R., Loscalzo, J., Kohane, I., & Zitnik, M. (2026) · Nature Medicine · doi:10.1038/s41591-025-04184-7 · Cited by 20 · see citing works
Mansurova, A., Tleubayeva, A., Nugumanova, A., Shomanov, A., & Şeker, Ş. E. (2025) · Information · doi:10.3390/info16110943 · Cited by 18 · see citing works
Chen, C., Li, N., Qi, J., Chang, H., Shi, W., Xie, J., Yuan, J., Yang, H., Guo, J., Xu, C., et al. (2026) · Environmental Science & Technology · doi:10.1021/acs.est.5c09526 · Cited by 11 · see citing works
Esmaeilzadeh, P. (2025) · International Journal for Quality in Health Care · doi:10.1093/intqhc/mzaf134 · Cited by 9 · see citing works
CLEAR: A vision to support clinical evidence lifecycle with continuous learning
Fang, Y., Zhang, G., Chen, F., Hripcsak, G. M., Peng, Y., Ryan, P. B., & Weng, C. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104884 · Cited by 6 · see citing works
Self-citationClinical applications of large language models in knee osteoarthritis: a systematic review
Ma, Z., Liu, Y., Zhang, Z., Chen, R., Fan, H., Cao, X., & Ni, L. (2025) · Frontiers in Medicine · doi:10.3389/fmed.2025.1670824 · Cited by 8 · see citing works
Self-citationBlašković, L., Tanković, N., Lorencin, I., & Šegota, S. B. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc9100256 · Cited by 4 · see citing works
Hu, H., Wang, L., Chen, L., Lin, H., Huang, L.-T., Wang, Z.-W., & Hou, X. (2026) · Journal of Medicinal Chemistry · doi:10.1021/acs.jmedchem.5c03511 · Cited by 5 · see citing works
Lightweight query-adaptive RAG framework for knowledge support in smart manufacturing
Zhou, T., Liu, Y., Kumar, M., & Zou, L. (2026) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2026.04.016 · Cited by 5 · see citing works
Self-citationAl-Atraqchi, O. M. A., Abdulmuhsin, A. A., Rehman, S. U., & Mjlae, S. (2026) · VINE Journal of Information and Knowledge Management Systems · doi:10.1108/vjikms-08-2024-0309 · Cited by 7 · see citing works
Promises and challenges of applying large language models in the healthcare domain
Wang, Q., Gong, Z., Lai, Z., Bu, L., Dahlweid, F.-M., & Sun, H. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1772274 · Cited by 4 · see citing works
Retrieval‑augmented large language models for depression screening and suicide risk stratification
Xie, W., Song, X., Lu, Z., Fei, G., Zhou, Y., Ding, S., Zuo, F., Wu, B., Gu, T., Xu, L., et al. (2026) · BMC Psychiatry · doi:10.1186/s12888-026-07988-0 · Cited by 6 · see citing works
Self-citationAdam, T., Abosabie, S. A. S., Dittmer, M., Wolf, E., Abosabie, S., Behnke, C., Baier, F., Weickmann, A., Köser, L., Correll, C., et al. (2026) · Research Synthesis Methods · doi:10.1017/rsm.2026.10093 · Cited by 6 · see citing works
Kartal, A., Manalil, N. F., Cheng, C. D., Chung, L. K., Gebhard, H., Greenberg, M. D., Härtl, R., & Elsayed, G. A. (2025) · Global Spine Journal · doi:10.1177/21925682251411225 · Cited by 3 · see citing works
Beyond Fine-Tuning: Robust Food Entity Linking Under Ontology Drift with FoodOntoRAG
Drole, J., Gjorgjevikj, A., Seljak, B. K., & Eftimov, T. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11400993 · Cited by 3 · see citing works
In-context learning for label-efficient cancer image classification in oncology
Shrestha, M., Mandal, B., Mandal, V., Shrestha, A., & Shrestha, A. B. (2025) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2025.101683 · Cited by 3 · see citing works
Adejumo, P., Thangaraj, P., Dhingra, L. S., Biswas, D., Aminorroaya, A., Shankar, S. V., Pedroso, A. F., Croon, P. M., & Khera, R. (2024) · medRxiv · doi:10.1101/2024.09.19.24313992 · Cited by 3 · see citing works
Federated knowledge retrieval elevates large language model performance on biomedical benchmarks
Joy, J., & Su, A. I. (2026) · GigaScience · doi:10.1093/gigascience/giag007 · Cited by 3 · see citing works
Drug and Dosage Recommendation Based on Explainable Generative AI Using Patient-Specific Modeling
Dağlarlı, E. (2026) · IEEE Access · doi:10.1109/access.2026.3657398 · Cited by 2 · see citing works
Contradictions in Context: Challenges for Retrieval-Augmented Generation in Healthcare
Javadi, S., Mirabi, S., Gangar, M., & Ofoghi, B. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-21289-4_3 · Cited by 2 · see citing works
Du, X., Anna, N., Oates, M., Wang, Y., Wang, X., Plasek, J. M., Aronson, S., Lebo, M. S., & Zhou, L. (2026) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2026.106424 · Cited by 4 · see citing works
Zhao, Y., Miao, Y., Guo, R., Luo, Y., Wang, H., & Wu, Y. X. (2026) · Journal of Medical Internet Research · doi:10.2196/90046 · Cited by 2 · see citing works
Leveraging Large Language Models in Healthcare: From Speech Documentation to Conversational Agents
Winkler, C. (2026) · Springer eBooks · doi:10.1007/978-3-032-16837-5_16 · Cited by 2 · see citing works
Using large language models to generate child-friendly education materials on myopia
Li, X., Zhang, Y., Zheng, T., Deng, Y., Lu, Y., Hu, J., Chen, S., Li, Y., & Wang, K. (2025) · Digital Health · doi:10.1177/20552076251362338 · Cited by 1 · see citing works
SePA: A Search-enhanced Predictive Agent for Personalized Health Coaching
Ozolcer, M., & Bae, S. W. (2025) · IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI ...) · doi:10.1109/bhi67747.2025.11269521 · Cited by 1 · see citing works
Rajaram, P., Lane, M. E., Andalibi, N., Haimson, O. L., Hooper, R. C., & Prescher, H. (2025) · Annals of Plastic Surgery · doi:10.1097/sap.0000000000004608 · Cited by 1 · see citing works
Li, Y., Lv, M., Cheng, L., Xie, Y., Gu, Q., He, H., Chen, Z., Fang, D., & Zhou, X. (2026) · European Journal of Radiology · doi:10.1016/j.ejrad.2026.112734 · Cited by 1 · see citing works
Retrieval-Augmented Graph Neural Model for Medical Question Answering System
Ramachandiran, R., Mishra, M., Santhosh Kumar, G., Suchithra, B., Lalithamani, P., & Keerthana, N. V. (2025) · International Conference on Evolutionary Computing and Mobile Sustainable Networks (ICECMSN) · doi:10.1109/icecmsn68058.2025.11383543 · Cited by 1 · see citing works
Sharma, O., Sharma, B., Gupta, V., Bakshi, A., Gupta, R., Dada, T., & Sharma, T. (2026) · Indian Journal of Ophthalmology · doi:10.4103/ijo.ijo_2205_25 · Cited by 1 · see citing works
A neural-symbolic AI agent system for biomedical concept mapping
Zhang, G., Fang, Y., Chen, F., Ta, C., Hripcsak, G. M., Ryan, P., Peng, Y., & Weng, C. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02594-6 · Cited by 1 · see citing works
Self-citationYan, B., Zhang, R., Chen, L., Song, X. Z., Cao, Z., & Li, Y. (2026) · Journal of Medical Internet Research · doi:10.2196/96304 · Cited by 1 · see citing works
Yılmaz, A., Yuceyalcin, F., Varol, R., Gökyayla, E., Erdem, O., Choi, D., Demirçalı, A. A., Gençoğlan, G., Posma, J. M., & Temelkuran, B. (2025) · medRxiv · doi:10.1101/2025.05.17.25327785 · Cited by 2 · see citing works
Cheng, W. Q. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf236 · Cited by 1 · see citing works
Context is key: context engineering as the next frontier of medical AI
Wu, D. J., Chen, J. H., & Goh, E. (2026) · BMJ Digital Health & AI · doi:10.1136/bmjdh-2026-000032 · Cited by 1 · see citing works
McLachlan, A. D., Court, R., Pilgrim, C., Longden, K. D., Brown, N. H., Osumi-Sutherland, D., Jefferis, G. S. X. E., & Armstrong, J. D. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.06.16.732577 · Cited by 1 · see citing works
Guo, Z., Mao, H., & Zhang, F. (2026) · Publications · doi:10.3390/publications14020027
Yan, B.-C., Zhang, R., Fan, Y., Li, Y., Chen, L., Song, X., Tang, Y., Tu, Y., Cao, Z., Shen, L., et al. (2025) · Research Square · doi:10.21203/rs.3.rs-7998391/v1
Shao, M., & Zhang, H. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02146-4 · Cited by 1 · see citing works
Jin, X., Wang, Z., Yang, H., Li, M., & Du, Y. (2025) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm66473.2025.11356717
Improving Conversational Literature Retrieval Quality via Personalized Profile-Based Re-Ranking
Zhang, S., Shao, H., & Xie, Z. (2026) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2026.3669741
RAG Technology for Reliable Medical Retrieval and Hallucination Mitigation
Wu, M. (2025) · International Conference on Computer Communication, Networks and Information Science (CCNIS) · doi:10.1109/ccnis69465.2025.00009
Stifini, D., Penna, A. D., Mihaljevic, A., Missios, P., Bitzer, M., Eickhoff, C., & Capobianco, I. (2026) · Research Square · doi:10.21203/rs.3.rs-8849187/v1
Almahmoud, Z., Agarwal, V., Mahmoud, R., & Sastry, N. (2026) · Scientific Reports · doi:10.1038/s41598-026-43577-7
A Multimodal RAG and ArXiv-Integrated Conversational Framework for Automated Research Discovery
Dalvi, A., Sawant, S., Syed, A., Siddavatam, I. N. A., & Venkataramanan, V. (2026) · International Research Journal of Multidisciplinary Technovation · doi:10.54392/irjmt2626
Güler, I., Grieb, G., Kraus, A., & Stelling, H. (2026) · Aesthetic Surgery Journal Open Forum · doi:10.1093/asjof/ojag052
Li, Y., Yan, B.-C., Zhang, R., Song, X. Z., Chen, L., & Cao, Z. (2026) · doi:10.2196/preprints.96304
Xu, C., Zhu, X., & Yue, Y. (2026) · International Conference on Control Engineering and Artificial Intelligence (CCEAI) · doi:10.1145/3793928.3793947
From Answer to Audio: Visual Traceability for AI-Assisted Meeting Retrieval
Nakata, T., & Nakamura, M. (2026) · International Conference on Advanced Visual Interfaces (AVI) · doi:10.1145/3811427.3811496
Guideline-grounded Retrieval-augmented Generation for Ophthalmic Clinical Decision Support
Chen, S., Cui, S., & Cao, Z. (2026) · doi:10.1145/3811238.3811871
Xu, H., Liu, J., & Cheng, Q. (2026) · Applied Sciences · doi:10.3390/app16125827
Kara, S., & Karcıoğlu, A. A. (2026) · Neural Computing and Applications · doi:10.1007/s00521-026-12279-6
Rong, J., Liang, M., Wang, Z., Ye, Z., Luo, J., & Liang, Y. I. X. I. U. (2026) · Digital Health · doi:10.1177/20552076261464997
LLM-Based Applications for Healthcare: A Systematic Mapping Study
Malaquias, R. S., Filho, I. D. M. B., de Lima, J. M. M., Gurgel, A. M., & de Brito, B. A. O. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-30488-9_1
Dong, M., Liu, X., Luo, Y., Zeng, Y., Fei, F., & Du, L. C. (2026) · Research Integrity and Peer Review · doi:10.1186/s41073-026-00231-0
Find My Code Twin: Improving SNIPPET SEARCH Performance Using LLMs in Practice
Ko, S., Jang, E., Choi, D., Ryu, D., Park, J., & Park, C. (2026) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3786583.3786881
ClauseRoute: Risk-aware clause-guided routing for retrieval-augmented legal reasoning
Du, Z., Wu, C., Lu, J., Gao, Y., & Me, C. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.133633
Lehmann, S., Wilhelmy, F., Schwaebe, F. V., von Dercks, N., Güresir, E., & Wach, J. (2026) · Applied Clinical Informatics · doi:10.1055/a-2913-8450
Song, J., Xu, Z., Xiao, M., Bi, C., Zhang, Y., Zheng, X., Li, X., Cao, Q., Lu, Z., Yang, H., et al. (2026) · Journal of Medical Internet Research · doi:10.2196/101711 · Cited by 1 · see citing works
Self-citationSong, J., Makhdzir, N., Haron, Z., Qin, C., & Wang, L. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1902424 · Cited by 1 · see citing works
Enhanced Hybrid Retrieval-Augmented Model for Question Answering in High-Sensitivity Domains
Aloqla, A., Salama, R. M., Alghamdi, W., & Sen, A. A. A. (2026) · Future Internet · doi:10.3390/fi18080434
Evidence-Bound Generation for Medicines Information with Verifiable Claims and Visible Uncertainty
Ferraro, L., Ricci, M., (1560388), G. M., & Greco, P. (2026) · Archives of Pharmacy Practice · doi:10.51847/tzcg1ksqsz
An agentic AI workflow for spine history taking and surgeon-facing clinical synthesisPreprint
Kaplan, B. A., Kaplan, A., Aydilek, A., Çeli̇ker, Ö., Ege, T., Şentürk, S., & Solmaz, İ. (2026) · Research Square · doi:10.21203/rs.3.rs-10617202/v1
Riad, A. K. I., Barek, M. A., Shahriar, H., Francia, G., Ahamed, S. I., El-Sheikh, E., & Rawajfih, Y. (2026) · IEEE International Conference on Digital Health (ICDH) · doi:10.1109/icdh72779.2026.00012
Evidence-Informed Occupational Therapy Decision Support Using Graph Retrieval-Augmented Generation
Kutsuna, I., Tomiyama, N., Masuo, A., Yorozuya, K., Onda, S., & Hoshino, A. (2026) · OTJR Occupational Therapy Journal of Research · doi:10.1177/15394492261476597
Generative large language models in medicine: a scoping review of recent methodological advances
LI, F., Li, J., Cao, W., Li, H., Li, Y., Sun, Z., Chen, D., Duan, T., Li, P., Tizhoosh, H., et al. (2026) · npj Health Systems · doi:10.1038/s44401-026-00137-5
What clinicians should know about how large language models reason
Lyndon, S., & Benrimoh, D. A. (2026) · Nature Reviews Psychology · doi:10.1038/s44159-026-00617-3
Johnson, E., Smith, R. P., & Brown, L. (2024) · Journal of Health Informatics and Digital Systems · doi:10.68159/j563691456
Salman, A. H., Fico, G., & Iadanza, E. (2026) · World Congress on Medical Physics and Biomedical Engineering, September 7 - 12, 2009, Munich, Germany · doi:10.1007/978-3-032-20287-1_52
Vithanage, D., Duong, Q. V., Deng, C., Acharya, R., Gu, Z., Song, T., & Yu, P. (2026) · Journal of Healthcare Informatics Research · doi:10.1007/s41666-026-00254-8
Li, R., Zhao, J., Cao, S., Duprey, A., & Liu, R. (2026) · Research Square · doi:10.21203/rs.3.rs-10562519/v2
Li, R., Zhao, J., Cao, S., Duprey, A., & Liu, R. (2026) · Research Square · doi:10.21203/rs.3.rs-10990749/v1
Wang, M., Chen, L., Yin, M., Zhang, B., Wu, J., Sun, H., Yan, L., Wei, B., Zhang, F., Bi, C., et al. (2026) · Frontiers in Physiology · doi:10.3389/fphys.2026.1883295
Garcia, I., & Herrera, D. (2026) · Journal of Health Informatics and Digital Systems · doi:10.68159/q638866091
A pragmatic workflow for integrating large language models into toxicological risk assessment
Bassan, A., Amberg, A., Barreca, M. L., Bradley, P. M., Bringezu, F., Souza, J. D. P., Guicheney, C., Piparo, E. L., Kovarich, S., Massarsky, A., et al. (2026) · Frontiers in Toxicology · doi:10.3389/ftox.2026.1877128
An empirical study of knowledge modelling for healthcare chatbots: From RAG to GraphRAG
Lin, Y., Tan, T., Chen, X., Alghamdi, J., & Janssen, H. (2026) · Smart Health · doi:10.1016/j.smhl.2026.100726
Spolaore, A., Scholl, J., Gkampenis, A., Machetanz, K., Tatagiba, M., & Naros, G. (2026) · Machine Learning and Knowledge Extraction · doi:10.3390/make8100328
QSet-Rank: Dispersion-Aware Question-Set Reranking for Robust Biomedical RAG
Jember, A. F., & Bochicchio, M. A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-42224-8_20
AI -Augmented Patient Data Analysis for Research Driven Diagnostics
Rajathi, N., & Deeksha, R. (2026) · doi:10.1109/icsl-dsga70378.2026.11736716
Moya, M. O. C., Cabrera, L. H., & Ramos, N. P. (2026) · SAP Artificial Intelligence in Dentistry · doi:10.62486/aid2026158
- Authors
- Zhang, G., Xu, Z., Jin, Q., Chen, F., Fang, Y., Liu, Y., Rousseau, J. F., Xu, Z., Lu, Z., Weng, C., & Peng, Y.
- Affiliations
- Columbia University (US)
- Weill Cornell Medicine (US)
- National Institutes of Health (US)
- The University of Texas Southwestern Medical Center (US)
- Corresponding author
- Weng, Chunhua
- Peng, Yifan
- ORCID
- Venue
- npj Digital Medicine
- Published
- 2025-05-02
- Volume
- 8
- Issue
- 1
- Pages
- 239-239
- ISSN
- 2398-6352 (ISSN-L)
- PMC
- PMC12048518
- OpenAlex
- W4410030662
- Cited by
- 81
- References
- 39
- Citations by year
- 2026: 60 · 2025: 19 · 2024: 2
- FWCI
- 31.69
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- npj Digital Medicine (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- U.S. National Library of Medicine (R01LM009886, R01LM014344, NIH Intramural Research Program)
- U.S. National Center for Advancing Clinical and Translational Science (UL1TR001873, UL1TR002384)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- retrieval-augmented generation, medical question answering, long context, hallucinations, medical literature, clinical decision support, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
