SKiM-GPT: combining biomedical literature-based discovery with large language model hypothesis evaluation

Existing LBD methods and tools primarily quantify the strength of association without assessing relevance to or support of a specific scientific hypothesis.

“However, while existing LBD methods can efficiently identify statistically significant associations between biomedical terms, they generally lack a structured framework for evaluating whether these discovered associations meaningfully support or contradict a specific scientific hypothesis.”

• Also stated in: § Background · plain

  “Tools such as Arrowsmith [34], Lion-LBD [27], BITOLA [15], and our tool Serial KinderMiner (SKiM) [22] primarily quantify the strength of association without assessing relevance to or support of a hypothesis.”

• cites [34] Torvik VI, Smalheiser NR. A quantitative model for linking two disparate sets of articles in MEDLINE. Bioinformatics. 2007;23(13):1658–65. https://doi.org/10.1093/bioinformatics/btm161 sourcePartly supported

  The Introduction says earlier approaches lacked a quantitative model for predicting which linking terms would be relevant to a given search. The Discussion says the paper introduces such a model, while the scientist ultimately decides whether a link is meaningful. These passages support the distinction between finding links and assessing relevance, but they do not establish that LBD methods generally lack a framework for evaluating whether an association supports or contradicts a specific hypothesis. (LLM, conf=0.86)

  Experimental — accuracy not yet measured
• cites [27] Pyysalo S, Baker S, Ali I, Haselwimmer S, Shah T, Young A, et al. LION LBD: A literature-based discovery system for cancer biology. Bioinformatics. 2019;35(9):1553–61. https://doi.org/10.1093/bioinformatics/bty845 sourcePartly supported

  Section 3.3 describes LBD scores based on co-occurrence and statistical association, and Section 4.3 says the system can suggest unverifiable or causally invalid connections. These passages support the concern that association scores alone do not establish meaningful support for a hypothesis. The paper does not establish the broader claim that existing LBD methods generally lack a structured framework for assessing whether evidence supports or contradicts a specific hypothesis; its Introduction also describes closed discovery as hypothesis testing. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites [15] Hristovski D, Friedman C, Rindflesch TC, Peterlin B. Exploiting semantic relations for literature-based discovery. AMIA annual symposium proceedings. 2006;2006:349–53.Not checked — no full text of the source

SKiM-GPT's three-stage workflow discards many spurious associations.

“This three-stage workflow discards many of the spurious associations as documented in earlier LBD studies [13]”

• cites [13] Gopalakrishnan V, Jha K, Jin W, Zhang A. A survey on literature based discovery approaches in biomedical domain. J Biomed Inform. 2019;93:103141. https://doi.org/10.1016/j.jbi.2019.103141. sourcePartly supported

  Section 4.2 says semantic-relation approaches reduce false positives, and the Bio-SbKDS evaluation reports fewer, more relevant connections. These findings support the general rationale for filtering spurious LBD associations, but the survey does not evaluate SKiM-GPT or establish that its particular three-stage workflow discards many of them. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

A dedicated relevance filtering step is essential for RAG-based systems to ensure only pertinent information is included in the LLM's context.

“Therefore, a dedicated relevance filtering step is essential for RAG-based systems [6, 39], ensuring that only information directly pertinent to the hypothesis at hand is included in the LLM’s context.”

• cites [6] Chen J, Lin H, Han X, Sun L. Benchmarking Large Language Models in Retrieval-Augmented Generation. Proceedings of the AAAI Conference on Artificial Intelligence 2024;38(16):17754–62. https://doi.org/10.1609/aaai.v38i16.29728 sourcePartly supported

  The Required Abilities of RAG section says retrieved documents can contain noise, and the Data Construction section describes selecting text chunks that best match a query. The paper does not establish that a dedicated relevance-filtering step is essential or that it can ensure only information directly pertinent to a hypothesis enters the LLM’s context. Page numbers were not available in our copy. (LLM, conf=0.86)

  Experimental — accuracy not yet measured
• cites [39] Yu Y, Ping W, Liu Z, Wang B, You J, Zhang C, et al. RankRAG: Unifying context ranking with retrieval-augmented generation in LLMs. Advances in Neural Information Processing Systems. 2024;37:121156–84. sourcePartly supported

  The Introduction (§1) says a mechanism is needed to preserve relevant content when selecting a small set of contexts, and describes RankRAG reranking retrieved contexts before generation. This supports a relevance-filtering step, but the paper does not establish that such a step is essential for every RAG system or that it ensures only information pertinent to a hypothesis enters the context. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

LLMs have a propensity to generate incorrect or fabricated information ('hallucinations').

“However, a known challenge with LLMs is their propensity to generate incorrect or fabricated information (“hallucinations”) [21].”

• cites [21] Maynez J, Narayan S, Bohnet B, McDonald R. On faithfulness and factuality in abstractive summarization. arXiv preprint. 2020;arXiv:2005.00661. https://doi.org/10.48550/arXiv.2005.00661. sourcePartly supported

  The BERTS2S conclusions report frequent hallucinations in generated summaries and find that over 90% of extrinsic hallucinations assessed were erroneous. The paper studies abstractive summarization systems, however, rather than establishing a general propensity of LLMs across tasks. Section 5.3 also cautions that hallucinations are not necessarily erroneous. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

Earlier tools such as KinderMiner and SKiM excel at finding associations between entities by surfacing statistically enriched term pairs.

“While earlier tools such as KinderMiner and SKiM excel at finding associations between entities by surfacing statistically enriched term pairs [19, 22]”

• cites [19] Kuusisto F, Steill J, Kuang Z, Thomson J, Page D, Stewart R. A simple text mining approach for ranking pairwise associations in biomedical applications. AMIA summits on translational science proceedings. 2017;2017:166.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [22] Millikin RJ, Raja K, Steill J, Lock C, Tu X, Ross I, et al. Serial KinderMiner (SKiM) discovers and annotates biomedical knowledge using co-occurrence and transformer models. BMC Bioinformatics. 2023;24(1). https://doi.org/10.1186/s12859-023-05539-y sourceSupported

Retrieving irrelevant, ambiguous, or weakly related documents can introduce noise and misleading context, increasing the risk of inaccuracies and hallucinations.

“Retrieving documents that are irrelevant, ambiguous, or only weakly related to the hypothesis can introduce noise and misleading context, increasing the risk of inaccuracies and hallucinations [16, 23].”

• cites [16] Hsieh CP, Sun S, Kriman S, Acharya S, Rekesh D, Jia F, et al. RULER: What’s the real context size of your long-context language models? arXiv preprint. 2024;arXiv:2404.06654 https://doi.org/10.48550/arXiv.2404.06654 sourcePartly supported

  In §5 Task Error Analysis (pp. 7, 9), the paper reports that distractors lead to incorrect retrieval and that adding distracting paragraphs to long-context QA increases hallucination and reduces reliance on relevant context. It does not specifically study documents retrieved for a hypothesis or distinguish ambiguous and weakly related documents, so the citing sentence extends the finding beyond the tested setting. (LLM, conf=0.85)

  Experimental — accuracy not yet measured
• cites [23] Modarressi A, Deilamsalehy H, Dernoncourt F, Bui T, Rossi R, Yoon S, et al. NoLiMa: Long-context evaluation beyond literal matching. arXiv preprint. 2025;arXiv:2502.05167. https://doi.org/10.48550/arXiv.2502.05167 sourcePartly supported

  The ablation study (Section 4.4.4, p. 8) finds that irrelevant, query-overlapping distractors mislead models and reduce performance. The Conclusion (p. 9) suggests a similar difficulty when such documents appear in search or RAG results. The paper does not establish the claim for ambiguous or weakly related documents generally, or specifically report hallucinations. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Without relevance filtering, SKiM-GPT showed an ICC(2,1) of 0.91 with human evaluators.

“Without relevance filtering, SKiM-GPT showed substantial agreement with human evaluators (Fig. 3A; root mean square error (RMSE) = 0.57, quadratic-weighted Cohen’s κ (QWK) = 0.84 [11], intraclass correlation coefficient (ICC (2,1) = 0.91 [18].”

• cites [18] Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. 2016;15(2):155. https://doi.org/10.1016/j.jcm.2016.02.012. sourceSupported

On a benchmark of 14 disease-gene-drug hypotheses, SKiM-GPT achieves strong ordinal agreement with four expert biologists (Cohen's κ = 0.84).

“On a benchmark consisting of 14 disease-gene-drug hypotheses, SKiM-GPT achieves strong ordinal agreement with four expert biologists (Cohen’s κ = 0.84), demonstrating its ability to replicate expert judgment.”

• Also stated in: § Score alignment evaluation · plain

  “Without relevance filtering, SKiM-GPT showed substantial agreement with human evaluators (Fig. 3A; root mean square error (RMSE) = 0.57, quadratic-weighted Cohen’s κ (QWK) = 0.84 [11], intraclass correlation coefficient (ICC (2,1) = 0.91 [18].”

• cites [11] Fleiss JL, and Cohen J. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educ Psychol Meas. 1973;33(3):613–19. https://doi.org/10.1177/001316447303300309. sourceNot checked — no full text of the source
• cites [18] Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. 2016;15(2):155. https://doi.org/10.1016/j.jcm.2016.02.012. sourceCould not verify

  The Introduction’s case description concerns a hypothetical scoliosis test–retest study using ICC, and the Conclusion discusses guidance for selecting and interpreting ICCs. Searches for SKiM-GPT, the 14 disease-gene-drug hypotheses, expert biologists, and Cohen’s κ found no passage addressing the claimed benchmark or κ = 0.84. (LLM, conf=0.78)

  Experimental — accuracy not yet measured

Researchers face a significant burden of manually interpreting potentially thousands of LBD associations, many of which are irrelevant or spurious.

“Consequently, researchers still face a significant burden of manually interpreting potentially thousands of associations, many of which are irrelevant or spurious [26].”

• cites [26] Preiss J, Stevenson M. Quantifying and filtering knowledge generated by literature based discovery. BMC Bioinformatics. 2017;18(S7). https://doi.org/10.1186/s12859-017-1641-9. sourceSupported

Without relevance filtering, SKiM-GPT showed an RMSE of 0.57 with human evaluators.

“Without relevance filtering, SKiM-GPT showed substantial agreement with human evaluators (Fig. 3A; root mean square error (RMSE) = 0.57, quadratic-weighted Cohen’s κ (QWK) = 0.84 [11], intraclass correlation coefficient (ICC (2,1) = 0.91 [18].”

The fine-tuned model misclassified only two relevant abstracts as irrelevant.

“The fine-tuned model misclassified only two relevant abstracts as irrelevant, reflecting a deliberate design choice to favor a low false-negative rate so that nearly all potentially useful evidence is retained for downstream hypothesis evaluation.”

When SKiM-GPT evaluates a positive hypothesis about an FDA-approved breast cancer drug improving patient outcomes without any retrieved abstracts, it produces strongly positive scores.

“When the positive hypothesis was evaluated without any retrieved abstracts, SKiM-GPT produced strongly positive scores, confirming expected leakage from the training data (Fig. 5B).”

• Also stated in: § Retrieved text vs. memory · boosted

  “Without any provided abstracts (“No text”), when asked whether each drug “improves breast-cancer patient outcomes,” SKiM-GPT assigns positive scores (median = + 2)”

When using the five most cited real abstracts, SKiM-GPT's positive scores correctly agree with the positive hypothesis.

“When using the five most cited real abstracts, the positive scores indicated correct agreement with the positive hypothesis (Fig. 5B).”

The relevance filtering step removes irrelevant or contextually ambiguous abstracts retrieved by SKiM.

“This filtering step removes irrelevant or contextually ambiguous abstracts retrieved by SKiM (e.g., “AHR” can refer to the “aryl hydrocarbon receptor” gene or the “adjusted hazard ratio” measure).”

Without relevance filtering, directional concordance between SKiM-GPT's median score and human median score occurred in 13 out of 14 hypotheses (93%).

“Directional concordance between SKiM-GPT’s median score and human median score occurred in 13 out of 14 hypotheses (93%), and the median SKiM-GPT score fell within the human score range in 12 of 14 cases (86%).”

The model achieved precision 0.90 and recall 1.0 on out-of-training data.

“The model’s precision 0.90 and recall 1.0 on this out-of-training data indicates the ability to generalize across biomedical domains.”

SKiM-GPT strongly supports hypotheses when provided with synthetic abstracts whose sentiment matches the hypothesis (neutral, negative, or positive).

“SKiM-GPT strongly supported neutral hypotheses when provided with neutral abstracts (Fig. 5C), negative hypotheses when supplied with negative abstracts (Fig. 5D), and positive hypotheses when provided with positive abstracts (Fig. 5B)”

• Also stated in: § Retrieved text vs. memory · plain

  “Feeding synthetic abstracts that state no relationship exists between the drug and breast cancer patient outcomes increases scores for neutral hypotheses (median = + 2) for all drugs”

• Also stated in: § Retrieved text vs. memory · plain

  “Providing synthetic abstracts that refute drug efficacy results in increased scores for negative hypotheses (median = + 2).”

With relevance filtering, SKiM-GPT showed an ICC(2,1) of 0.91 with human evaluators.

“With relevance filtering, agreement between SKiM-GPT and human evaluators remained high (Fig. 3B; RMSE = 0.64, QWK = 0.84, ICC (2,1) = 0.91), with directional concordance of 100% (14/14) and median scores falling within the human evaluators’ range in 12 of 14 cases (86%).”

The one misclassification by SKiM-GPT involved the gene name SLN because no relevant abstracts were returned due to the abbreviation SLN also standing for sentinel lymph node.

“In the one case where SKiM-GPT was unable to correctly classify, the gene name SLN returned no relevant abstracts, as the abbreviation SLN can also stand for “sentinel lymph node” which is often a term associated with cancers.”

SKiM-GPT is a RAG system that combines SKiM's co-occurrence search and retrieval with frontier LLMs to evaluate user-defined hypotheses.

“We introduce SKiM-GPT, a retrieval-augmented generation (RAG) system that combines SKiM’s co-occurrence search and retrieval with frontier LLMs to evaluate user-defined hypotheses.”

With relevance filtering, SKiM-GPT showed an RMSE of 0.64 with human evaluators.

“With relevance filtering, agreement between SKiM-GPT and human evaluators remained high (Fig. 3B; RMSE = 0.64, QWK = 0.84, ICC (2,1) = 0.91), with directional concordance of 100% (14/14) and median scores falling within the human evaluators’ range in 12 of 14 cases (86%).”

Without relevance filtering, the median SKiM-GPT score fell within the human score range in 12 of 14 cases (86%).

“Directional concordance between SKiM-GPT’s median score and human median score occurred in 13 out of 14 hypotheses (93%), and the median SKiM-GPT score fell within the human score range in 12 of 14 cases (86%).”

SKiM-GPT produces quantitative and qualitative support that aligns with expert judgement.

“produces quantitative and qualitative support that aligns with expert judgement.”

• AssumptionRests on an assumption: The human-assigned labels (true-positive/true-negative) based on current clinical or mechanistic evidence are correct.

With relevance filtering, median SKiM-GPT scores fell within the human evaluators' range in 12 of 14 cases (86%).

“With relevance filtering, agreement between SKiM-GPT and human evaluators remained high (Fig. 3B; RMSE = 0.64, QWK = 0.84, ICC (2,1) = 0.91), with directional concordance of 100% (14/14) and median scores falling within the human evaluators’ range in 12 of 14 cases (86%).”

With relevance filtering, directional concordance between SKiM-GPT and human evaluators was 100% (14/14).

“With relevance filtering, agreement between SKiM-GPT and human evaluators remained high (Fig. 3B; RMSE = 0.64, QWK = 0.84, ICC (2,1) = 0.91), with directional concordance of 100% (14/14) and median scores falling within the human evaluators’ range in 12 of 14 cases (86%).”

Relevance filtering removes false-positive abstracts caused by semantic ambiguity.

“In these cases, false-positive abstracts are removed via relevance filtering.”

The fine-tuned model was designed to favor a low false-negative rate so that nearly all potentially useful evidence is retained for downstream hypothesis evaluation.

“The fine-tuned model misclassified only two relevant abstracts as irrelevant, reflecting a deliberate design choice to favor a low false-negative rate so that nearly all potentially useful evidence is retained for downstream hypothesis evaluation.”

Publication bias (negative or inconclusive results being less likely to be published) may skew SKiM-GPT's scores, especially when few abstracts are available to score a hypothesis.

“Additionally, negative or inconclusive results are less likely to be published, and these unpublished results would be invisible to SKiM-GPT; this may skew SKiM-GPT’s scores, especially in the case where few abstracts are available to score a hypothesis.”

The effectiveness of RAG depends on the quality and relevance of the retrieved documents.

“Still, the effectiveness of RAG depends on the quality and relevance of the retrieved documents.”

SKiM-GPT reduces the workload on LBD tool users.

“reducing the workload on LBD tool users.”

• AssumptionRests on an assumption: Evaluating and ranking hits comprises a large percentage of time and effort when conducting a SKiM search.

SKiM-GPT's scores mirror the sentiment of the provided abstracts rather than the LLM's training data.

“Together, panels B–D indicate that SKiM-GPT’s scores mirror the sentiment of the abstracts rather than the LLM’s training data”

Reducing the burden of evaluating and ranking hits is now feasible with state-of-the-art language models.

“reducing this burden is now feasible with state-of-the-art language models.”

• AssumptionRests on an assumption: Evaluating and ranking hits comprises a large percentage of time and effort when conducting a SKiM search.

The model can generalize across biomedical domains.

“The model’s precision 0.90 and recall 1.0 on this out-of-training data indicates the ability to generalize across biomedical domains.”

SKiM-GPT grounds its analysis in the provided texts even when its parametric memory conflicts with those texts.

“indicating that SKiM-GPT grounds its analysis in the provided texts even in cases where its parametric memory conflicts with those texts (Fig. 5C and D).”

SKiM-GPT helps move the LBD field from only automatically finding associations towards automatically evaluating them.

“helping to move from only automatically finding associations towards automatically evaluating them”

SKiM-GPT couples high-recall document retrieval with context-aware LLM reasoning to surface plausible mechanistic relationships from biomedical literature.

“SKiM‑GPT couples high‑recall document retrieval with context‑aware LLM reasoning to surface plausible mechanistic relationships hidden in an ever‑growing corpus of biomedical literature”

The strongly positive scores produced by SKiM-GPT without retrieved abstracts indicate parametric-knowledge leakage from its training data.

“When the positive hypothesis was evaluated without any retrieved abstracts, SKiM-GPT produced strongly positive scores, confirming expected leakage from the training data (Fig. 5B).”