Cited by 105Bioinformatics and Genomic NetworksBiomedical Text Mining and OntologiesGenomics and Phylogenetic Studies

Evaluation of large language models for discovery of gene set function

Open access
Hu, Mengzhou, Alkhairy, Sahar, Lee, Ingoo, Pillich, Rudolf T., Fong, Dylan, Smith, Kevin, Bachelder, Robin, Ideker, Trey +1 more·2025·Nature Methods

Published version of a preprint

Abstract

Gene set enrichment is a mainstay of functional genomics, but it relies on gene function databases that are incomplete. Here we evaluate five large language models (LLMs) for their ability to discover the common functions represented by a gene set, supported by molecular rationale and a self-confidence assessment. For curated gene sets from Gene Ontology, GPT-4 suggests functions similar to the curated name in 73% of cases, with higher self-confidence predicting higher similarity. Conversely, random gene sets correctly yield zero confidence in 87% of cases. Other LLMs (GPT-3.5, Gemini Pro, Mixtral Instruct and Llama2 70b) vary in function recovery but are falsely confident for random sets. In gene clusters from omics data, GPT-4 identifies common functions for 45% of cases, fewer than functional enrichment but with higher specificity and gene coverage. Manual review of supporting rationale and citations finds these functions are largely verifiable. These results position LLMs as valuable omics assistants.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41592-024-02525-x

Cites

88 of 88 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Text embeddings integrate omics data with various biological entities such as pathways, functional annotations and cell types, extending the model's capabilities beyond tissue alignment and decomposition, making it adaptable to a broader range of biological tasks.

    “First, text embeddings integrate omics data with various biological entities such as pathways, functional annotations68 and cell types69, extending the model’s capabilities beyond tissue alignment and decomposition, making it adaptable to a broader range of biological tasks.”

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

1 retracted

Flagged first, then by citations

2 of the 90 references couldn’t be matched to a known paper.

Research works printed but not linked: 3

  1. [31]Devlin J, Chang M-W, Lee K & Toutanova K BERT: pre-training of deep bidirectional transformers for language understanding. In Proc. 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (eds Burstein J.) 4171– 4186 (Association for Computational Linguistics, 2019).
  2. [66]Li G, Al Kader Hammoud HA, Itani H, Khizbullin D & Ghanem B CAMEL: Communicative Agents for ‘Mind’ Exploration of Large Scale Language Model Society. In Advances in Neural Information Processing Systems (eds Oh A) 36, 51991– 52008 (NeurIPS, 2023).
  3. [70]Holtzman A, Buys J, Du L, Forbes M & Choi Y The curious case of neural text degeneration. In The Eighth International Conference on Learning Representations (ICLR, 2020).

Cited by

Sorted by citations

Authors
Hu, M., Alkhairy, S., Lee, I., Pillich, R. T., Fong, D., Smith, K., Bachelder, R., Ideker, T., & Pratt, D.
Affiliations
University of California San Diego (US)
Corresponding author
  • Ideker, Trey
  • Pratt, Dexter
Venue
Nature Methods
Published
2024-11-28
Volume
22
Issue
1
Pages
82-91
ISSN
  • 1548-7091 (ISSN-L)
  • 1548-7105
OpenAlex
W4404824622
Cited by
104
References
90
Citations by year
2026: 43 · 2025: 52 · 2024: 8 · 2023: 1
FWCI
13.98
Type
journal article
Language
en
Open access
Locations
  • Nature Methods (publishedVersion)
  • Research Square (acceptedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • eScholarship (California Digital Library) (submittedVersion)
  • PubMed Central (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • U.S. Department of Health & Human Services | National Institutes of Health (u24ca269436, u01mh115747, u24hg12107, ot2od023742)
  • Schmidt Futures
  • Foundation for the National Institutes of Health (u24ca269436)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Bioinformatics and Genomic Networks
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
Keywords
large language models, gene set enrichment analysis, gene function prediction, Gene Ontology, GPT-4, omics data, gene clusters, functional genomics, LLM evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review