The DOI matches its Crossref record; 87 of the 90 references it cites were found in a registry or an index.
Evaluation of large language models for discovery of gene set function
Open accessPublished version of a preprint
Abstract
Gene set enrichment is a mainstay of functional genomics, but it relies on gene function databases that are incomplete. Here we evaluate five large language models (LLMs) for their ability to discover the common functions represented by a gene set, supported by molecular rationale and a self-confidence assessment. For curated gene sets from Gene Ontology, GPT-4 suggests functions similar to the curated name in 73% of cases, with higher self-confidence predicting higher similarity. Conversely, random gene sets correctly yield zero confidence in 87% of cases. Other LLMs (GPT-3.5, Gemini Pro, Mixtral Instruct and Llama2 70b) vary in function recovery but are falsely confident for random sets. In gene clusters from omics data, GPT-4 identifies common functions for 45% of cases, fewer than functional enrichment but with higher specificity and gene coverage. Manual review of supporting rationale and citations finds these functions are largely verifiable. These results position LLMs as valuable omics assistants.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“First, text embeddings integrate omics data with various biological entities such as pathways, functional annotations68 and cell types69, extending the model’s capabilities beyond tissue alignment and decomposition, making it adaptable to a broader range of biological tasks.”
From A visual–omics foundation model to bridge histopathology with spatial transcriptomics (2025) · cites this paper as [68]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
1 retracted
Zhang, X., Zhao, X., Fiskus, W. C., Lin, J., Lwin, T., Rao, R., Zhang, Y., Chan, J. C., Fu, K., Márquez, V. E., et al. (2012) · Cancer Cell · doi:10.1016/j.ccr.2012.09.003 · Cited by 278
RetractedVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Subramanian, A., Tamayo, P., Mootha, V. K., Mukherjee, S., Ebert, B. L., Gillette, M. A., Paulovich, A., Pomeroy, S. L., Golub, T. R., Lander, E. S., et al. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0506580102 · Cited by 53,625
Gene Ontology: tool for the unification of biology
Ashburner, M., Ball, C. A., Blake, J. A., Botstein, D., Butler, H., Cherry, J. M., Davis, A. P., Dolinski, K., Dwight, S. S., Eppig, J. T., et al. (2000) · Nature Genetics · doi:10.1038/75556 · Cited by 43,206
KEGG: Kyoto Encyclopedia of Genes and Genomes
Kanehisa, M. (2000) · Nucleic Acids Research · doi:10.1093/nar/28.1.27 · Cited by 37,077
Systematic and integrative analysis of large gene lists using DAVID bioinformatics resources
Huang, D. W., Sherman, B. T., & Lempicki, R. A. (2009) · Nature Protocols · doi:10.1038/nprot.2008.211 · Cited by 35,890
KEGG: Kyoto Encyclopedia of Genes and Genomes
Ogata, H., Goto, S., Sato, K., Fujibuchi, W., Bono, H., & Kanehisa, M. (1999) · Nucleic Acids Research · doi:10.1093/nar/27.1.29 · Cited by 31,232
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
NCBI GEO: archive for functional genomics data sets—update
Barrett, T., Wilhite, S. E., Ledoux, P., Evangelista, C., Kim, I. F., Tomashevsky, M., Marshall, K. A., Phillippy, K. H., Sherman, P. M., Holko, M., et al. (2012) · Nucleic Acids Research · doi:10.1093/nar/gks1193 · Cited by 10,418
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
Research works printed but not linked: 3
- Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. BERT: pre-training of deep bidirectional transformers for language understanding. In Proc. 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (eds Burstein, J. et al.) 4171–4186 (Association for Computational Linguistics, 2019).
- Li, G., Al Kader Hammoud, H. A., Itani, H., Khizbullin, D. & Ghanem, B. CAMEL: Communicative Agents for ‘Mind’ Exploration of Large Scale Language Model Society. In Advances in Neural Information Processing Systems (eds Oh, A. et al.) 36, 51991–52008 (NeurIPS, 2023).
- Holtzman, A., Buys, J., Du, L., Forbes, M. & Choi, Y. The curious case of neural text degeneration. In The Eighth International Conference on Learning Representations (ICLR, 2020).
Cited by
Sorted by citations
The Gene Ontology knowledgebase in 2026
Aleksander, S., Balhoff, J. P., Carbon, S., Cherry, J. M., Ebert, D., Feuermann, M., Gaudet, P., Harris, N. L., Hill, D. P., Kalita, P., et al. (2025) · Nucleic Acids Research · doi:10.1093/nar/gkaf1292 · Cited by 281
Biomni: A general-purpose biomedical ai agent
Huang, K., Zhang, S., Wang, H., Qu, Y., Lu, Y., Roohani, Y., Li, R., Qiu, L., Li, G., Zhang, J., et al. (2025) · doi:10.1101/2025.05.30.656746 · Cited by 125
“Previous approaches have largely relied on specialist agentic workflows tailored to nar-47 row biomedical tasks 13–19 , which restricts their capacity to move fluidly and generalize across the full spectrum of biomedical domains, as needed to answer key research questions.”
A visual–omics foundation model to bridge histopathology with spatial transcriptomics
Chen, W., Zhang, P., Tran, T. N., Xiao, Y., Li, S., Shah, V. V., Cheng, H., Brannan, K. W., Youker, K., Lai, L., et al. (2025) · Nature Methods · doi:10.1038/s41592-025-02707-1 · Cited by 118
“First, text embeddings integrate omics data with various biological entities such as pathways, functional annotations 68 and cell types 69 , extending the model’s capabilities beyond tissue alignment and decomposition, making it adaptable to a broader range of biological tasks.”
GeneAgent: self-verification language agent for gene-set analysis using domain databases
Wang, Z., Jin, Q., Wei, C.-H., Tian, S., Lai, P., Zhu, Q., Day, C., Ross, C., Leaman, R., & Lu, Z. (2025) · Nature Methods · doi:10.1038/s41592-025-02748-6 · Cited by 114
Kim, J., Wang, K., Weng, C., & Liu, C. (2024) · The American Journal of Human Genetics · doi:10.1016/j.ajhg.2024.08.010 · Cited by 53
Metadata
- Authors
- Hu, M., Alkhairy, S., Lee, I., Pillich, R. T., Fong, D., Smith, K., Bachelder, R., Ideker, T., & Pratt, D.
- Affiliations
- University of California San Diego (US)
- Corresponding author
- Ideker, Trey
- Pratt, Dexter
- ORCID
- Venue
- Nature Methods
- Published
- 2024-11-28
- Volume
- 22
- Issue
- 1
- Pages
- 82-91
- ISSN
- 1548-7091 (ISSN-L)
- 1548-7105
- PMC
- PMC11725441
- OpenAlex
- W4404824622
- Cited by
- 104
- References
- 90
- Citations by year
- 2026: 43 · 2025: 52 · 2024: 8
- FWCI
- 13.98
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Licence
- Locations
- Nature Methods (publishedVersion)
- Research Square (acceptedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- eScholarship (California Digital Library) (submittedVersion)
- PubMed Central (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- U.S. Department of Health & Human Services | National Institutes of Health (u24ca269436, u01mh115747, u24hg12107, ot2od023742)
- Schmidt Futures
- Foundation for the National Institutes of Health (u24ca269436)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Bioinformatics and Genomic Networks
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
- Keywords
- large language models, gene set enrichment analysis, gene function prediction, Gene Ontology, GPT-4, omics data, gene clusters, functional genomics, LLM evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
