Genomic language model predicts protein co-regulation and function
Open accessPublished version of a preprint
The DOI matches its Crossref record; 76 of the 80 references it cites were found in a registry or an index.
Abstract
Deciphering the relationship between a gene and its genomic context is fundamental to understanding and engineering biological systems. Machine learning has shown promise in learning latent relationships underlying the sequence-structure-function paradigm from massive protein sequence datasets. However, to date, limited attempts have been made in extending this continuum to include higher order genomic context information. Evolutionary processes dictate the specificity of genomic contexts in which a gene is found across phylogenetic distances, and these emergent genomic patterns can be leveraged to uncover functional relationships between gene products. Here, we train a genomic language model (gLM) on millions of metagenomic scaffolds to learn the latent functional and regulatory relationships between genes. gLM learns contextualized protein embeddings that capture the genomic context as well as the protein sequence itself, and encode biologically meaningful and functionally relevant information (e.g. enzymatic function, taxonomy). Our analysis of the attention patterns demonstrates that gLM is learning co-regulated functional modules (i.e. operons). Our findings illustrate that gLM’s unsupervised deep learning of the metagenomic corpus is an effective and promising approach to encode functional semantics and regulatory syntax of genes in their genomic contexts and uncover complex relationships between genes in a genomic region.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures

![Fig. 2 Contextualized protein embedding analysis and comparison with concepts in natural language modeling. A A word upon contextualization can be mapped to embedding space. For many words, the semantic meaning varies in different types of literature, and therefore their contextualized embeddings cluster with source text type. Figure was created for qualitative visualization. B The input protein embedding (output of ESM2 and context-free protein embedding) is the same across all occurrences of the protein in the database. Upon contextualization with gLM, contextualized protein embeddings of the same protein (last hidden layer of gLM at inference time) cluster with biome type, analogous to the source text type in natural language (A). Contextualization of 30 other multi-biome MGYPs can be found in Supplementary Fig. 3. C A word’s meaning upon contextualization varies across a continuous spectrum and can be ambiguous even with contextualization (e.g. double entendre). D Reaction 1, carried out by the MCR complex, either backward (Methanotrophy) or forward (Methanogenesis). E Principal Component Analysis (PCA) of context-free protein embeddings of McrA sequences in genomes (total explained variances = 0.56), colored by metabolic classification of the organism (ANME, methanogen) based on previous studies and labeled by class-level taxonomy. F PCA of contextualized McrA embeddings (total explained variance = 0.68), where gLM embeddings cluster with the direction of Reaction 1 that the MCR complex is likely to carry out. G Geometric relationship between contextualized protein embeddings based on the semantic closeness of words. H Input (context-free) protein embeddings of Cas1, Cas2, lipopolysaccharide synthases (LPS) and polyketide synthases (PKS) showing clustering based on structural and sequence similarity. I Clustering of contextualized protein embeddings where phage defense proteins cluster (Cas1 and Cas2) and biosynthetic gene products cluster (lipopolysaccharide synthases [LPS] and polyketide synthases [PKS]). Source data are provided as a Source Data file.](https://figures.cochat.ai/paper-figures/4a1f2e455bc049e6a0a4a8ded2d06fcc/genomic-language-model-predicts-protein-co-regulation-and-function-figure-2.jpg)

![Fig. 4 Attention analysis. A Correlation coefficients (Pearson’s rho) between attention heads across layers and operons. Darker color corresponds to stronger correlation with previously identified operons. Attention patterns of the second layer-seventh head [L2-H7] is most strongly correlated with the operons. B Three random examples of contigs and predicted operonic relationship between neighboring proteins. Proteins are listed in the order they are encoded in the contig. Ground truth E.coli K-12 operons (top row), raw attention scores in the attention head [L2-H7] most correlated with operons (middle row) and logistic regression prediction using all attention heads (last row) where false positive predictions (or possibly misannotated ground truths in the case of flagellar proteins in the first example) are marked in red. C Five-fold cross-validation precision-recall curves of logistic regression trained using all operons and attention heads. D AAA+ regulator associations characterized using attention-based prediction of operons (Extended Fig. 11A) corresponding to labeled examples in panels E and F. E ESM2 generated input protein embeddings of AAA+ regulator proteins that are structural homologs to TnsC (grey and red; using Foldseek60). Structural homologs of TnsC with confirmed involvement in Tn7-like transposons upon manual inspection were designated “TnsC-like AAA+ (manually inspected)” and are colored red. Other MGYP proteins annotated as “TnsC” against the UniRef90 database (orange) were added as positive controls for TnsC function. NuoA (NADH-quinone oxidoreductase subunit A; purple) were added as structural and functional negative controls. DnaB helicases (blues) were added as functional negative controls, as these proteins have similar folds to TnsC but are not associated with transposition. F Combined input protein and context embeddings of genes in panel E. These embeddings are generated through concatenation of pLM (ESM2) embeddings and context (last layer of gLM) embeddings. Negative controls (NuoA and DnaB helicases) form distinct clusters in both E and F. Numbered labels in grey boxes indicate the AAA+ proteins with various functional association predictions listed in panel D and Supplementary Fig. 7. Raw distance based clustering of the embeddings are shown in Supplementary Fig. 8. Source data are provided as a Source Data file.](https://figures.cochat.ai/paper-figures/8a3dda47fccc4d87a405c27ce88a12b8/genomic-language-model-predicts-protein-co-regulation-and-function-figure-4.jpg)

Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As a result, Transformer-based DNA models are constrained to short context lengths and use schemes that aggregate nucleotides into the basic units of language models, called tokens, thereby sacrificing single-nucleotide resolution (15, 16, 21–23).”
From Sequence modeling and design from molecular to genome scale with Evo (2024) · cites this paper as [23]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This is a unique finding since literature broadly has reported common co-regulation of genes in protein complexes at both transcription and translational levels96–102.”
From EXOSC3 G191 Variants Trigger System-Wide Recalibration of RNA Processing Machinery (2026) · cites this paper as [96]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 3×
“Our model is based on the transformer 24 architecture and is trained using millions of unla-belled metagenomic sequences via the masked language modeling 25 objective, with the hypothesis that its ability to attend to different parts of a multi-gene sequence will result in the learning of gene…”
InfluentialHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“This is in contrast to Long-Short-Term-Memory networks (LSTMs) 26 that sequentially processes tokens.”
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Fast and sensitive protein alignment using DIAMOND
Buchfink, B., Xie, C., & Huson, D. H. (2015) · Nature Methods · doi:10.1038/nmeth.3176 · Cited by 14,201 · cited 1×
“0.7.145 57 against the Uni-Ref90 database 58 with an e-value cut-off 1E-5.”
InfluentialScaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“…key areas for further development: First, the transformer architecture has shown to be successful in ef fi cient scaling; in both natural language 50 and protein language processing 23 , increasing the number of parameters in the model along with the training dataset size have been shown to lead…”
The Complete Genome Sequence of Escherichia coli K-12
Blattner, F. R., Plunkett, G., Bloch, C. A., Perna, N. T., Burland, V., Riley, M., Collado-Vides, J., Glasner, J. D., Rode, C. K., Mayhew, G. F., et al. (1997) · Science · doi:10.1126/science.277.5331.1453 · Cited by 7,452
Evolutionary-scale prediction of atomic level protein structure with a language model
Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., et al. (2022) · doi:10.1101/2022.07.20.500902 · Cited by 5,990
Accurate prediction of protein structures and interactions using a 3-track network
Baek, M., DiMaio, F., Anishchenko, I., Dauparas, J., Ovchinnikov, S., Lee, G. R., Wang, J., Cong, Q., Kinch, L. N., Schaeffer, R. D., et al. (2021) · doi:10.1101/2021.06.14.448402 · Cited by 5,549
Self-citation
Research works printed but not linked: 3
- Coenen, A. et al. Visualizing and Measuring the Geometry of BERT. In: Proceedings of the Neural Information Processing Systems, 2019.
- Kiros, R., Salakhutdinov, R. & Zemel, R. Multimodal Neural Language Models. In: Proceedings of the 31st International Conference on Machine Learning, Vol. 32, No. 2, pp. 595–603. PMLR, Beijing, China, 22–24 Jun 2014.
- Loshchilov, I. & Hutter, F. Decoupled Weight Decay Regularization. In: Proceedings of the International Conference on Learning Representations, 2019.
Cited by
Sequence modeling and design from molecular to genome scale with Evo
Nguyen, E., Poli, M., Durrant, M. G., Kang, B., Katrekar, D., Li, D. B., Bartie, L. J., Thomas, A. W., King, S. H., Brixi, G., et al. (2024) · Science · doi:10.1126/science.ado9336 · Cited by 580
“As a result, Transformer-based DNA models are constrained to short context lengths and use schemes that aggregate nucleotides into the basic units of language models, called tokens, thereby sacrificing single-nucleotide resolution ( 15 , 16 , 21 – 23 ).”
AI-Driven Antimicrobial Peptide Discovery: Mining and Generation
Szymczak, P., Zarzecki, W., Wang, J., Duan, Y., Wang, J., Coelho, L. P., de la Fuente‐Núñez, C., & Szczurek, E. (2025) · Accounts of Chemical Research · doi:10.1021/acs.accounts.0c00594 · Cited by 138
DeGroat, W., Abdelhalim, H., Peker, E., Sheth, N., Narayanan, R., Zeeshan, S., Liang, B. T., & Ahmed, Z. (2024) · Scientific Reports · doi:10.1038/s41598-024-78553-6 · Cited by 120
Large language models in biomedicine and healthcare
Zhou, J., Li, H., Chen, S., Chen, Z., Han, Z., & Gao, X. (2025) · Yearbook of pediatric endocrinology · doi:10.1038/s44387-025-00047-1 · Cited by 76
“For example, several LLMs related to genomics have been proposed, including Evo 32 , gLM 33 , and Caduceus 34 .”
Artificial intelligence for omics data analysis
Ahmed, Z., Wan, S., Zhang, F., & Zhong, W. (2024) · BMC Methods · doi:10.1186/s44330-024-00004-5 · Cited by 100
“…recent developed genomic language model (gLM), trained on millions of metagenomic scaffolds to learn the latent functional and regulatory relationships between genes, has proven to be a potent and promising method to close the gap between genomic-context and gene sequence-structure–function [6].”
Background
Metadata
- Authors
- Hwang, Y., Cornman, A. L., Kellogg, E. H., Ovchinnikov, S., & Girguis, P. R.
- Affiliations
- Harvard University (US)
- Tatta Bio (US)
- St. Jude Children's Research Hospital (US)
- Corresponding author
- Hwang, Yunha
- Ovchinnikov, Sergey
- Girguis, Peter R.
- Venue
- Nature Communications
- Published
- 2024-04-03
- Volume
- 15
- Issue
- 1
- Pages
- 2880-2880
- ISSN
- 2041-1723 (ISSN-L)
- PMC
- PMC10991518
- OpenAlex
- W4393870943
- Cited by
- 99
- References
- 80
- Citations by year
- 2026: 32 · 2025: 47 · 2024: 20
- FWCI
- 13.44
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Nature Communications (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Gordon and Betty Moore Foundation (9208)
- National Science Foundation (OCE-1635365)
- National Aeronautics and Space Administration (80NSSC19K1427)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Bioinformatics and Genomic Networks
- Keywords
- genomic language models, operon, protein embeddings, gene function prediction, gene expression regulation, attention patterns, phylogenetic context, functional modules, enzyme function
- MeSH terms
- Machine Learning, Operon, Phylogeny, Proteins, Semantics, Metagenomics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Genomic language model predicts protein co-regulation and function. https://app.cochat.ai/papers/10.1038/s41467-024-46947-9
- The paper
- Hwang, Y., Cornman, A. L., Kellogg, E. H., Ovchinnikov, S., & Girguis, P. R. (2024). Genomic language model predicts protein co-regulation and function. Nature Communications. https://doi.org/10.1038/s41467-024-46947-9
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.