CoChat
Sign Up
PreprintCited by 22Genomics and Phylogenetic StudiesRNA Research and SplicingRNA and protein synthesis mechanisms

The DOI matches its Crossref record; the references it cites are not checked yet.

Orthrus: Towards Evolutionary and Functional RNA Foundation Models

Open access
Fradkin, Philip, Shi, Ruian, Dalal, Taykhoom, Isaev, Keren, Frey, Brendan J, Lee, Leo J, Morris, Quaid, Wang, Bo·2024·Nature Methods
DOI 10.1101/2024.10.10.617658PMC PMC11482885

Preprint — published as 10.1038/s41592-026-03064-3 · Paper page

Open this paper in CoChat — free account

Abstract

In the face of rapidly accumulating genomic data, our ability to accurately predict key mature RNA properties that underlie transcript function and regulation remains limited. Pre-trained genomic foundation models offer an avenue to adapt learned RNA representations to biological prediction tasks. However, existing genomic foundation models are trained using strategies borrowed from textual or visual domains that do not leverage biological domain knowledge. Here, we introduce Orthrus, a Mamba-based mature RNA foundation model pre-trained using a novel self-supervised contrastive learning objective with biological augmentations. Orthrus is trained by maximizing embedding similarity between curated pairs of RNA transcripts, where pairs are formed from splice isoforms of 10 model organisms and transcripts from orthologous genes in 400+ mammalian species from the Zoonomia Project. This training objective results in a latent representation that clusters RNA sequences with functional and evolutionary similarities. We find that the generalized mature RNA isoform representations learned by Orthrus significantly outperform existing genomic foundation models on five mRNA property prediction tasks, and requires only a fraction of fine-tuning data to do so. Finally, we show that Orthrus is capable of capturing divergent biological function of individual transcript isoforms.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1101/2024.10.10.617658

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

The citation lists haven’t been fetched yet

References 78 · Cited by 22

Research works printed but not linked: 61

  1. 10.1126/science.1254806 · DOI
  2. 10.1038/nmeth.3810 · DOI · Paper page
  3. 10.1038/nrm4069 · DOI
  4. 10.1038/nmeth.4435 · DOI · Paper page
  5. 10.1016/J.CELL.2018.12.015 · DOI · Paper page
  6. Linder, J. , Koplik, S. E. , Kundaje, A. & Seelig, G . Deciphering the impact of genetic variation on human polyadenylation using APARENT2. Genome Biol 23, 232 (2022). · DOI · Paper page
  7. Agarwal, V. & Kelley, D. R . The genetic and biochemical determinants of mRNA degradation rates in mammals. Genome Biol 23, 245 (2022). · DOI · Paper page
  8. Merico, D. et al. G p.Met645Arg causes Wilson disease by promoting exon 6 skipping. NPJ Genom Med 5, 16 (2020). · DOI
  9. 10.1038/gim.2015.30 · DOI · Paper page
  10. Celaj, A. , et al. An rna foundation model enables discovery of disease mechanisms and candidate therapeutics. bioRxiv (2023).
  11. Lotfollahi, M. et al. Predicting cellular responses to complex perturbations in high-throughput screens. Molecular Systems Biology 19 (2023). · DOI · Paper page
  12. 10.1093/bioinformatics/btab083 · DOI · Paper page
  13. Chen, J. , et al. Interpretable RNA Foundation Model from Unannotated Data for Highly Accurate RNA Structure and Function Predictions. arXiv e-prints arXiv:2204.00300 (2022). · DOI · Paper page
  14. Nguyen, E. , et al. HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution. arXiv e-prints (2023).
  15. Devlin, J. , Chang, M.-W. , Lee, K. & Toutanova, K . BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv e-prints arXiv:1810.04805 (2018). · arXiv
  16. 10.1093/bioinformatics/btae118 · DOI
  17. Song, Y. , Wang, T. , Mondal, S. K. & Sahoo, J. P . A comprehensive survey of few-shot learning: Evolution, applications, challenges, and opportunities (2022). URL https://arxiv.org/abs/2205.06743.2205.06743.
  18. Tomasev, N. , et al. Pushing the limits of self-supervised ResNets: Can we outperform supervised learning without labels on ImageNet? arXiv e-prints (2022).
  19. Radford, A. , et al. Learning Transferable Visual Models From Natural Language Supervision. arXiv e-prints arXiv:2103.00020 (2021). · arXiv
  20. 2021, Variant interpretation using population databases: Lessons from gnomAD, Human Mutation, 43, 1012
  21. 10.1038/s41586-021-03205-y · DOI · Paper page
  22. 10.1038/nature10530 · DOI · Paper page
  23. 10.1126/science.abn2937 · DOI
  24. 10.1126/science.abn3943 · DOI · Paper page
  25. Dalla-Torre, H. , et al. The nucleotide transformer: Building and evaluating robust foundation models for human genomics. bioRxiv (2023). · DOI · Paper page
  26. Lu, A. X. , Lu, A. X. & Moses, A . Evolution Is All You Need: Phylogenetic Augmentation for Contrastive Learning. arXiv e-prints (2020).
  27. Chen, T. , Kornblith, S. , Norouzi, M. & Hinton, G. A Simple Framework for Contrastive Learning of Visual Representations. arXiv e-prints arXiv:2002.05709 (2020). · arXiv
  28. 10.1126/science.abn3107 · DOI · Paper page
  29. McInnes, L. , Healy, J. & Melville, J. Umap: Uniform manifold approximation and projection for dimension reduction (2020). URL https://arxiv.org/abs/1802.03426.1802.03426.
  30. 10.1093/nar/gkaa1087 · DOI · Paper page
  31. 10.1093/nar/gkv1189 · DOI · Paper page
  32. 10.1038/nrg2810 · DOI
  33. 10.1093/nar/gkac1032 · DOI
  34. Yeh, C.-H. , et al. Decoupled Contrastive Learning. arXiv e-prints arXiv:2110.06848 (2021). · arXiv
  35. 10.1186/s13059-024-03221-x · DOI
  36. Schertzer, M. D. et al. Cas13d-mediated isoform-specific rna knockdown with a unified computational and experimental toolbox. bioRxiv 2023.09.12.557474 (2023). URL https://www.biorxiv.org/content/10.1101/2023.09.12.557474v1. · DOI
  37. 10.1038/s41586-024-07487-w · DOI · Paper page
  38. Warren, C. F. A. , Wong-Brown, M. W. & Bowden, N. A . BCL-2 family isoforms in apoptosis and cancer. Cell Death Dis. 10, 177 (2019). · DOI
  39. 10.1038/s41419-020-2589-7 · DOI
  40. Lee, N. K. , Tang, Z. , Toneyan, S. & Koo, P. K . EvoAug: improving generalization and interpretability of genomic deep neural networks with evolution-inspired data augmentations. Genome Biol 24, 105 (2023). · DOI
  41. Lu, A. X. , Zhang, H. , Ghassemi, M. & Moses, A . Self-supervised contrastive learning of protein representations by mutual information maximization. bioRxiv (2020). · DOI
  42. Pertea, M. et al. CHESS: a new human gene catalog curated from thousands of large-scale RNA sequencing experiments reveals extensive transcriptional noise. Genome Biol 19, 208 (2018). · DOI · Paper page
  43. 10.1101/gr.156919.113 · DOI
  44. van den Oord, A. , Li, Y. & Vinyals, O . Representation Learning with Contrastive Predictive Coding. arXiv e-prints (2018).
  45. Yun, S. , et al. CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features. arXiv e-prints arXiv:1905.04899 (2019). · DOI · Paper page
  46. Zhang, H. , Cisse, M. , Dauphin, Y. N. & Lopez-Paz, D. mixup: Beyond Empirical Risk Minimization. arXiv e-prints arXiv:1710.09412 (2017). · arXiv
  47. Vaswani, A . Attention is all you need. Advances in Neural Information Processing Systems (2017).
  48. Gu, A. , Goel, K. & Ŕe, C. Efficiently modeling long sequences with structured state spaces (2022). URL https://arxiv.org/abs/2111.00396.2111.00396.
  49. 10.1038/s41467-023-37960-5 · DOI · Paper page
  50. Sohn, K. , Lee, D. , Sugiyama, M. , Luxburg, U. , Guyon, I. & Garnett, R. (eds) Improved deep metric learning with multi-class n-pair loss objective. (eds Lee, D. , Sugiyama, M. , Luxburg, U. , Guyon, I . & Garnett, R. ) Advances in Neural Information Processing Systems, Vol. 29 (Curran Associates, Inc., 2016).
  51. Balestriero, R. , et al. A Cookbook of Self-Supervised Learning. arXiv e-prints (2023).
  52. Bardes, A. , Ponce, J. & LeCun, Y . VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning. arXiv e-prints (2021).
  53. Grill, J.-B. , et al. Bootstrap your own latent: A new approach to self-supervised Learning. arXiv e-prints (2020).
  54. 10.1038/s41594-022-00819-2 · DOI
  55. 10.1093/nar/gkab1058 · DOI
  56. 10.1093/genetics/iyad031 · DOI
  57. 10.1038/75556 · DOI · Paper page
  58. Zhou, N. et al. The CAFA challenge reports improved protein function predic-tion and new functional annotations for hundreds of genes through experimental screens. Genome Biol 20, 244 (2019).
  59. 10.1093/nar/gkac958 · DOI
  60. 10.1038/s41467-023-39897-1 · DOI
  61. Ben-Shaul, I. , Shwartz-Ziv, R. , Galanti, T. , Dekel, S. & LeCun, Y . Reverse Engineering Self-Supervised Learning. arXiv e-prints (2023).

Metadata

Authors
Fradkin, P., Shi, R., Dalal, T., Isaev, K., Frey, B. J., Lee, L. J., Morris, Q., & Wang, B.
Affiliations
  • University of Toronto (CA)
  • New York Genome Center (US)
  • Memorial Sloan Kettering Cancer Center (US)
  • University Health Network (CA)
Corresponding author
  • Lee, Leo J
  • Morris, Quaid
  • Wang, Bo
ORCID
Venue
Nature Methods
Publisher
openRxiv
Published
2024-10-12
Volume
23
Issue
5
Pages
935-945
ISSN
1548-7091 (ISSN-L)
DOI
10.1101/2024.10.10.617658
PMC
PMC11482885
OpenAlex
W7154722552
Cited by
22
References
78
Citations by year
2026: 13 · 2025: 9
FWCI
26.87
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Methods (publishedVersion)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (publishedVersion, cc-by)
Funders
  • National Human Genome Research Institute (R01HG013328)
  • National Cancer Institute (P30CA008748)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › RNA Research and Splicing
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › RNA and protein synthesis mechanisms
Keywords
contrastive learning, Mamba, splice variants, transcript regulation, self-supervised pre-training, bioaugmentation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review