RiNALMo: general-purpose RNA language models can generalize well on structure prediction tasks
Open accessThe DOI matches its Crossref record; 25 of the 25 references it cites were found in a registry or an index.
Abstract
While RNA has recently been recognized as an interesting small-molecule drug target, many challenges remain to be addressed before we take full advantage of it. This emphasizes the necessity to improve our understanding of its structures and functions. Over the years, sequencing technologies have produced an enormous amount of unlabeled RNA data, which hides a huge potential. Motivated by the successes of protein language models, we introduce RiboNucleic Acid Language Model (RiNALMo) to unveil the hidden code of RNA. RiNALMo is the largest RNA language model to date, with 650M parameters pre-trained on 36M non-coding RNA sequences from several databases. It can extract hidden knowledge and capture the underlying structure information implicitly embedded within the RNA sequences. RiNALMo achieves state-of-the-art results on several downstream tasks. Notably, we show that its generalization capabilities overcome the inability of other deep learning methods for secondary structure prediction to generalize on unseen RNA families. RiNALMo, a large-scale RNA language model trained on non-coding RNA sequences, captures structural information and achieves state-of-the-art performance on multiple tasks, notably generalizing to unseen RNA families in secondary structure prediction.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Evolutionary-scale prediction of atomic level protein structure with a language model
Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., et al. (2022) · doi:10.1101/2022.07.20.500902 · Cited by 5,990
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280 · cited 2×
“…RNAs (ncRNAs)fromtheRNAcentraldatabase 24,22 pre-trainedanensembleof LMs ranging from 25M to 400M parameters trained on a much larger dataset of 1B sequences with the architecture analogous to the ESM protein LM 13 enhanced by several advanced techniques such as RoPE and fused layer norm.”
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning
Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3095381 · Cited by 2,329
Research works printed but not linked: 47
- Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), (Minneapolis, Minnesota, USA, 2019).
- Nambiar, A. et al. Transforming the language of life: transformer neural networks for protein prediction tasks. In: Proc. 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, 1–8 (ACM, 2020). · DOI · Paper page
- Childs-Disney, J. L. et al. Targeting RNA structures with small molecules. Nat. Rev. Drug Discov. 21, 736–762 (2022). · DOI · Paper page
- Garner, A. L. Contemporary progress and opportunities in RNA-targeted drug discovery. ACS Med. Chem. Lett. 14, 251–259 (2023). · DOI
- Schneider, B. et al. When will RNA get its AlphaFold moment? Nucleic Acids Res. 51, 9522–9532 (2023). · DOI · Paper page
- Wang, X. et al. Uni-RNA Universal pre-trained models revolutionize RNA research. bioRxiv https://doi.org/10.1101/2023.07.11.548588 (2023). · DOI · Paper page
- Vaswani, A. et al. Attention is all you need. Adv. Neural Inf. Process. Syst. 30 (2017).
- RNAcentral Consortium RNAcentral 2021: secondary structure integration, improved sequence search and new member databases. Nucleic Acids Res. 49 212–220 (2020). · DOI · Paper page
- Zhang, Y. et al. Multiple sequence alignment-based RNA language model and its application to structural inference. Nucleic Acids Res. 52, 3–3 (2024). · DOI · Paper page
- Chen, K. et al. Self-supervised learning on millions of primary RNA sequences from 72 vertebrates improves sequence-based RNA splicing prediction, Brief. Bioinform. 25, bbae163 (2024). · DOI
- Ji, Y., Zhou, Z., Liu, H. & Davuluri, R. V. DNABERT pre-trained bidirectional encoder representations from transformers model for DNA-language in genome. Bioinformatics 37, 2112–2120 (2021). · DOI · Paper page
- Celaj, A. et al. An RNA foundation model enables discovery of disease mechanisms and candidate therapeutics. bioRxiv https://doi.org/10.1101/2023.09.20.558508 (2023). · DOI
- Chu, Y. et al. A 5’ UTR language model for decoding untranslated regions of mRNA and function predictions. Nat. Mach. Intell. 6, 449–460 (2024). · DOI · Paper page
- Reuter, J. S. & Mathews, D. H. RNAstructure software for RNA secondary structure prediction and analysis. BMC Bioinform. 11, 129 (2010). · DOI · Paper page
- Do, C. B., Woods, D. A. & Batzoglou, S. CONTRAfold: RNA secondary structure prediction without physics-based models. Bioinformatics 22, 90–98 (2006). · DOI · Paper page
- Singh, J., Hanson, J., Paliwal, K. & Zhou, Y. RNA secondary structure prediction using an ensemble of two-dimensional deep neural networks and transfer learning. Nat. Commun. 10, 5407 (2019). · DOI · Paper page
- Fu, L. et al. UFold: fast and accurate RNA secondary structure prediction with deep learning. Nucleic Acids Res. 50, 14–14 (2021). · DOI · Paper page
- Sato, K., Akiyama, M. & Sakakibara, Y. RNA secondary structure prediction using deep learning with thermodynamic integration. Nat. Commun. 12, 941 (2021). · DOI · Paper page
- Danaee, P. et al. bpRNA large-scale automated annotation and analysis of RNA secondary structure. Nucleic Acids Res. 46, 5381–5394 (2018). · DOI · Paper page
- Berman, H. M. et al. The Protein Data Bank. Nucleic Acids Res. 28, 235–242 (2000). · DOI · Paper page
- Yang, H. et al. Tools for the automatic identification and classification of RNA base pairs. Nucleic Acids Res. 31, 3450–3460 (2003). · DOI · Paper page
- Andronescu, M., Condon, A., Hoos, H. H., Mathews, D. H. & Murphy, K. P. Efficient parameter estimation for RNA secondary structure prediction. Bioinformatics 23, 19–28 (2007). · DOI · Paper page
- Andronescu, M., Condon, A., Hoos, H. H., Mathews, D. H. & Murphy, K. P. Computational approaches for RNA energy parameter estimation. RNA 16, 2304–2318 (2010). · DOI · Paper page
- Kalvari, I. et al. Rfam 14: expanded coverage of metagenomic, viral and microRNA families. Nucleic Acids Res. 49, 192–200 (2020). · DOI · Paper page
- Wayment-Steele, H. K. et al. RNA secondary structure packages evaluated and improved by high-throughput experiments. Nat. Methods 19, 1234–1242 (2022). · DOI · Paper page
- Szikszai, M., Wise, M., Datta, A., Ward, M. & Mathews, D. H. Deep learning models for RNA secondary structure prediction (probably) do not generalize across families. Bioinformatics 38, 3892–3899 (2022). · DOI · Paper page
- Justyna, M., Antczak, M. & Szachniuk, M. Machine learning for RNA 2D structure prediction benchmarked on experimental data. Brief. Bioinforma. 24, 153 (2023). · DOI · Paper page
- Sloma, MichaelFandMathews,DavidH. Exact calculation of loop formation probability identifies folding motifs in RNA secondary structures. RNA 22, 1808–1818 (2016). · DOI · Paper page
- Mathews, D. H. How to benchmark RNA secondary structure prediction accuracy. Methods 162, 60–67 (2019). · DOI · Paper page
- Hamada, M., Sato, K., Kiryu, H., Mituyama, T. & Asai, K. Predictions of RNA secondary structure by combining homologous sequence information. Bioinformatics 25, 330–338 (2009). · DOI
- Will, S., Reiche, K., Hofacker, I. L., Stadler, P. F. & Backofen, R. Inferring noncoding RNA families and classes by means of genome-scale structure-based clustering. PLoS Comput. Biol. 3, 65 (2007). · DOI · Paper page
- Hamada, M., Sato, K. & Asai, K. Improving the accuracy of predicting secondary structure for aligned RNA sequences. Nucleic Acids Res. 39, 393–402 (2010). · DOI
- Puton, T., Kozlowski, L. P., Rother, K. M. & Bujnicki, J. M. CompaRNA: a server for continuous benchmarking of automated methods for RNA secondary structure prediction. Nucleic Acids Res. 41, 4307–4323 (2013). · DOI
- Nawrocki, E. P. & Eddy, S. R. Infernal 1.1: 100-fold faster RNA homology searches. Bioinformatics 29, 2933–2935 (2013). · DOI · Paper page
- Scalzitti, N. et al. Spliceator: multi-species splice site prediction using convolutional neural networks. BMC Bioinforma. 22, 1–26 (2021). · DOI · Paper page
- Scalzitti, N., Jeannin-Girardon, A., Collet, P., Poch, O. & Thompson, J. D. A benchmark study of ab initio gene prediction methods in diverse eukaryotic organisms. BMC Genom. 21, 1–20 (2020). · DOI · Paper page
- Kalvari, I. et al. Rfam 13.0: shifting to a genome-centric resource for non-coding RNA families. Nucleic Acids Res. 46, 335–342 (2018). · DOI · Paper page
- Noviello, T. M. R., Ceccarelli, F., Ceccarelli, M. & Cerulo, L. Deep learning predicts short non-coding RNA functions from only raw sequence data. PLoS Comput. Biol. 16, 1008415 (2020). · DOI
- Cao, J. et al. High-throughput 5’ UTR engineering for enhanced protein production in non-viral gene therapies. Nat. Commun. 12, 4138 (2021). · DOI · Paper page
- Anand, R. ett al. RNA-FrameFlow: flow matching for de novo 3D RNA backbone design. https://doi.org/10.48550/arXiv.2406.13839 (2024). · DOI
- Churkin, A., Barash, D. (eds.) RNA design: methods and protocols. https://doi.org/10.1007/978-1-0716-4079-1 (Springer, 2024). · doi:10.1007/978-1-0716-4079-1
- Xiong, R. et al. On layer normalization in the transformer architecture. In: Proc. International Conference on Machine Learning, 10524–10533 (PMLR, 2020).
- Parisien, M., Cruz, J. A., Westhof, E. & Major, F. New metrics for comparing and assessing discrepancies between RNA 3D structures and models. RNA 15, 1875–1885 (2009). · DOI
- Kerpedjiev, P., Hammer, S. & Hofacker, I. L. Forna (force-directed rna): Simple and effective online rna secondary structure diagrams. Bioinformatics 31, 3377–3379 (2015). · DOI · Paper page
- Paszke, A. et al. Pytorch: an imperative style, high-performance deep learning library. Adv. Neural Inf. Process. Syst. 32, 8024–8035 (2019).
- Sayers, E. W. et al. Database resources of the National Center for Biotechnology Information in 2023. Nucleic Acids Res. 51, 29–38 (2022). · DOI · Paper page
- Martin, F. J. et al. Ensembl 2023. Nucleic Acids Res. 51, 933–941 (2022). · DOI · Paper page
Cited by
ERNIE-RNA: an RNA language model with structure-enhanced representations
Yin, W., Zhang, Z., Zhang, S., He, L., Zhang, R., Jiang, R., Liu, G., Wang, J., Zhang, X., Qin, T., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-64972-0 · Cited by 55
Orthrus: toward evolutionary and functional RNA foundation models
Fradkin, P., Shi, R. “., Dalal, T., Isaev, K., Frey, B. J., Lee, L. J., Morris, Q., & Wang, B. (2026) · Nature Methods · doi:10.1038/s41592-026-03064-3 · Cited by 22
BiRNA-BERT allows efficient RNA language modeling with adaptive tokenization
Tahmid, M. T., Shahgir, H. S., Mahbub, S., Dong, Y., & Bayzid, M. S. (2025) · Communications Biology · doi:10.1038/s42003-025-08982-0 · Cited by 16
sincFold: end-to-end learning of short- and long-range interactions in RNA secondary structure
(2024) · doi:10.1093/bib/bbae271 · Cited by 14
HydraRNA: a hybrid architecture based full-length RNA language model
Li, G., Jiang, F., Zhu, J., Cui, H., Wang, Z., & Chen, W. (2025) · Genome Biology · doi:10.1186/s13059-025-03853-7 · Cited by 10
Metadata
- Authors
- Penić, R. J., Vlašić, T., Huber, R. G., Wan, Y., & Šikić, M.
- Venue
- Nature Communications
- Published
- 2025-07-01
- Volume
- 16
- Issue
- 1
- Pages
- 5671-5671
- ISSN
- 2041-1723 (ISSN-L)
- OpenAlex
- W4411886797
- Cited by
- 122
- References
- 25
- FWCI
- 29.40
- Type
- journal article
- Language
- en
- Open access
- gold
- Licence
- creativecommons.org/licenses/by-nc-nd/4.0 (tdm)
- creativecommons.org/licenses/by-nc-nd/4.0 (vor)
- Funders
- National Research Foundation Singapore (NRF-CRP27-2021RS-0001)
- Agency for Science, Technology and Research (I23D1AG079)
- National Supercomputing Centre (NSCC), Singapore
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: RiNALMo: general-purpose RNA language models can generalize well on structure prediction tasks. https://app.cochat.ai/papers/10.1038/s41467-025-60872-5
- The paper
- Penić, R. J., Vlašić, T., Huber, R. G., Wan, Y., & Šikić, M. (2025). RiNALMo: general-purpose RNA language models can generalize well on structure prediction tasks. Nature Communications. https://doi.org/10.1038/s41467-025-60872-5
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.