Multi-purpose RNA language modelling with motif-aware pretraining and type-guided fine-tuning
Open accessThe DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Abstract
Pretrained language models have shown promise in analysing nucleotide sequences, yet a versatile model excelling across diverse tasks with a single pretrained weight set remains elusive. Here we introduce RNAErnie, an RNA-focused pretrained model built upon the transformer architecture, employing two simple yet effective strategies. First, RNAErnie enhances pretraining by incorporating RNA motifs as biological priors and introducing motif-level random masking in addition to masked language modelling at base/subsequence levels. It also tokenizes RNA types (for example, miRNA, lnRNA) as stop words, appending them to sequences during pretraining. Second, subject to out-of-distribution tasks with RNA sequences not seen during the pretraining phase, RNAErnie proposes a type-guided fine-tuning strategy that first predicts possible RNA types using an RNA sequence and then appends the predicted type to the tail of sequence to refine feature embedding in a post hoc way. Our extensive evaluation across seven datasets and five tasks demonstrates the superiority of RNAErnie in both supervised and unsupervised learning. It surpasses baselines with up to 1.8% higher accuracy in classification, 2.2% greater accuracy in interaction prediction and 3.3% improved F1 score in structure prediction, showcasing its robustness and adaptability with a unified pretrained foundation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Mfold web server for nucleic acid folding and hybridization prediction
Zuker, M. (2003) · Nucleic Acids Research · doi:10.1093/nar/gkg595 · Cited by 13,404
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
Predicting effective microRNA target sites in mammalian mRNAs
Agarwal, V., Bell, G. W., Nam, J.-W., & Bartel, D. P. (2015) · eLife · doi:10.7554/elife.05005 · Cited by 7,318
Lorenz, R., Bernhart, S. H., Höner zu Siederdissen, C., Tafer, H., Flamm, C., Stadler, P. F., & Hofacker, I. L. (2011) · Algorithms for Molecular Biology · doi:10.1186/1748-7188-6-26 · Cited by 5,098
MicroRNA targets in Drosophila
Enright, A. J., John, B., Gaul, U., Tuschl, T., Sander, C., & Marks, D. S. (2003) · Genome Biology · doi:10.1186/gb-2003-5-1-r1 · Cited by 3,597
Ancient DNA and the rewriting of human history: be sparing with Occam’s razor
Haber, M., Mezzavilla, M., Xue, Y., & Tyler-Smith, C. (2016) · Genome Biology · doi:10.1186/s13059-015-0866-z · Cited by 1,924
SpanBERT: Improving Pre-training by Representing and Predicting Spans
Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., & Levy, O. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00300 · Cited by 1,827 · cited 1×
“In the TBTH architecture, the RNAErnie approach directly combines the RNAErnie basic block and the task-specific head to construct an end-to-end neural network for downstream tasks and then trains the neural network using the labelled datasets in a supervised learning manner.”
MethodSpanBERT: Improving Pre-training by Representing and Predicting Spans
Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., & Levy, O. (2019) · arXiv · doi:10.48550/arxiv.1907.10529 · Cited by 1,827
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey
Min, B., Ross, H., Sulem, E., Veyseh, A. P. B., Nguyen, T. H., Sainz, O., Agirre, E., Heintz, I., & Roth, D. (2024) · ACM Computing Surveys · doi:10.1145/3605943 · Cited by 1,387
Frankish, A., Diekhans, M., Jungreis, I., Lagarde, J., Loveland, J. E., Mudge, J. M., Sisu, C., Wright, J. C., Armstrong, J., Barnes, I., et al. (2021) · Nucleic Acids Research · doi:10.1093/nar/gkaa1087 · Cited by 1,334
Cited by
ERNIE-RNA: an RNA language model with structure-enhanced representations
Yin, W., Zhang, Z., Zhang, S., He, L., Zhang, R., Jiang, R., Liu, G., Wang, J., Zhang, X., Qin, T., et al. (2025) · Nature Communications · doi:10.1038/s41467-025-64972-0 · Cited by 55
Knowledge-guided large language model for material science
Wang, G., Hu, J., Zhou, J., Liu, S., Li, Q., & Sun, Z. (2025) · Review of Materials Research · doi:10.1016/j.revmat.2025.100007 · Cited by 16
HydraRNA: a hybrid architecture based full-length RNA language model
Li, G., Jiang, F., Zhu, J., Cui, H., Wang, Z., & Chen, W. (2025) · Genome Biology · doi:10.1186/s13059-025-03853-7 · Cited by 10
BAGEL: Protein engineering via exploration of an energy landscape
(2025) · doi:10.1371/journal.pcbi.1013774 · Cited by 7
Metadata
- Authors
- Wang, N., Bian, J., Li, Y., Li, X., Mumtaz, S., Kong, L., & Xiong, H.
- Venue
- Nature Machine Intelligence
- Published
- 2024-05-13
- Volume
- 6
- Issue
- 5
- Pages
- 548-557
- ISSN
- 2522-5839 (ISSN-L)
- OpenAlex
- W4396871204
- Cited by
- 132
- References
- 17
- FWCI
- 15.34
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Multi-purpose RNA language modelling with motif-aware pretraining and type-guided fine-tuning. https://app.cochat.ai/papers/10.1038/s42256-024-00836-4
- The paper
- Wang, N., Bian, J., Li, Y., Li, X., Mumtaz, S., Kong, L., & Xiong, H. (2024). Multi-purpose RNA language modelling with motif-aware pretraining and type-guided fine-tuning. Nature Machine Intelligence. https://doi.org/10.1038/s42256-024-00836-4
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.