HydraRNA: a hybrid architecture based full-length RNA language model

Abstract

BACKGROUND: RNA, an essential component of the central dogma of molecular biology, plays versatile roles in all cellular processes. RNA large language models (LLMs) are emerging as powerful methods in RNA research to decipher its intricate network of function and regulation. However, previous RNA LLMs were based on the Transformer model and pre-trained on short segment of non-coding RNAs, which limits their general usability. Here we present the first full-length RNA foundation model, HydraRNA, which is based on a hybrid architecture of bidirectional state space model and multi-head attention mechanism. RESULTS: HydraRNA is pre-trained on a large amount of both protein-coding mRNAs and non-coding RNAs. Despite being pre-trained with the fewest parameters and the least GPU resources, HydraRNA learns better RNA representations and outperforms the existing foundation models on a variety of mRNA-related tasks, including coding/non-coding RNA classification, prediction of RNA secondary structure, RBP binding sites, splicing and polyadenylation sites, mRNA stability and translation efficiency. Furthermore, HydraRNA can accurately predict the effect of mutations and estimate the relative contributions of different mRNA regions to the RNA stability and translation. CONCLUSIONS: Our results demonstrate that the hybrid architecture outperforms the pure Transformer architectures in RNA language modeling. We anticipate that HydraRNA will enable dissecting the diverse properties of mRNA, accelerating the research of mRNA regulation and facilitating the optimal design of mRNA therapeutics.