Prediction of RNA–protein interactions using a nucleotide language model
Open accessThe DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
Motivation: The accumulation of sequencing data has enabled researchers to predict the interactions between RNA sequences and RNA-binding proteins (RBPs) using novel machine learning techniques. However, existing models are often difficult to interpret and require additional information to sequences. Bidirectional encoder representations from transformer (BERT) is a language-based deep learning model that is highly interpretable. Therefore, a model based on BERT architecture can potentially overcome such limitations. Results: Here, we propose BERT-RBP as a model to predict RNA–RBP interactions by adapting the BERT architecture pretrained on a human reference genome. Our model outperformed state-of-the-art prediction models using the eCLIP-seq data of 154 RBPs. The detailed analysis further revealed that BERT-RBP could recognize both the transcript region type and RNA secondary structure only based on sequence information. Overall, the results provide insights into the fine-tuning mechanism of BERT in biological contexts and provide evidence of the applicability of the model to other RNA-related problems. Availability and implementation: Python source codes are freely available at https://github.com/kkyamada/bert-rbp. The datasets underlying this article were derived from sources in the public domain: [RBPsuite (http://www.csbio.sjtu.edu.cn/bioinf/RBPsuite/), Ensembl Biomart (http://asia.ensembl.org/biomart/martview/)]. Supplementary information: Supplementary data are available at Bioinformatics Advances online.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
MMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets
Steinegger, M., & Söding, J. (2017) · Nature Biotechnology · doi:10.1038/nbt.3988 · Cited by 4,840
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280 · cited 1×
“…field of bioinformatics, several BERT architectures pre-trained on a massive corpus of protein sequences have been recently proposed, demonstrating their capability to decode the context of biological sequences (Rao et al. , 2019; Rives et al. , 2020; Elnaggar et al. , 2020; Iuchi et al. , 2021).”
BackgroundTranscriptome-wide Identification of RNA-Binding Protein and MicroRNA Target Sites by PAR-CLIP
Hafner, M., Landthaler, M., Burger, L., Khorshid, M., Hausser, J., Berninger, P., Rothballer, A., Ascano, M., Jungkamp, A.-C., Munschauer, M., et al. (2010) · Cell · doi:10.1016/j.cell.2010.03.009 · Cited by 2,873 · cited 1×
“Altered CLIP-seq protocols have also been developed (Hafner et al. , 2010; K¨onig et al. , 2010; Van Nostrand et al. , 2016).”
MethodPredicting the sequence specificities of DNA- and RNA-binding proteins by deep learning
Alipanahi, B., Delong, A., Weirauch, M. T., & Frey, B. J. (2015) · Nature Biotechnology · doi:10.1038/nbt.3300 · Cited by 2,873
ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning
Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3095381 · Cited by 2,329
A primer in BERTology: What we know about how BERT works
Rogers, A., Kovaleva, O., & Rumshisky, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00349 · Cited by 1,552 · cited 3×
“In addition to its performance, BERT is highly interpretable and suitable for translating extended contextual information compared to conventional deep learning architectures, such as CNNs and long short-term memory (Rogers et al. , 2020).”
MethodInfluentialHITS-CLIP yields genome-wide insights into brain alternative RNA processing
Licatalosi, D. D., Mele, A., Fak, J. J., Ule, J., Kayikci, M., Chi, S. W., Clark, T. A., Schweitzer, A. C., Blume, J. E., Wang, X., et al. (2008) · Nature · doi:10.1038/nature07488 · Cited by 1,352
A large-scale binding and functional map of human RNA-binding proteins
Van Nostrand, E. L., Freese, P., Pratt, G. A., Wang, X., Wei, X., Xiao, R., Blue, S. M., Chen, J.-Y., Cody, N. A. L., Dominguez, D., et al. (2020) · Nature · doi:10.1038/s41586-020-2077-3 · Cited by 1,317
iCLIP reveals the function of hnRNP particles in splicing at individual nucleotide resolution
König, J., Zarnack, K., Rot, G., Curk, T., Kayikci, M., Zupan, B., Turner, D. J., Luscombe, N. M., & Ule, J. (2010) · Nature Structural & Molecular Biology · doi:10.1038/nsmb.1838 · Cited by 1,126
Quantifying Attention Flow in Transformers
Abnar, S., & Zuidema, W. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.385 · Cited by 995
Evaluating Protein Transfer Learning with TAPE
Rao, R., Bhattacharya, N., Thomas, N., Duan, Y., Chen, X., Canny, J., Abbeel, P., & Song, Y. S. (2019) · doi:10.1101/676825 · Cited by 719 · cited 1×
“…field of bioinformatics, several BERT architectures pre-trained on a massive corpus of protein sequences have been recently proposed, demonstrating their capability to decode the context of biological sequences (Rao et al. , 2019; Rives et al. , 2020; Elnaggar et al. , 2020; Iuchi et al. , 2021).”
BackgroundSequence, Structure, and Context Preferences of Human RNA Binding Proteins
Dominguez, D., Freese, P., Alexis, M. S., Su, A., Hochman, M., Palden, T., Bazile, C., Lambert, N. J., Van Nostrand, E. L., Pratt, G. A., et al. (2018) · Molecular Cell · doi:10.1016/j.molcel.2018.05.001 · Cited by 562
Assessing BERT's Syntactic Abilities
Goldberg, Y. (2019) · arXiv · doi:10.48550/arxiv.1901.05287 · Cited by 385 · cited 2×
“The syntactic relationship among tokens is another intensely researched topic in BERTology and may incorporate hidden contextual patterns of nucleotide sequences (Goldberg, 2019).”
BackgroundTransformer protein language models are unsupervised structure learners
Rao, R., Meier, J., Sercu, T., Ovchinnikov, S., & Rives, A. (2020) · doi:10.1101/2020.12.15.422761 · Cited by 341
BERTology Meets Biology: Interpreting Attention in Protein Language Models
Vig, J., Madani, A., Varshney, L. R., Xiong, C., Socher, R., & Rajani, N. F. (2020) · doi:10.1101/2020.06.26.174417 · Cited by 338
Analyzing the Structure of Attention in a Transformer Language Model
Vig, J., & Belinkov, Y. (2019) · Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w19-4808 · Cited by 301 · cited 2×
“Researchers in an emerging field, called BERTology, intend to elucidate how BERT learns contextual information by analyzing attention, which essentially represents the flow of information within a model (Vig and Belinkov, 2019).”
BackgroundAPPRIS: annotation of principal and alternative splice isoforms
Rodriguez, J. M., Maietta, P., Ezkurdia, I., Pietrelli, A., Wesselink, J.-J., Lopez, G., Valencia, A., & Tress, M. L. (2013) · Nucleic Acids Research · doi:10.1093/nar/gks1058 · Cited by 232 · cited 1×
“For each gene, we selected the most prominent isoform based on the APPRIS annotation (Rodriguez et al. , 2013), transcript support level, and length of the transcript (the longer, the better).”
Method
Cited by
Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook.
AlSaad, R., Abd-alrazaq, A., Boughorbel, S., Ahmed, A., Renault, M.-A., Damseh, R., & Sheikh, J. (2024) · Journal of medical Internet research · doi:10.2196/59505 · Cited by 349
“…prediction (SpliceBERT [122]), assessment of long noncoding RNAs’ coding potential (LncCat [123]), RNA-binding protein interactions (BERT-RBP [124]), RNA modification identification (BERT-m7G [125]), and predictions related to protein expression and messenger RNA degradation (CodonBERT…”
Prediction of antibiotic resistance mechanisms using a protein language model
Yagimoto, K., Hosoda, S., Sato, M., & Hamada, M. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae550 · Cited by 25
Self-citationLi, Q., Cheng, X., Song, C., & Liu, T. (2023) · Symmetry · doi:10.3390/sym15030731 · Cited by 20
BiRNA-BERT allows efficient RNA language modeling with adaptive tokenization
Tahmid, M. T., Shahgir, H. S., Mahbub, S., Dong, Y., & Bayzid, M. S. (2025) · Communications Biology · doi:10.1038/s42003-025-08982-0 · Cited by 16
He, J., Zhou, T., Hu, L.-F., Jiao, Y., Wang, J., Yan, S., Jia, S., Chen, Q., Zhu, W., Zhang, J., et al. (2026) · Nature Communications · doi:10.1038/s41467-026-72351-6
Metadata
- Authors
- Yamada, K., & Hamada, M.
- Venue
- Bioinformatics Advances
- Publisher
- Oxford University Press (OUP)
- Published
- 2022-01-01
- Volume
- 2
- Issue
- 1
- Pages
- vbac023-vbac023
- ISSN
- 2635-0041 (ISSN-L)
- OpenAlex
- W4225917625
- Cited by
- 68
- References
- 21
- FWCI
- 3.87
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.