CoChat
Sign Up
PreprintCited by 112RNA and protein synthesis mechanismsMachine Learning in BioinformaticsGenomics and Phylogenetic Studies

The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.

Multiple sequence-alignment-based RNA language model and its application to structural inference

Open access
Zhang, Yikun, Lang, Mei, Jiang, Jiuhong, Gao, Zhiqiang, Xu, Fan, Litfin, Thomas, Chen, Ke, Singh, Jaswinder, Huang, Xiansong, Song, Guoli, Tian, Yonghong, Zhan, Jian, Chen, Jie, Zhou, Yaoqi·2023·Nucleic Acids Research
DOI 10.1101/2023.03.15.532863PMC PMC10783488

Preprint — published as 10.1093/nar/gkad1031 · Paper page

Open this paper in CoChat — free accountOpen PDF

Abstract

Compared to proteins, DNA and RNA are more difficult languages to interpret because 4-letter-coded DNA/RNA sequences have less information content than 20-letter-coded protein sequences. While BERT (Bidirectional Encoder Representations from Transformers)-like language models have been developed for RNA, they are ineffective at capturing the evolutionary information from homologous sequences because unlike proteins, RNA sequences are less conserved. Here, we have developed an unsupervised Multiple sequence-alignment-based RNA language model (RNA-MSM) by utilizing homologous sequences from an automatic pipeline, RNAcmap. The resulting unsupervised, two-dimensional attention maps and one-dimensional embeddings from RNA-MSM can be directly mapped with high accuracy to 2D base pairing probabilities and 1D solvent accessibilities, respectively. Further fine-tuning led to significantly improved performance on these two downstream tasks over existing state-of-the-art techniques. We anticipate that the pre-trained RNA-MSM model can be fine-tuned on many other tasks related to RNA structure and function.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1101/2023.03.15.532863

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 67

  1. 10.18653/v1/N19-1423 · DOI · Paper page
  2. 2020, in Advances in Neural Information Processing Systems, 33, 1877
  3. 10.1016/j.csbj.2021.03.022 · DOI
  4. 2022, Deep learning methods for 3D structural proteome and interactome modeling, Curr. Opin. Struct. Biol, 73, 102329, 10.1016/j.sbi.2022.102329 · DOI · Paper page
  5. 2019, Unified rational protein engineering with sequence-based deep representation learning, Nat. Methods, 16, 1315, 10.1038/s41592-019-0598-1 · DOI · Paper page
  6. 10.1093/bioinformatics/btaa003 · DOI
  7. 10.1073/pnas.2016239118 · DOI · Paper page
  8. 2019, Evaluating Protein Transfer Learning with TAPE, Adv. Neural Inf. Process. Syst, 32, 9689
  9. 10.1093/bioinformatics/btac020 · DOI · Paper page
  10. 10.1109/TPAMI.2021.3095381 · DOI · Paper page
  11. 2019, Evaluating protein transfer learning with TAPE, Adv. Neural Inf. Process. Syst, 32, 9689
  12. 10.1093/bioinformatics/btm098 · DOI · Paper page
  13. 2021, DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome, Bioinformatics, 37, 2112, 10.1093/bioinformatics/btab083 · DOI · Paper page
  14. 10.1007/978-3-031-13829-4_13 · DOI
  15. 10.1093/bib/bbac053 · DOI
  16. 2022, preMLI: a pre-trained method to uncover microRNA–lncRNA potential interactions, Brief. Bioinform, 23, bbab470, 10.1093/bib/bbab470 · DOI
  17. 2020, Learning distributed representations of RNA and protein sequences and its application for predicting lncRNA-protein interactions, Comput. Struct. Biotechnol. J, 18, 20, 10.1016/j.csbj.2019.11.004 · DOI
  18. Chen, J. et al. Interpretable RNA Foundation Model from Unannotated Data for Highly Accurate RNA Structure and Function Predictions. arxiv 1–23 (2022). · DOI · Paper page
  19. 10.1017/S0033583503003901 · DOI
  20. 2016, Novel function discovery through sequence and structural data mining, Curr. Opin. Struct. Biol, 38, 53, 10.1016/j.sbi.2016.05.017 · DOI
  21. 10.1261/rna.073015.119 · DOI
  22. 2021, in Proceedings of the 38th International Conference on Machine Learning, 139, 8844
  23. 10.1093/nar/25.17.3389 · DOI · Paper page
  24. 10.1261/rna.1556009 · DOI
  25. 10.1101/gr.5890907 · DOI
  26. Vasavada, M. , Byron, K. , Song, Y. & Wang, J. T. L. Genome-Wide Search for Pseudoknotted Noncoding RNA: A Comparative Study. in Pattern Recognition in Computational Molecular Biology 155–164 (2015). doi:https://doi.org/10.1002/9781119078845.ch9. · DOI
  27. 10.1093/bioinformatics/btt509 · DOI
  28. 10.1093/NAR/GKAA1047 · DOI
  29. 10.1093/bioinformatics/btab391 · DOI
  30. 2016, RNA folding with hard and soft constraints, Algorithms Mol. Biol, 11, 8, 10.1186/s13015-016-0070-z · DOI
  31. 10.1101/2022.10.03.510702 · DOI
  32. 10.1101/2023.02.01.526559 · DOI
  33. 10.1186/1471-2105-9-340 · DOI
  34. 10.1261/rna.053694.115 · DOI
  35. 10.1093/nar/gky285 · DOI
  36. 10.1038/s41467-019-13395-9 · DOI
  37. 2019, DMfold: A Novel Method to Predict RNA Secondary Structure With Pseudoknots Based on Deep Learning and Improved Base Pair Maximization Principle, Front. Genet, 10, 143, 10.3389/fgene.2019.00143 · DOI
  38. 2021, RNA secondary structure prediction using deep learning with thermodynamic integration, Nat. Commun, 12, 941, 10.1038/s41467-021-21194-4 · DOI
  39. 2022, UFold: fast and accurate RNA secondary structure prediction with deep learning, Nucleic Acids Res, 50, e14, 10.1093/nar/gkab1074 · DOI
  40. 2021, Improved RNA secondary structure and tertiary base-pairing prediction using evolutionary profile, mutational coupling and two-dimensional transfer learning, Bioinformatics, 37, 2589, 10.1093/bioinformatics/btab165 · DOI
  41. 10.1261/rna.057364.116 · DOI
  42. 2019, Enhanced prediction of RNA solvent accessibility with long short-term memory neural networks and improved sequence profiles, Bioinformatics, 35, 1686, 10.1093/bioinformatics/bty876 · DOI
  43. 2020, Single-sequence and profile-based prediction of RNA solvent accessibility using dilated convolutional neural network, Bioinformatics, 36, 5169
  44. Rao, R. , Meier, J. , Sercu, T. , Ovchinnikov, S. & Rives, A. Transformer protein language models are unsupervised structure learners. in International Conference on Learning Representations (2021). · DOI · Paper page
  45. 10.1093/nar/gkv932 · DOI
  46. Lorenz, R. ‘ViennaRNA Package 2.0’. Algorithms Mol. Biol. 6–26 (2011). · DOI · Paper page
  47. 2020, LinearPartition: linear-time approximation of RNA folding partition function and base-pairing probabilities, Bioinformatics, 36, i258, 10.1093/bioinformatics/btaa460 · DOI
  48. 10.1093/nar/gkq792 · DOI
  49. 2022, Ensemble deep learning: A review, Eng. Appl. Artif. Intell, 115, 105151, 10.1016/j.engappai.2022.105151 · DOI
  50. The RNAcentral Consortium, 2019, RNAcentral: a hub of information for non-coding RNA sequences, Nucleic Acids Res, 47, D221, 10.1093/nar/gky1034 · DOI
  51. 2021, Genome Warehouse: A Public Repository Housing Genome-scale Data, Genomics. Proteomics Bioinformatics, 19, 584, 10.1016/j.gpb.2021.04.001 · DOI
  52. 2020, MGnify: the microbiome analysis resource in 2020, Nucleic Acids Res, 48, D570
  53. 2022, Predicting RNA solvent accessibility from multi-scale context feature via multi-shot neural network, Anal. Biochem, 654, 114802, 10.1016/j.ab.2022.114802 · DOI
  54. 2021, in Proceedings of the 38th International Conference on Machine Learning, 139, 4651
  55. 2022, Deep learning models for RNA secondary structure prediction (probably) do not generalize across families, Bioinformatics, 38, 3892, 10.1093/bioinformatics/btac415 · DOI
  56. 10.1093/nar/gkw1000 · DOI
  57. 10.1093/bioinformatics/bts565 · DOI · Paper page
  58. 10.1093/bioinformatics/btz282 · DOI
  59. 10.1093/nar/gkv716 · DOI
  60. 10.1186/s12859-019-3019-7 · DOI · Paper page
  61. He, K. , Zhang, X. , Ren, S. & Sun, J. Deep Residual Learning for Image Recognition. in 2016 IEEE Conference on Computer Vision and Pattern Recognition (2016). · DOI · Paper page
  62. 10.1371/journal.pcbi.1005324 · DOI · Paper page
  63. 10.1093/nar/gkg601 · DOI
  64. 2012, in Advances in Neural Information Processing Systems, 25, 1097
  65. 10.1109/TPAMI.2019.2913372 · DOI · Paper page
  66. Vaswani, A. et al. Attention is All you Need. in Advances in Neural Information Processing Systems (eds. Guyon, I. et al. ) vol. 30 (Curran Associates, Inc., 2017).
  67. Loshchilov, I. & Hutter, F. SGDR: Stochastic Gradient Descent with Warm Restarts. in International Conference on Learning Representations (2017).

Metadata

Authors
Zhang, Y., Lang, M., Jiang, J., Gao, Z., Xu, F., Litfin, T., Chen, K., Singh, J., Huang, X., Song, G., Tian, Y., Zhan, J., Chen, J., & Zhou, Y.
Affiliations
  • Peking University (CN)
  • Shenzhen Bay Laboratory (CN)
  • Peng Cheng Laboratory (CN)
  • Griffith University (AU)
Corresponding author
  • Zhan, Jian
  • Chen, Jie
  • Zhou, Yaoqi
ORCID
Venue
Nucleic Acids Research
Publisher
openRxiv
Published
2023-03-16
Volume
52
Issue
1
Pages
e3-e3
ISSN
0305-1048 (ISSN-L)
DOI
10.1101/2023.03.15.532863
PMC
PMC10783488
OpenAlex
W4388539614
Cited by
112
References
1
Citations by year
2026: 24 · 2025: 60 · 2024: 25 · 2023: 3
FWCI
10.07
Type
journal article
Language
en
Open access
Licence
cc-by-nc
Locations
  • Nucleic Acids Research (publishedVersion, cc-by-nc)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
  • Griffith Research Online (submittedVersion, cc-by-nc)
Funders
  • Griffith University
  • National Natural Science Foundation of China (61972217, 62176249, 32071459, 62006133, 62271465, 22350710182)
  • National Key Research and Development Program of China (2021YFF1200400, 2022ZD0160100, 2022ZD0118201)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › RNA and protein synthesis mechanisms
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
Keywords
RNA language models, multiple sequence alignment, base pairing probability, solvent accessibility, attention maps, RNA secondary structure
MeSH terms
Machine Learning, DNA, Language, Proteins, RNA, Solvents, Sequence Alignment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review