The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.
MULAN: Multimodal Protein Language Model for Sequence and Structure Encoding
Open accessPreprint — published as 10.1093/bioadv/vbaf117 · Paper page
Abstract
Most protein language models (PLMs), which are used to produce high-quality protein representations, use only protein sequences during training. However, the known protein structure is crucial in many protein property prediction tasks, so there is a growing interest in incorporating the knowledge about the protein structure into a PLM. In this study, we propose MULAN, a MULtimodal PLM for both sequence and ANgle-based structure encoding. MULAN has a pre-trained sequence encoder and an introduced Structure Adapter, which are then fused and trained together. According to the evaluation on 7 downstream tasks of various nature, both small and medium-sized MULAN models show consistent improvement in quality compared to both sequence-only ESM-2 and structure-aware SaProt. Importantly, our model offers a cheap increase in the structural awareness of the protein representations due to finetuning of existing PLMs instead of training from scratch. We perform a detailed analysis of the proposed model and demonstrate its awareness of the protein structure. The implementation, training data and model checkpoints are available at https://github.com/DFrolova/MULAN .
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
Flagged first, then by citations
Research works printed but not linked: 35
- A. V. Finkelstein and O. Ptitsyn , Protein physics: a course of lectures. Elsevier, 2016. · DOI
- 10.1073/pnas.47.9.1309 · DOI · Paper page
- 2021, Prottrans: Toward understanding the language of life through self-supervised learning, IEEE transactions on pattern analysis and machine intelligence, 44, 7112
- 2022, Language models of protein sequences at the scale of evolution enable accurate structure prediction, BioRxiv, 2022, 500902
- A. Elnaggar , H. Essam , W. Salah-Eldin , W. Moustafa , M. Elkerdawy , C. Rochereau , and B. Rost , “Ankh: Optimized protein language model unlocks general-purpose modelling,” bioRxiv, pp. 2023–01, 2023. · DOI
- 10.1126/science.ade2574 · DOI · Paper page
- J. Su , C. Han , Y. Zhou , J. Shan , X. Zhou , and F. Yuan , “Saprot: Protein language modeling with structure-aware vocabulary,” bioRxiv, pp. 2023–10, 2023. · DOI
- 10.1093/bioinformatics/btt473 · DOI · Paper page
- 2018, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint
- 10.1093/NAR/GKAB1061 · DOI · Paper page
- 10.1073/pnas.2016239118 · DOI · Paper page
- M. Heinzinger , K. Weissenow , J. G. Sanchez , A. Henkel , M. Steinegger , and B. Rost , “Prostt5: Bilingual language model for protein sequence and structure,” bioRxiv, pp. 2023–07, 2023. · DOI · Paper page
- P. Gonzalez , T. S. Alstrøm , and T. May , “On batching variable size inputs for training end-to-end speech enhancement systems,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5. · DOI
- R. Rao , N. Bhattacharya , N. Thomas , Y. Duan , P. Chen , J. Canny , P. Abbeel , and Y. Song , “Evaluating protein transfer learning with tape,” Advances in neural information processing systems, vol. 32, 2019. · DOI · Paper page
- 2021, Structure-based protein function prediction using graph convolutional networks, Nature communications, 12, 3168, 10.1038/s41467-021-23303-9 · DOI
- 2021, Light attention predicts protein location from the language of life, Bioinformatics Advances, 1, vbab035, 10.1093/bioadv/vbab035 · DOI · Paper page
- 2018, Critical assessment of methods of protein structure prediction (casp)—round xii, Proteins: Structure, Function, and Bioinformatics, 86, 7, 10.1002/prot.25415 · DOI
- 10.1093/bib/bbw129 · DOI
- 1999, Evaluation and improvement of multiple sequence methods for protein secondary structure prediction, Proteins: Structure, Function, and Bioinformatics, 34, 508, 10.1002/(SICI)1097-0134(19990301)34:4<508::AID-PROT10>3.0.CO;2-4 · DOI
- L. Van der Maaten and G. Hinton , “Visualizing data using t-sne.” Journal of machine learning research, vol. 9, no. 11, 2008.
- 2024, Structure-informed protein language model, arXiv preprint
- 10.1002/j.1460-2075.1986.tb04288.x · DOI · Paper page
- M. van Kempen , S. S. Kim , C. Tumescheit , M. Mirdita , J. Lee , C. L. Gilchrist , J. Söding , and M. Steinegger , “Fast and accurate protein structure search with foldseek,” Nature Biotechnology, pp. 1–4, 2023. · DOI · Paper page
- S. Mansoor , M. Baek , U. Madan , and E. Horvitz , “Toward more general embeddings for protein design: Harnessing joint representations of sequence and structure,” bioRxiv, pp. 2021–09, 2021. · DOI
- Z. Zheng , Y. Deng , D. Xue , Y. Zhou , F. Ye , and Q. Gu , “Structure-informed language models are protein designers,” bioRxiv, 2023. [Online]. Available: https://www.biorxiv.org/content/early/2023/02/03/2023.02.03.526917
- 2022, Lm-gvp: an extensible sequence and structure informed deep learning framework for protein property prediction, Scientific reports, 12, 6832, 10.1038/s41598-022-10775-y · DOI
- 2023, Masked inverse folding with sequence transfer for protein representation learning, Protein Engineering, Design and Selection, 36, gzad015, 10.1093/protein/gzad015 · DOI
- Z. Zhang , C. Wang , M. Xu , V. Chenthamarakshan , A. Lozano , P. Das , and J. Tang , “A systematic study of joint representation learning on protein sequences and structures,” Preprint at http://arxiv.org/abs/2303.06275, 2023. · arXiv
- 2017, Decoupled weight decay regularization, arXiv preprint
- C. Dallago , J. Mou , K. E. Johnston , B. J. Wittmann , N. Bhattacharya , S. Goldman , A. Madani , and K. K. Yang , “Flip: Benchmark tasks in fitness landscape inference for proteins,” bioRxiv, 2021. [Online]. Available: https://www.biorxiv.org/content/early/2021/11/11/2021.11.09.467890
- 2020, Meltome atlas—thermal proteome stability across the tree of life, Nature methods, 17, 495, 10.1038/s41592-020-0801-4 · DOI
- 10.1021/pr100618t · DOI
- 2022, Peer: a comprehensive and multi-task benchmark for protein sequence understanding, Advances in Neural Information Processing Systems, 35, 35 156
- 10.1038/nature17995 · DOI · Paper page
- 1999, Ab initio protein structure prediction of casp iii targets using rosetta, Proteins: Structure, Function, and Bioinformatics, 37, 171, 10.1002/(SICI)1097-0134(1999)37:3+<171::AID-PROT21>3.0.CO;2-Z · DOI
Metadata
- Authors
- Frolova, D., Pak, M. A., Litvin, A., Sharov, I., Ivankov, D. N., & Oseledets, I.
- Affiliations
- Skolkovo Institute of Science and Technology (RU)
- Corresponding author
- Frolova, Daria
- Venue
- Bioinformatics Advances
- Publisher
- openRxiv
- Published
- 2024-06-02
- Volume
- 5
- Issue
- 1
- Pages
- vbaf117-vbaf117
- ISSN
- 2635-0041 (ISSN-L)
- PMC
- PMC12452268
- OpenAlex
- W4410572064
- Cited by
- 12
- References
- 1
- Citations by year
- 2026: 8 · 2025: 3 · 2024: 1
- FWCI
- 1.63
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Bioinformatics Advances (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Drug Discovery › Computational Drug Discovery Methods
- Keywords
- protein language models, protein-protein interaction prediction, protein sequence encoding, ESM-2, pre-trained model fine-tuning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
