An Analysis of Protein Language Model Embeddings for Fold Prediction
Open accessPreprint — published as 10.1093/bib/bbac142 · Paper page
The DOI matches its Crossref record; 2 of the 2 references it cites were found in a registry or an index.
Abstract
The identification of the protein fold class is a challenging problem in structural biology. Recent computational methods for fold prediction leverage deep learning techniques to extract protein fold-representative embeddings mainly using evolutionary information in the form of multiple sequence alignment (MSA) as input source. In contrast, protein language models (LM) have reshaped the field thanks to their ability to learn efficient protein representations (protein-LM embeddings) from purely sequential information in a self-supervised manner. In this paper, we analyze a framework for protein fold prediction using pre-trained protein-LM embeddings as input to several fine-tuning neural network models which are supervisedly trained with fold labels. In particular, we compare the performance of six protein-LM embeddings: the LSTM-based UniRep and SeqVec, and the transformer-based ESM-1b, ESM-MSA, ProtBERT, and ProtT5; as well as three neural networks: Multi-Layer Perceptron (MLP), ResCNN-BGRU (RBG), and Light-Attention (LAT). We separately evaluated the pairwise fold recognition (PFR) and direct fold classification (DFC) tasks on well-known benchmark datasets. The results indicate that the combination of transformer-based embeddings, particularly those obtained at amino acid-level, with the RBG and LAT fine-tuning models performs remarkably well in both tasks. To further increase prediction accuracy, we propose several ensemble strategies for PFR and DFC, which provide a significant performance boost over the current state-of-the-art results. All this suggests that moving from traditional protein representations to protein-LM embeddings is a very promising approach to protein fold-related tasks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
Modeling aspects of the language of life through transfer-learning protein sequences
Heinzinger, M., Elnaggar, A., Wang, Y., Dallago, C., Nechaev, D., Matthes, F., & Rost, B. (2019) · BMC Bioinformatics · doi:10.1186/s12859-019-3220-8 · Cited by 606 · cited 4×
“UniRep [71] and SeqVec [73] are two protein-LMs trained using recurrent layers with long short-term memory (LSTM) [69] units.”
Influential
Research works printed but not linked: 101
- 2020, Improved protein structure prediction using potentials from deep learning, Nature, 577, 706, 10.1038/s41586-019-1923-7 · DOI · Paper page
- 10.1126/science.abj8754 · doi:10.1126/science.abj8754 · Paper page
- Mihaly Varadi , Stephen Anyango , Mandar Deshpande , Sreenath Nair , Cindy Natassia , Galabina Yordanova , David Yuan , Oana Stroe , Gemma Wood , Agata Laydon , et al. AlphaFold Protein Structure Database: massively expanding the structural coverage of protein-sequence space with high-accuracy models. Nucleic Acids Research, 2021. · DOI · Paper page
- 10.1006/jmbi.1995.0159 · DOI
- John-Marc Chandonia , Lindsey Guan , Shiangyi Lin , Changhua Yu , Naomi K. Fox , and Steven E. Brenner . SCOPe: improvements to the structural classification of proteins–extended database to facilitate variant interpretation and machine learning. Nucleic Acids Research, 2021. · DOI · Paper page
- 10.1016/S0969-2126(97)00260-8 · DOI · Paper page
- 2021, CATH: increased structural coverage of functional space, Nucleic acids research, 49, D266, 10.1093/nar/gkaa1079 · DOI · Paper page
- 10.1093/nar/28.1.235 · DOI · Paper page
- 2021, RCSB Protein Data Bank: powerful new tools for exploring 3d structures of biological macromolecules for basic and applied research and education in fundamental biology, biomedicine, biotechnology, bioengineering and energy sciences, Nucleic Acids Research, 49, D437, 10.1093/nar/gkaa1038 · DOI · Paper page
- 10.1146/annurev.bi.59.070190.005043 · DOI · Paper page
- 10.1038/358086a0 · DOI · Paper page
- 10.1006/jmbi.1999.3377 · DOI · Paper page
- 2010, Protein folds and protein folding, Protein Engineering, Design & Selection, 24, 11
- 2013, On the universe of protein folds, Annual Review of Biophysics, 42, 559, 10.1146/annurev-biophys-083012-130432 · DOI
- 2008, A survey of protein fold recognition algorithms, Journal of Computer Science, 4, 768, 10.3844/jcssp.2008.768.776 · DOI
- 2018, A comprehensive review and comparison of different computational methods for protein remote homology detection, Briefings in Bioinformatics, 19, 231, 10.1093/bib/bbw108 · DOI · Paper page
- 2019, In Machine Learning Paradigms, 149, 101, 10.1007/978-3-319-94030-4_5 · DOI
- 10.1093/nar/25.17.3389 · DOI · Paper page
- 10.1093/bioinformatics/bti125 · DOI · Paper page
- 2014, MRFalign: Protein homology detection through alignment of Markov random fields, PLoS Computational Biology, 10, e1003500, 10.1371/journal.pcbi.1003500 · DOI · Paper page
- 10.1142/S0219720003000186 · DOI · Paper page
- Jian Peng and Jinbo Xu . Boosting protein threading accuracy. In Annual International Conference on Research in Computational Molecular Biology, pages 31–45, 2009. · DOI
- 10.1093/bioinformatics/btr350 · DOI · Paper page
- 10.1093/bioinformatics/bts213 · DOI
- 2018, Protein alignment based on higher order conditional random fields for template-based modeling, PLoS ONE, 13, e0197912, 10.1371/journal.pone.0197912 · DOI
- 10.1093/bioinformatics/btx217 · DOI · Paper page
- 2019, Detecting distant-homology protein structures by aligning deep neural-network based contact maps, PLoS Computational Biology, 15, 1
- 10.1093/bioinformatics/btl102 · DOI · Paper page
- 10.1186/1471-2105-15-S9-S14 · DOI
- 2015, Improving protein fold recognition by deep learning networks, Scientific Reports, 5, 17573, 10.1038/srep17573 · DOI · Paper page
- 2019, Protein fold recognition based on multi-view modeling, Bioinformatics, 35, 2982, 10.1093/bioinformatics/btz040 · DOI
- Ke Yan , Jie Wen an Yong Xu , and Bin Liu . Protein fold recognition based on auto-weighted multi-view graph embedding learning model. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 2020. · DOI
- 2021, MLDH-Fold: Protein fold recognition based on multi-view low-rank modeling, Neurocomputing, 421, 127, 10.1016/j.neucom.2020.09.028 · DOI
- Bin Liu, Yulin Zhu , and Ke Yan . Fold-LTR-TCP: protein fold recognition based on triadic closure principle. Briefings in Bioinformatics, 2019. · DOI
- Jiangyi Shao , Ke Yan , and Bin Liu . FoldRec-C2C: protein fold recognition by combining cluster-to-cluster model and protein similarity network. Briefings in Bioinformatics, 2020. · DOI
- Jiangyi Shao and Bin Liu . ProtFold-DFG: protein fold recognition by combining Directed Fusion Graph and PageRank algorithm. Briefings in Bioinformatics, 2020. · DOI
- 2017, Improving protein fold recognition by extracting fold-specific features from predicted residue–residue contacts, Bioinformatics, 33, 3749, 10.1093/bioinformatics/btx514 · DOI
- Bin Liu, Chen-Chen Li , and Ke Yan . DeepSVM-fold: protein fold recognition by combining support vector machines and pairwise sequence similarity scores generated by deep learning networks. Briefings in Bioinformatics, 2019. · DOI
- Yihe Pang and Bin Liu . SelfAT-Fold: protein fold recognition based on residue-based and motif-based self-attention networks. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 2020. · DOI
- Yan Liu , Yi-Heng Zhu , Xiaoning Song , Jiangning Song , and Dong-Jun Yu . Why can deep convolutional neural networks improve protein fold recognitionã a visual explanation by interpretation. Briefings in Bioinformatics, 2021. · DOI
- 2021, Improving protein fold recognition using triplet network and ensemble deep learning, Briefings in Bioinformatics, 22, bbab248, 10.1093/bib/bbab248 · DOI
- 2021, Protein fold recognition from sequences using convolutional and recurrent neural networks, IEEE/ACM Transactions on Computational Biology and Bioinformatics, 18, 2848, 10.1109/TCBB.2020.3012732 · DOI
- 10.1186/s12859-021-04034-6 · DOI · Paper page
- 2016, Recent progress in machine learning-based methods for protein fold recognition, International journal of Molecular Sciences, 17, 2118, 10.3390/ijms17122118 · DOI · Paper page
- 10.1093/bioinformatics/17.4.349 · DOI · Paper page
- 10.1093/bioinformatics/btl170 · DOI · Paper page
- 10.1093/bioinformatics/btp500 · DOI · Paper page
- 2011, Improving taxonomy-based protein fold recognition by using global and local features, Proteins: Structure, Function, and Bioinformatics, 79, 2053, 10.1002/prot.23025 · DOI
- 2015, Advancing the accuracy of protein fold recognition by utilizing profiles from hidden Markov models, IEEE Transactions on Nanobioscience, 14, 761, 10.1109/TNB.2015.2457906 · DOI
- Daozheng Chen , Xiaoyu Tian , Bo Zhou , and Jun Gao . ProFold: Protein fold classification with additional structural features and a novel ensemble classifier. BioMed Research International, 2016:1–10, 2016. · DOI
- 2016, An ensemble approach to protein fold classification by integration of template-based assignment and support vector machine classifier, Bioinformatics, 33, 863
- 2019, Protein fold recognition using deep kernelized extreme learning machine and linear discriminant analysis, Neural Computing and Applications, 31, 4201, 10.1007/s00521-018-3346-z · DOI
- 2020, An enhanced protein fold recognition for low similarity datasets using convolutional and skip-gram features with deep neural network, IEEE Transactions on NanoBioscience, 20, 42
- 10.1186/s12859-020-3475-0 · DOI · Paper page
- 10.1093/bioinformatics/btx780 · DOI · Paper page
- 2020, Progress in neural nlp: Modeling, learning, and reasoning, Engineering, 6, 275, 10.1016/j.eng.2019.12.014 · DOI · Paper page
- 2021, The language of proteins: Nlp, machine learning & protein sequences, Computational and Structural Biotechnology Journal, 19, 1750, 10.1016/j.csbj.2021.03.022 · DOI · Paper page
- 10.1093/nar/gkg095 · DOI · Paper page
- 10.1093/NAR/GKAA913 · DOI · Paper page
- 10.1093/bioinformatics/btu739 · DOI · Paper page
- 10.1093/nar/gkaa1100 · DOI · Paper page
- 2018, Clustering huge protein sequence sets in linear time, Nature Communications, 9, 1
- 2019, Protein-level assembly increases protein sequence recovery from metagenomic samples manyfold, Nature Methods, 16, 603, 10.1038/s41592-019-0437-4 · DOI · Paper page
- 10.1371/journal.pone.0141287 · DOI · Paper page
- Tomas Mikolov , Kai Chen , Greg Corrado , and Jeffrey Dean . Efficient estimation of word representations in vector space. In International Conference on Learning Representations, 2013.
- Tomas Mikolov , Ilya Sutskever , Kai Chen , Greg Corrado , and Jeff Dean . Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111–3119, 2013.
- 10.1162/neco.1997.9.8.1735 · DOI · Paper page
- 10.1093/bioinformatics/btaa003 · DOI · Paper page
- 2019, Unified rational protein engineering with sequence-based deep representation learning, Nature Methods, 16, 1315, 10.1038/s41592-019-0598-1 · DOI · Paper page
- 2018, Deep contextualized word representations, arXiv preprint
- 10.1186/s12859-019-2669-9 · DOI · Paper page
- Tristan Bepler and Bonnie Berger . Learning protein sequence embeddings using information from structure. In International Conference on Learning Representations, 2019.
- 2017, Attention is all you need, In Advances in Neural Information Processing Systems, 30, 5998
- Roshan Rao , Nicholas Bhattacharya , Neil Thomas , Yan Duan , Xi Chen , John Canny , Pieter Abbeel , and Yun S. Song . Evaluating protein transfer learning with TAPE. In Advances in neural information processing systems, 2019.
- Alexander Rives , Joshua Meier , Tom Sercu , Siddharth Goyal , Zeming Lin , Jason Liu , Demi Guo , Myle Ott , C. Lawrence Zitnick , Jerry Ma , et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 118(15), 2021. · DOI · Paper page
- Roshan M. Rao , Jason Liu , Robert Verkuil , Joshua Meier , John Canny , Pieter Abbeel , Tom Sercu , and Alexander Rives . Msa transformer. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8844–8856, 2021.
- Ahmed Elnaggar , Michael Heinzinger , Christian Dallago , Ghalia Rehawi , Wang Yu , Llion Jones , Tom Gibbs , Tamas Feher , Christoph Angerer , Martin Steinegger , Debsindhu Bhowmik , and Burkhard Rost . ProtTrans: Towards cracking the language of lifes code through self-supervised deep learning and high performance computing. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–16, 2021. · DOI · Paper page
- 2018, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint
- Zihang Dai , Zhilin Yang , Yiming Yang , Jaime Carbonell , Quoc V. Le , and Ruslan Salakhutdinov . Transformer-XL: attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, page 2978–2988, 2019.
- Zhilin Yang , Zihang Dai , Yiming Yang , Jaime Carbonell , Ruslan Salakhutdinov , and Quoc V. Le . XLNet: generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems, 2019.
- 2019, Exploring the limits of transfer learning with a unified text-to-text transformer, arXiv preprint
- Zhenzhong Lan , Mingda Chen , Sebastian Goodman , Kevin Gimpel , Piyush Sharma , and Radu Soricut . ALBERT: a lite BERT for self-supervised learning of language representations. In International Conference on Learning Representations, 2020.
- Kevin Clark , Minh-Thang Luong , Quoc V. Le , and Christopher D. Manning . ELECTRA: pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations, 2020.
- 2021, Light attention predicts protein location from the language of life, Bioinformatics Advances, 11, vbab035
- 2021, Unsupervised protein embeddings outperform hand-crafted sequence and structure features at predicting molecular function, Bioinformatics, 37, 162, 10.1093/bioinformatics/btaa701 · DOI · Paper page
- 2021, Embeddings from deep learning transfer GO annotations beyond homology, Scientific reports, 11, 1
- 2021, The power of universal contextualized protein embeddings in cross-species protein function prediction, Evolutionary Bioinformatics, 17, 1
- Joshua Meier , Roshan Rao , Robert Verkuil , Jason Liu , Tom Sercu , and Alexander Rives . Language models enable zero-shot prediction of the effects of mutations on protein function. In Advances on Neural Information Processing Systems, 2021. · DOI · Paper page
- Celine Marquet , Michael Heinzinger , Tobias Olenyi , Christian Dallago , Michael Bernhofer , Kyra Erckert , Dmitrii Nechaev , and Burkhard Rost . Embeddings from protein language models predict conservation and variant effects. Research Square preprint, 2021. · DOI · Paper page
- Jesse Vig , Ali Madani , Lav R. Varshney , Caiming Xiong , Richard Socher , and Nazneen Rajani . BERTology meets biology: Interpreting attention in protein language models. In International Conference on Learning Representations, 2021. · DOI · Paper page
- 10.1186/s12859-019-2669-9 · DOI · Paper page
- 10.1093/nar/gkw1081 · DOI · Paper page
- 2016, Multiplicative lstm for sequence modelling, arXiv preprint
- 1997, Bidirectional recurrent neural networks, IEEE Transactions on Signal Processing, 45, 2673, 10.1109/78.650093 · DOI · Paper page
- 2019, Axial attention in multidimensional transformers, arXiv preprint
- Sergey Ioffe and Christian Szegedy . Batch normalization: Accelerating deep network training by reducing internal covariate shift. International Conference on Machine Learning, pages 448–456, 2015.
- 2014, Dropout: a simple way to prevent neural networks from overfitting, The journal of Machine Learning Research, 15, 1929
- 2014, Empirical evaluation of gated recurrent neural networks on sequence modeling, arXiv preprint
- Hao Wang , Yitong Wang , Zheng Zhou , Xing Ji , Dihong Gong , Jingchao Zhou , Zhifeng Li , and Wei Liu . CosFace: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5265–5274, 2018. · DOI · Paper page
- 2014, Adam: A method for stochastic optimization, arXiv preprint
- 10.1006/jmbi.1990.9999 · doi:10.1006/jmbi.1990.9999 · Paper page
Metadata
- Authors
- Villegas-Morcillo, A., Gomez, A. M., & Sanchez, V.
- Affiliations
- Universidad de Granada (ES)
- Corresponding author
- Villegas-Morcillo, Amelia
- Venue
- Briefings in Bioinformatics
- Publisher
- openRxiv
- Published
- 2022-02-10
- Volume
- 23
- Issue
- 3
- ISSN
- 1467-5463 (ISSN-L)
- OpenAlex
- W4225891318
- Cited by
- 46
- References
- 2
- Citations by year
- 2026: 5 · 2025: 13 · 2024: 15 · 2023: 12 · 2022: 1
- FWCI
- 3.35
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Licence
- cc-by-nc-nd
- Locations
- Briefings in Bioinformatics (publishedVersion)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Institutional Repository of the University of Granada (University of Granada) (submittedVersion, cc-by-nc-nd)
- Funders
- Family Process Institute (BES-2017-079792)
- Ministerio de Ciencia e Innovación (PID2019-104206GB-I00)
- Agencia Estatal de Investigación (10.13039/501100011033)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Structural Biology › Enzyme Structure and Function
- Keywords
- protein fold prediction, protein language models, protein embeddings, Earth system models, ProtBERT, ProtT5, ensemble learning, fine-tuning, protein representation learning
- MeSH terms
- Language, Natural Language Processing, Proteins, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: An Analysis of Protein Language Model Embeddings for Fold Prediction. https://app.cochat.ai/papers/10.1101/2022.02.07.479394
- The paper
- Villegas-Morcillo, A., Gomez, A. M., & Sanchez, V. (2022). An Analysis of Protein Language Model Embeddings for Fold Prediction. Briefings in Bioinformatics. https://doi.org/10.1101/2022.02.07.479394
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.