ProteinGLUE: A multi-task benchmark suite for self-supervised protein modeling
Open accessPreprint — published as 10.1038/s41598-022-19608-4 · Paper page
The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.
Abstract
Self-supervised language modeling is a rapidly developing approach for the analysis of protein sequence data. However, work in this area is heterogeneous and diverse, making comparison of models and methods difficult. Moreover, models are often evaluated only on one or two downstream tasks, making it unclear whether the models capture generally useful properties. We introduce the ProteinGLUE benchmark for the evaluation of protein representations: a set of seven tasks for evaluating learned protein representations. We also offer reference code, and we provide two baseline models with hyperparameters specifically trained for these benchmarks. Pre-training was done on two tasks, masked symbol prediction and next sentence prediction. We show that pre-training yields higher performance on a variety of downstream tasks such as secondary structure and protein interaction interface prediction, compared to no pre-training. However, the larger base model does not outperform the smaller medium. We expect the ProteinGLUE benchmark dataset introduced here, together with the two baseline pre-trained models and their performance evaluations, to be of great value to the field of protein sequence-based property prediction. Availability: code and datasets from https://github.com/ibivu/protein-glue
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
Research works printed but not linked: 64
- 2019, Netsurfp-2.0: Improved prediction of protein structural features by integrated deep learning, Proteins: Struct. Funct. Bioinforma, 87, 520, 10.1002/prot.25674 · DOI · Paper page
- 10.1093/bioinformatics/btaa629 · DOI · Paper page
- 10.1073/pnas.1907373117 · DOI · Paper page
- 2019, Improving multi-task deep neural networks via knowledge distillation for natural language understanding, arXiv preprint
- 2018, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint
- 2019, Language models are unsupervised multitask learners, OpenAI blog, 1, 9
- 2019, Evaluating protein transfer learning with tape, Adv. neural information processing systems, 32, 9689
- Rives, A. et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proc. Natl. Acad. Sci. 118 (2021). · DOI · Paper page
- 2020, Progen: Language modeling for protein generation, arXiv preprint
- 2020, Prottrans: towards cracking the language of life’s code through self-supervised deep learning and high performance computing, arXiv preprint
- 2018, Glue: A multi-task benchmark and analysis platform for natural language understanding, arXiv preprint
- 10.1073/pnas.37.4.205 · DOI · Paper page
- 10.1002/bip.360221211 · DOI · Paper page
- 10.1093/bioinformatics/btx005 · DOI
- 10.1006/jmbi.1998.2144 · DOI · Paper page
- 10.1093/bioinformatics/btz428 · DOI
- Potocnakova, L. , Bhide, M. & Pulzova, L. B. An introduction to b-cell epitope mapping and in silico epitope prediction. J. immunology research 2016 (2016). · DOI · Paper page
- 2021, Serendip-ce: sequence-based interface prediction for conformational epitopes, Bioinformatics, 37, 3421, 10.1093/bioinformatics/btab321 · DOI
- 10.1021/bi00327a032 · DOI · Paper page
- Gils, J. et al. How sticky are our proteins?: Quantifying hydrophobicity of the human proteome. arXiv.org (2021).
- 10.1093/NAR/GKAA913 · DOI · Paper page
- 2020, On the effect of dropping layers of pre-trained transformer models, arXiv preprint
- 2021, Large-scale news classification using bert language model: Spark nlp approach, In 6th International Conference on Sustainable Information Engineering and Technology, 2021, 240
- Blum, A. & Mitchell, T. Combining labeled and unlabeled data with co-training. In Proceedings of the eleventh annual conference on Computational learning theory, 92–100 (1998). · DOI · Paper page
- 2009, Semi-supervised learning (chapelle, o. et al., eds.; 2006)[book reviews], IEEE Transactions on Neural Networks, 20, 542, 10.1109/TNN.2009.2015974 · DOI · Paper page
- 10.1109/TIT.1965.1053799 · DOI · Paper page
- 2018, Universal language model fine-tuning for text classification, arXiv preprint
- Peters, M. E. et al. Deep contextualized word representations. In Proceedings of NAACL-HLT, 2227–2237 (2018).
- 10.1101/2021.09.03.458832 · doi:10.1101/2021.09.03.458832
- Vaswani, A. et al. Attention is all you need. In Advances in neural information processing systems, 5998–6008 (2017).
- Brown, T. B. et al. Language models are few-shot learners. In Larochelle, H. , Ranzato, M. , Hadsell, R. , Balcan, M. & Lin, H. (eds.) Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual (2020).
- 10.1093/bib/bbw129 · DOI · Paper page
- 10.1093/bioinformatics/btaa524 · DOI · Paper page
- 10.7717/peerj.7126 · DOI · Paper page
- 2019, Unified rational protein engineering with sequence-based deep representation learning, Nat. methods, 16, 1315, 10.1038/s41592-019-0598-1 · DOI · Paper page
- 10.1186/s12859-019-2669-9 · DOI · Paper page
- 10.1093/bioinformatics/btaa003 · DOI · Paper page
- 2019, Learning protein sequence embeddings using information from structure, arXiv preprint
- 2021, Pre-training of deep bidirectional protein sequence representations with structural information, IEEE Access, 9, 123912, 10.1109/ACCESS.2021.3110269 · DOI · Paper page
- Rives, A. et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proc. Natl. Acad. Sci. 118 (2021). · DOI · Paper page
- 10.1093/nar/25.17.3389 · DOI · Paper page
- 10.1093/nar/gkaa1100 · DOI · Paper page
- 2018, Clustering huge protein sequence sets in linear time, Nat. communications, 9, 1
- 10.1038/s41592-019-0437-4 · DOI · Paper page
- 2020, Bertology meets biology: Interpreting attention in protein language models, arXiv preprint
- 10.1093/bioinformatics/bty481 · DOI · Paper page
- 10.1101/2021.02.12.430858 · DOI · Paper page
- 10.1093/nar/gkt1240 · DOI · Paper page
- 2018, Deep generative models of genetic variation capture the effects of mutations, Nat. methods, 15, 816, 10.1038/s41592-018-0138-4 · DOI · Paper page
- 10.1016/j.cels.2017.11.003 · DOI · Paper page
- 10.1093/bioinformatics/btx431 · DOI · Paper page
- 10.1093/nar/gkv485 · DOI · Paper page
- 1999, Evaluation and improvement of multiple sequence methods for protein secondary structure prediction, Proteins: Struct. Funct. Bioinforma, 34, 508, 10.1002/(SICI)1097-0134(19990301)34:4<508::AID-PROT10>3.0.CO;2-4 · DOI · Paper page
- 2019, Critical assessment of methods of protein structure prediction (casp)—round xiii, Proteins: Struct. Funct. Bioinforma, 87, 1011, 10.1002/prot.25823 · DOI · Paper page
- 10.1093/bioinformatics/bty1006 · DOI · Paper page
- 10.1093/bioinformatics/btg224 · DOI · Paper page
- 10.1093/protein/1.3.159 · DOI · Paper page
- 10.1186/s12859-014-0430-y · DOI · Paper page
- 10.1093/bioinformatics/btq302 · DOI · Paper page
- 10.1093/nar/gkt1043 · DOI · Paper page
- 2019, Research for practice: troubling trends in machine-learning scholarship, Commun. ACM, 62, 45, 10.1145/3316774 · DOI
- 2019, Large batch optimization for deep learning: Training bert in 76 minutes, arXiv preprint
- Hendrycks, D. & Gimpel, K. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR abs/1606.08415 (2016). 1606.08415. · arXiv
- 2016, A medium-scale distributed system for computer science research: Infrastructure for the long term, Computer, 49, 54, 10.1109/MC.2016.127 · DOI
Metadata
- Authors
- Capel, H., Weiler, R., Dijkstra, M., Vleugels, R., Bloem, P., & Feenstra, K. A.
- Affiliations
- Vrije Universiteit Amsterdam (NL)
- Corresponding author
- Feenstra, K. Anton
- Venue
- Scientific Reports
- Publisher
- openRxiv
- Published
- 2021-12-14
- Volume
- 12
- Issue
- 1
- Pages
- 16047-16047
- ISSN
- 2045-2322 (ISSN-L)
- PMC
- PMC9512797
- OpenAlex
- W4297243376
- Cited by
- 11
- References
- 1
- Citations by year
- 2026: 1 · 2025: 3 · 2024: 3 · 2023: 2 · 2022: 2
- FWCI
- 0.82
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Scientific Reports (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc)
- Research Square (acceptedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- VU Research Portal (publishedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Drug Discovery › Computational Drug Discovery Methods
- Keywords
- protein representation, benchmark suite, next sentence prediction, pre-training, secondary structure prediction, multi-task learning
- MeSH terms
- Amino Acid Sequence, Amino Acids, Natural Language Processing, Proteins, Benchmarking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: ProteinGLUE: A multi-task benchmark suite for self-supervised protein modeling. https://app.cochat.ai/papers/10.1101/2021.12.13.472460
- The paper
- Capel, H., Weiler, R., Dijkstra, M., Vleugels, R., Bloem, P., & Feenstra, K. A. (2021). ProteinGLUE: A multi-task benchmark suite for self-supervised protein modeling. Scientific Reports. https://doi.org/10.1101/2021.12.13.472460
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.