The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.
DistilProtBert: A distilled protein language model used to distinguish between real proteins and their randomly shuffled counterparts
Open accessPreprint — published as 10.1093/bioinformatics/btac474 · Paper page
Abstract
Summary: Recently, Deep Learning models, initially developed in the field of Natural Language Processing (NLP), were applied successfully to analyze protein sequences. A major drawback of these models is their size in terms of the number of parameters needed to be fitted and the amount of computational resources they require. Recently, “distilled” models using the concept of student and teacher networks have been widely used in NLP. Here, we adapted this concept to the problem of protein sequence analysis, by developing DistilProtBert, a distilled version of the successful ProtBert model. Implementing this approach, we reduced the size of the network and the running time by 50%, and the computational resources needed for pretraining by 98% relative to ProtBert model. Using two published tasks, we showed that the performance of the distilled model approaches that of the full model. We next tested the ability of DistilProtBert to distinguish between real and random protein sequences. The task is highly challenging if the composition is maintained on the level of singlet, doublet and triplet amino acids. Indeed, traditional machine learning algorithms have difficulties with this task. Here, we show that DistilProtBert preforms very well on singlet, doublet, and even triplet-shuffled versions of the human proteome, with AUC of 0.92, 0.91, and 0.87 respectively. Finally, we suggest that by examining the small number of false-positive classifications (i.e., shuffled sequences classified as proteins by DistilProtBert) we may be able to identify de-novo potential natural-like proteins based on random shuffling of amino acid sequences. Availability: https://github.com/yarongef/DistilProtBert Contact: yaron.geffen@biu.ac.il
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
Flagged first, then by citations
Research works printed but not linked: 26
- 2019, Unified rational protein engineering with sequence-based deep representation learning, Nature Methods, 16, 1315, 10.1038/s41592-019-0598-1 · DOI · Paper page
- 10.1093/bioinformatics/btx431 · DOI
- 10.1126/science.abj8754 · DOI · Paper page
- Brandes, N. , Ofer, D. , Peleg, Y. , Rappoport, N. , & Linial, M. (2022). ProteinBERT: a universal deep-learning model of protein sequence and function. Bioinformatics. https://doi.org/10.1093/bioinformatics/btac020 · DOI
- 10.1002/(SICI)1097-0134(19990301)34:4508::AID-PROT103.0.CO;2-4 · DOI
- Devlin, J. , Chang, M.-W. , Lee, K. , & Toutanova, K. (2018). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv Prepr arXiv181004805.
- Elnaggar, A. , Heinzinger, M. , Dallago, C. , Rehawi, G. , Wang, Y. , Jones, L. , Gibbs, T. , Feher, T. , Angerer, C. , Steinegger, M. , Bhowmik, D. , & Rost, B. (2021). ProtTrans: Towards Cracking the Language of Lifes Code Through Self-Supervised Deep Learning and High Performance Computing. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1–1. https://doi.org/10.1109/TPAMI.2021.3095381 · DOI · Paper page
- Hinton, G. , Vinyals, O. , & Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv Prepr arXiv181004805.
- Jiao, X. , Yin, Y. , Shang, L. , Jiang, X. , Chen, X. , Li, L. , Wang, F. , & Liu, Q. (2019). TinyBERT: Distilling BERT for Natural Language Understanding. arXiv Prepr arXiv190910351. · DOI · Paper page
- Sønderby, 2019, NetSurfP□2.0: Improved prediction of protein structural features by integrated deep learning, Proteins: Structure, Function, and Bioinformatics, 87, 520, 10.1002/prot.25674 · DOI
- Lan, Z. , Chen, M. , Goodman, S. , Gimpel, K. , Sharma, P. , & Soricut, R. (2019). ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. arXiv Prepr arXiv190911942.
- Liu, Y. , Ott, M. , Goyal, N. , Du, J. , Joshi, M. , Chen, D. , Levy, O. , Lewis, M. , Zettlemoyer, L. , & Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv Prepr arXiv190711692.
- 2018, Critical assessment of methods of protein structure prediction (CASP)-Round XII, Proteins: Structure, Function, and Bioinformatics, 86, 7, 10.1002/prot.25415 · DOI
- 2021, The language of proteins: NLP, machine learning & amp; protein sequences, Computational and Structural Biotechnology Journal, 19, 1750, 10.1016/j.csbj.2021.03.022 · DOI
- 10.1002/prot.10559 · DOI
- 2019, Evaluating Protein Transfer Learning with TAPE, Advances in Neural Information Processing Systems, 32, 9689
- 10.1073/pnas.2016239118 · DOI · Paper page
- Sanh, V. , Debut, L. , Chaumond, J. , & Wolf, T. (2019). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv Prepr arXiv191001108.
- Sun, Z. , Yu, H. , Song, X. , Liu, R. , Yang, Y. , & Zhou, D. (2020). MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices. arXiv Prepr arXiv200402984. · DOI · Paper page
- 10.1093/bioinformatics/btm098 · DOI · Paper page
- Vaswani, A. , Shazeer, N. , Parmar, N. , Uszkoreit, J. , Jones, L. , Gomez, A. N. , Kaiser, L. , & Polosukhin, I. (2017). Attention Is All You Need. arXiv Prepr arXiv170603762.
- Vig, J. , Madani, A. , Varshney, L. R. , Xiong, C. , Socher, R. , & Rajani, N. F. (2020). BERTology Meets Biology: Interpreting Attention in Protein Language Models. arXiv Prepr arXiv200615222. · DOI · Paper page
- 2021, PSSM-Distil: Protein Secondary Structure Prediction (PSSP) on Low-Quality PSSM by Knowledge Distillation with Contrastive Learning, Proceedings of the AAAI Conference on Artificial Intelligence, 35, 617, 10.1609/aaai.v35i1.16141 · DOI
- Wolf, T. , Debut, L. , Sanh, V. , Chaumond, J. , Delangue, C. , Moi, A. , Cistac, P. , Rault, T. , Louf, R. , Funtowicz, M. , Davison, J. , Shleifer, S. , von Platen, P. , Ma, C. , Jernite, Y. , Plu, J. , Xu, C. , Scao, T. le , Gugger, S. ,… Rush, A. M. (2019). HuggingFace’s Transformers: State-of-the-art Natural Language Processing. arXiv Prepr arXiv191003771. · DOI · Paper page
- 10.1073/pnas.1914677117 · DOI · Paper page
- Yang, Y. , Gao, J. , Wang, J. , Heffernan, R. , Hanson, J. , Paliwal, K. , & Zhou, Y. (2016). Sixty-five years of the long march in protein secondary structure pre-diction: the final stretch? Briefings in Bioinformatics, bbw129. https://doi.org/10.1093/bib/bbw129 · DOI
Metadata
- Authors
- Geffen, Y., Ofran, Y., & Unger, R.
- Affiliations
- Bar-Ilan University (IL)
- Corresponding author
- Unger, Ron
- ORCID
- Unger, Ron
- Venue
- Bioinformatics
- Publisher
- openRxiv
- Published
- 2022-05-10
- Volume
- 38
- Issue
- Supplement_2
- Pages
- ii95-ii98
- ISSN
- 1367-4803 (ISSN-L)
- OpenAlex
- W4296780589
- Cited by
- 40
- References
- 1
- Citations by year
- 2026: 8 · 2025: 12 · 2024: 9 · 2023: 11
- FWCI
- 2.61
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Bioinformatics (publishedVersion)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
- PubMed (publishedVersion)
- Funders
- Bar-Ilan University
- Data Science Institute, Columbia University
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Fractal and DNA sequence analysis
- Keywords
- ProtBERT, knowledge distillation, protein language models, protein sequence analysis, model compression, human proteome
- MeSH terms
- Amino Acid Sequence, Amino Acids, Humans, Natural Language Processing, Sequence Analysis, Protein, Proteome
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
