PDBench: evaluating computational methods for protein-sequence design
Open accessThe DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Abstract
Summary: Ever increasing amounts of protein structure data, combined with advances in machine learning, have led to the rapid proliferation of methods available for protein-sequence design. In order to utilize a design method effectively, it is important to understand the nuances of its performance and how it varies by design target. Here, we present PDBench, a set of proteins and a number of standard tests for assessing the performance of sequence-design methods. PDBench aims to maximize the structural diversity of the benchmark, compared with previous benchmarking sets, in order to provide useful biological insight into the behaviour of sequence-design methods, which is essential for evaluating their performance and practical utility. We believe that these tools are useful for guiding the development of novel sequence design algorithms and will enable users to choose a method that best suits their design target. Availability and implementation: https://github.com/wells-wood-research/PDBench Supplementary information: Supplementary data are available at Bioinformatics online.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
Accurate prediction of protein structures and interactions using a 3-track network
Baek, M., DiMaio, F., Anishchenko, I., Dauparas, J., Ovchinnikov, S., Lee, G. R., Wang, J., Cong, Q., Kinch, L. N., Schaeffer, R. D., et al. (2021) · doi:10.1101/2021.06.14.448402 · Cited by 5,549
Structure validation by Cα geometry: ϕ,ψ and Cβ deviation
Lovell, S. C., Davis, I. W., Arendall, W. B., de Bakker, P. I. W., Word, J. M., Prisant, M. G., Richardson, J. S., & Richardson, D. C. (2003) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.10286 · Cited by 4,595 · cited 2×
“The over prediction of glycine by our initial model is likely related to the flexibility of this residue, which allows it to adopt a broad combinations of Φ and Ψ (Lovell et al. 2003).”
BackgroundRives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280
Highly accurate protein structure prediction for the human proteome
Tunyasuvunakool, K., Adler, J., Wu, Z., Green, T., Zielinski, M., Žídek, A., Bridgland, A., Cowie, A., Meyer, C., Laydon, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03828-1 · Cited by 3,025
The Rosetta All-Atom Energy Function for Macromolecular Modeling and Design
Alford, R. F., Leaver-Fay, A., Jeliazkov, J. R., O’Meara, M. J., DiMaio, F. P., Park, H., Shapovalov, M. V., Renfrew, P. D., Mulligan, V. K., Kappel, K., et al. (2017) · Journal of Chemical Theory and Computation · doi:10.1021/acs.jctc.7b00125 · Cited by 1,532
The coming of age of de novo protein design
Huang, P.-S., Boyken, S. E., & Baker, D. (2016) · Nature · doi:10.1038/nature19946 · Cited by 1,478 · cited 1×
“It is interesting that all of the deep learning based methods performed well when designing “mainly β ” structures, as these are traditionally very challenging targets (Woolfson et al. 2015; Huang, Boyken, and Baker 2016).”
MethodTHE KINETICS OF FORMATION OF NATIVE RIBONUCLEASE DURING OXIDATION OF THE REDUCED POLYPEPTIDE CHAIN
Anfinsen, C. B., Haber, E., Sela, M., & White, F. H. (1961) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.47.9.1309 · Cited by 1,182
Shapovalov, M. V., & Dunbrack, R. L. (2011) · Structure · doi:10.1016/j.str.2011.03.019 · Cited by 1,000 · cited 1×
“It is well documented that the identity of the side chain has a large impact on the backbone conformation (Shapovalov and Dunbrack 2011), therefore it is likely that it is important to the quality of predictions.”
BackgroundCritical assessment of methods of protein structure prediction (CASP)—Round XIII
Kryshtafovych, A., Schwede, T., Topf, M., Fidelis, K., & Moult, J. (2019) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.25823 · Cited by 506
Single-sequence protein structure prediction using language models from deep learning
Chowdhury, R., Bouatta, N., Biswas, S., Rochereau, C., Church, G. M., Sorger, P. K., & AlQuraishi, M. (2021) · doi:10.1101/2021.08.02.454840 · Cited by 455
Single-sequence protein structure prediction using a language model and deep learning
Chowdhury, R., Bouatta, N., Biswas, S., Floristean, C., Kharkar, A., Roy, K., Rochereau, C., Ahdritz, G., Zhang, J., Church, G. M., et al. (2022) · Nature Biotechnology · doi:10.1038/s41587-022-01432-w · Cited by 455
Protein length in eukaryotic and prokaryotic proteomes
Brocchieri, L. (2005) · Nucleic Acids Research · doi:10.1093/nar/gki615 · Cited by 414 · cited 1×
“Polypeptides vary in length, from small peptides composed of only a few amino acids to huge proteins containing tens of thousands of amino acids, with a median length of around 300 (Brocchieri 2005).”
BackgroundWhat has de novo protein design taught us about protein folding and biophysics?
Baker, D. (2019) · Protein Science · doi:10.1002/pro.3588 · Cited by 161
Research works printed but not linked: 19
- Das, 2008, Macromolecular modeling with rosetta, Annu. Rev. Biochem, 77, 363, 10.1146/annurev.biochem.77.062906.171838 · DOI · Paper page
- Goldenzweig, 2016, Automated structure- and sequence-based design of proteins for high bacterial expression and stability, Mol. Cell, 63, 337, 10.1016/j.molcel.2016.06.012 · DOI · Paper page
- Henikoff, 1992, Amino acid substitution matrices from protein blocks, Proc. Natl. Acad. Sci. USA, 89, 10915, 10.1073/pnas.89.22.10915 · DOI · Paper page
- Huang, 2019, Convolutional networks with dense connectivity, IEEE Trans. Pattern Anal. Mach. Intell, 8704
- Huang, 2019, EvoEF2: Accurate and fast energy function for computational protein design, Bioinformatics, 36, 1135, 10.1093/bioinformatics/btz740 · DOI · Paper page
- Joosten, 2011, A series of PDB related databases for everyday needs, Nucleic Acids Res, 39, D411, 10.1093/nar/gkq1105 · DOI · Paper page
- Knudsen, 2010, The CATH database, Hum. Genomics, 4, 207, 10.1186/1479-7364-4-3-207 · DOI · Paper page
- Ludwiczak, 2018, Combining Rosetta with molecular dynamics (MD): A benchmark of the MD-based ensemble protein design, J. Struct. Biol, 203, 54, 10.1016/j.jsb.2018.02.004 · DOI
- O'Connell, 2018, Spin2: Predicting sequence profiles from protein structures using deep neural networks, Proteins, 86, 629, 10.1002/prot.25489 · DOI · Paper page
- Ollikainen, 2013, Computational protein design quantifies structural constraints on amino acid covariation, PLoS Comput. Biol, 9, e1003313, 10.1371/journal.pcbi.1003313 · DOI
- Qi, 2020, DenseCPD: Improving the accuracy of neural-network-based computational protein sequence design with DenseNet, J. Chem. Inf. Model, 60, 1245, 10.1021/acs.jcim.0c00043 · DOI · Paper page
- Stam, 2021, DE-STRESS: A user-friendly web application for the evaluation of protein designs, Protein Eng. Des. Sel, 34, 10.1093/protein/gzab029 · DOI
- Strokach, 2020, Fast and flexible protein design using deep graph neural networks, Cell Syst, 11, 402, 10.1016/j.cels.2020.08.016 · DOI · Paper page
- Wood, 2017, ISAMBARD: An open-source computational environment for biomolecular analysis, modelling and design, Bioinformatics, 33, 3043, 10.1093/bioinformatics/btx352 · DOI · Paper page
- Woolfson, 2021, A brief history of de novo protein design: Minimal, rational, and computational, J. Mol. Biol, 433, 167160, 10.1016/j.jmb.2021.167160 · DOI · Paper page
- Woolfson, 2015, De novo protein design: How do we expand into the universe of possible protein structures?, Curr. Opin. Struct. Biol, 33, 16, 10.1016/j.sbi.2015.05.009 · DOI · Paper page
- Wu, 2022, High-resolution de novo structure prediction from primary sequence, bioRxiv
- Yue, 1992, Inverse protein folding problem: Designing polymer sequences, Proc. Natl. Acad. Sci. USA, 89, 4163, 10.1073/pnas.89.9.4163 · DOI · Paper page
- Zhang, 2020, ProDCoNN: Protein design using a convolutional neural network, Proteins, 88, 819, 10.1002/prot.25868 · DOI · Paper page
Cited by
From sequence to function through structure: Deep learning for protein design
(2022) · doi:10.1016/j.csbj.2022.11.014 · Cited by 124
Assessing the generalization capabilities of TCR binding predictors via peptide distance analysis
Castorina, L. V., Grazioli, F., Machart, P., Mösch, A., & Errica, F. (2025) · PLOS One · doi:10.1371/journal.pone.0324011 · Cited by 7
Self-citationAssessing the Generalization Capabilities of TCR Binding Predictors via Peptide Distance Analysis
Castorina, L. V., Grazioli, F., Machart, P., Mösch, A., & Errica, F. (2023) · doi:10.1101/2023.07.29.551100 · Cited by 7
Self-citation(2024) · doi:10.1093/bib/bbae146 · Cited by 6
Zheng, Z., Zhang, B., Zhong, B., Yu, J., Liu, K., Li, Z., Zhu, J., Wei, T., & Chen, H.-F. (2026) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1014290 · Cited by 1
Metadata
- Authors
- Castorina, L. V., Petrenas, R., Subr, K., & Wood, C. W.
- Venue
- Bioinformatics
- Publisher
- Oxford University Press (OUP)
- Published
- 2023-01-01
- Volume
- 39
- Issue
- 1
- ISSN
- 1367-4803 (ISSN-L)
- 1367-4811
- OpenAlex
- W3199169486
- Cited by
- 21
- References
- 15
- FWCI
- 1.34
- Type
- journal article
- Language
- en
- Open access
- gold
- Funders
- Engineering and Physical Sciences Research Council (EP/S003002/1)
- UK Research and Innovation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.