A fully automated benchmarking suite to compare macromolecular complexes
Open accessPublished version of a preprint
Abstract
Protein structure prediction has a long history of benchmarking efforts such as critical assessment of structure prediction, continuous automated model evaluation and critical assessment of prediction of interactions. With the rise of artificial intelligence-based methods for prediction of macromolecular complexes, benchmarking with large datasets and robust, unsupervised scores to compare predictions against a reference has become essential. Also, the increasing size and complexity of experimentally determined reference structures by crystallography or cryogenic electron microscopy poses challenges for structure comparison methods. Here we review the current state of the art in scoring methodologies, identify existing limitations and present more suitable approaches for scoring of tertiary and quaternary structures, protein–protein interfaces and protein–ligand complexes. Our methods are designed to scale efficiently, enabling the assessment of large, complex systems. All developments are available in the structure benchmarking framework of OpenStructure. OpenStructure is open source software and available for free at https://openstructure.org/ .
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764 · cited 1× · see citing works
“However, despite the impressive recent progress in prediction accuracy, even the latest developments in the field of structure prediction such as AlphaFold 19,20 and RoseTTAFold 21 have their limitations, highlighting the importance of continuing benchmarking efforts in the future.”
Accurate structure prediction of biomolecular interactions with AlphaFold 3
Abramson, J., Adler, J., Dunger, J., Evans, R., Green, T., Pritzel, A., Ronneberger, O., Willmore, L., Ballard, A. J., Bambrick, J., et al. (2024) · Nature · doi:10.1038/s41586-024-07487-w · Cited by 15,861 · cited 5× · see citing works
“However, despite the impressive recent progress in prediction accuracy, even the latest developments in the field of structure prediction such as AlphaFold 19,20 and RoseTTAFold 21 have their limitations, highlighting the importance of continuing benchmarking efforts in the future.”
InfluentialNeedleman, S. B., & Wunsch, C. D. (1970) · Journal of Molecular Biology · doi:10.1016/0022-2836(70)90057-4 · Cited by 11,288 · cited 1× · see citing works
“Pairwise sequence alignments are constructed via the Needleman–Wunsch dynamic programming algorithm 63 , which employs the BLOSUM62 substitution matrix 64 for proteins and NUC44 for nucleotides (ftp:// ftp.ncbi.nih.gov/blast/matrices/NUC.4.4).”
InfluentialExploring Network Structure, Dynamics, and Function Using NetworkX
Hagberg, A. A., Schult, D. A., & Swart, P. J. (2008) · Proceedings of the Python in Science Conference · doi:10.25080/tcwv9851 · Cited by 7,477 · cited 1× · see citing works
“All graph operations are performed using the Python NetworkX software 71 .”
Amino acid substitution matrices from protein blocks.
Henikoff, S., & Henikoff, J. G. (1992) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.89.22.10915 · Cited by 6,218 · cited 1× · see citing works
“Pairwise sequence alignments are constructed via the Needleman–Wunsch dynamic programming algorithm 63 , which employs the BLOSUM62 substitution matrix 64 for proteins and NUC44 for nucleotides (ftp:// ftp.ncbi.nih.gov/blast/matrices/NUC.4.4).”
InfluentialAccurate prediction of protein structures and interactions using a 3-track network
Baek, M., DiMaio, F., Anishchenko, I., Dauparas, J., Ovchinnikov, S., Lee, G. R., Wang, J., Cong, Q., Kinch, L. N., Schaeffer, R. D., et al. (2021) · doi:10.1101/2021.06.14.448402 · Cited by 5,542 · cited 1× · see citing works
“However, despite the impressive recent progress in prediction accuracy, even the latest developments in the field of structure prediction such as AlphaFold 19,20 and RoseTTAFold 21 have their limitations, highlighting the importance of continuing benchmarking efforts in the future.”
Protein complex prediction with AlphaFold-Multimer
Evans, R., O’Neill, M., Pritzel, A., Antropova, N., Senior, A., Green, T., Žídek, A., Bates, R., Blackwell, S., Yim, J., et al. (2021) · doi:10.1101/2021.10.04.463034 · Cited by 3,808 · cited 5× · see citing works
“The algorithm described for AlphaFold-Multimer 26 selects an anchor chain in the reference and initializes mappings by superposing all matching-sequence model chains followed by iterative chain pairing by minimal centroid distance, searching for the mapping with lowest centroid root mean squared…”
InfluentialA solution for the best rotation to relate two sets of vectors
Kabsch, W. (1976) · Acta Crystallographica Section A · doi:10.1107/s0567739476001873 · Cited by 3,462 · cited 3× · see citing works
“First, scores reliant on rigid superposition of representative backbone atoms (typically Cα for proteins), such as the r.m.s.d. 32 , global distance test (GDT) 33 or TM-score 24 .”
Accurate Bond and Angle Parameters for X-ray Protein Structure Refinement
Engh, R., & Huber, R. (1991) · Acta Crystallographica Section A Foundations of Crystallography · doi:10.1107/s0108767391001071 · Cited by 2,497 · cited 1× · see citing works
“66.”
Scoring function for automated assessment of protein structure template quality
Zhang, Y., & Skolnick, J. (2004) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.20264 · Cited by 2,392 · cited 12× · see citing works
“Contact-based scores, along with GDT and TM-scores, naturally penalize incomplete models by design, but r.m.s.d.-based measures do not as no distance between model and reference can be computed for missing residues.”
InfluentialREFMAC 5 dictionary: organization of prior chemical knowledge and guidelines for its use
Vagin, A. A., Steiner, R. A., Lebedev, A. A., Potterton, L., McNicholas, S. J., Long, F., & Murshudov, G. N. (2004) · Acta Crystallographica Section D Biological Crystallography · doi:10.1107/s0907444904023510 · Cited by 1,351 · cited 1× · see citing works
“Amino acid and nucleotide parameters are now extracted from the CCP4 MON_LIB 65 instead of the previously used amino acid parameters from ref.”
Mol* Viewer: modern web app for 3D visualization and analysis of large biomolecular structures
Sehnal, D., Bittrich, S., Deshpande, M., Svobodová, R., Berka, K., Bazgier, V., Velankar, S., Burley, S. K., Koča, J., & Rose, A. S. (2021) · Nucleic Acids Research · doi:10.1093/nar/gkab314 · Cited by 1,154 · cited 1× · see citing works
“Structural models in the manuscript figures were displayed in Mol* 62 .”
Mariani, V., Biasini, M., Barbato, A., & Schwede, T. (2013) · Bioinformatics · doi:10.1093/bioinformatics/btt473 · Cited by 1,138 · cited 3× · see citing works
“36 and summarized in Supplementary Section 2 but involved technical adjustments to natively handle multi-chain complexes with model/reference chain mappings from QSMap/QSMapR.”
Self-citationLGA: a method for finding 3D similarities in protein structures
Zemla, A. (2003) · Nucleic Acids Research · doi:10.1093/nar/gkg571 · Cited by 1,069 · cited 11× · see citing works
“Contact-based scores, along with GDT and TM-scores, naturally penalize incomplete models by design, but r.m.s.d.-based measures do not as no distance between model and reference can be computed for missing residues.”
InfluentialBertoni, M., Kiefer, F., Biasini, M., Bordoli, L., & Schwede, T. (2017) · Scientific Reports · doi:10.1038/s41598-017-09654-8 · Cited by 791 · cited 16× · see citing works
“QS-score (referred to as QS-global in OpenStructure) is designed to compare complexes and differentiate between quaternary states.”
InfluentialSelf-citationCAPRI: A Critical Assessment of PRedicted Interactions
Janin, J., Henrick, K., Moult, J., Eyck, L. T., Sternberg, M. J. E., Vajda, S., Vakser, I., & Wodak, S. J. (2003) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.10381 · Cited by 703 · cited 2× · see citing works
“The critical assessment of prediction of interactions (CAPRI) experiment 4,5 assesses the accuracy of the interface prediction in protein–protein complexes, with new rounds announced approximately every 6 months.”
Methods of protein structure comparison
Kufareva, I., & Abagyan, R. (2012) · Methods in molecular biology · doi:10.1007/978-1-61779-588-6_10 · Cited by 625 · cited 1× · see citing works
“https://doi.org/10.1038/s41592-025-02973-z differences in the local environment 34 .”
InfluentialDalby, A., Nourse, J. G., Hounshell, W. D., Gushurst, A. K. I., Grier, D., Leland, B., & Laufer, J. (1992) · Journal of chemical information and computer sciences · doi:10.1021/ci00007a012 · Cited by 600 · cited 2× · see citing works
“However, the recommended approach is to load ligand structures from SDF files 61 , which inherently provide detailed connectivity information.”
A large‐scale experiment to assess protein structure prediction methods
Moult, J., Pedersen, J. T., Judson, R., & Fidelis, K. (1995) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.340230303 · Cited by 555 · cited 2× · see citing works
“Historically, CASP uses GDT_TS (total score), which averages GDT scores with distance thresholds 1, 2, 4 and 8 Å .”
Deng, Z., Chuaqui, C., & Singh, J. (2004) · Journal of Medicinal Chemistry · doi:10.1021/jm030331x · Cited by 544 · see citing works
US-align: universal structure alignments of proteins, nucleic acids, and macromolecular complexes
Zhang, C., Shine, M., Pyle, A. M., & Zhang, Y. (2022) · Nature Methods · doi:10.1038/s41592-022-01585-1 · Cited by 511 · cited 8× · see citing works
“US-align 25 also maximizes the TM-score, but avoids exhaustive enumeration by deriving an initial mapping with a greedy search algorithm, which is then iteratively optimized.”
InfluentialZRANK: Reranking protein docking predictions with an optimized energy function
Pierce, B., & Weng, Z. (2007) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.21373 · Cited by 468 · cited 1× · see citing works
“This usage is distinct from scores that may reflect energy-based evaluations, such as those generated by tools such as ZRank 31 , which are used during modeling or docking but are not direct measures of structural similarity.”
DockQ: A Quality Measure for Protein-Protein Docking Models
Basu, S., & Wallner, B. (2016) · PLOS ONE · doi:10.1371/journal.pone.0161879 · Cited by 460 · cited 2× · see citing works
“The DockQ score 37 was introduced in CAPRI recently as an effort to combine these three scores into one continuous number, avoiding a classification approach and thus making it more suitable as a target score for automated modeling methods 38 .”
Assessment of blind predictions of protein–protein interactions: Current status of docking methods
Mendez, R., Leplae, R., Maria, L. D., & Wodak, S. (2003) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.10393 · Cited by 438 · cited 1× · see citing works
“37,73.”
Optimizing Fragment and Scaffold Docking by Use of Molecular Interaction Fingerprints
Marcou, G., & Rognan, D. (2007) · Journal of Chemical Information and Modeling · doi:10.1021/ci600342e · Cited by 404 · cited 1× · see citing works
“…to assess how well participants could model receptor–ligand complexes—a symmetry-corrected r.m.s.d. to measure the absolute accuracy of the predicted ligand within the binding site—and contact-based scores to evaluate the reproduction of native receptor–ligand noncovalent interactions 13,42,43 .”
Community-wide assessment of GPCR structure modelling and ligand docking: GPCR Dock 2008
participants, G. D. 2., Michino, M., Abola, E. E., Brooks, C. L., Dixon, J. S., Moult, J., & Stevens, R. C. (2009) · Nature Reviews Drug Discovery · doi:10.1038/nrd2877 · Cited by 276 · cited 2× · see citing works
“A local superposition was needed as the reference structures in CASP were larger and more flexible than those in GPCR dock 15,16 and the first stage of the D3R Grand Challenge 3 9 .”
Validation of Structures in the Protein Data Bank
Gore, S., Sanz-García, E., Hendrickx, P., Gutmanas, A., Westbrook, J., Yang, H.-W., Feng, Z.-K., Baskaran, K., Berrisford, J., Hudson, B., et al. (2017) · Structure · doi:10.1016/j.str.2017.10.009 · Cited by 245 · cited 1× · see citing works
“Considerations such as structure quality validation 53 and detailed stereochemical analysis are not included in this work.”
D3R grand challenge 2015: evaluation of protein-ligand pose and affinity predictions.
Gathiaka, S., Liu, S., Chiu, M., Yang, H., Stuckey, J. A., Kang, Y. N., Delproposto, J., Kubish, G., Dunbar, J. B., Carlson, H. A., et al. (2016) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-016-9946-8 · Cited by 207 · cited 4× · see citing works
“A local superposition was needed as the reference structures in CASP were larger and more flexible than those in GPCR dock 15,16 and the first stage of the D3R Grand Challenge 3 9 .”
InfluentialGaieb, Z., Liu, S., Gathiaka, S., Chiu, M., Yang, H., Shao, C., Feher, V. A., Walters, W. P., Kuhn, B., Rudolph, M. G., et al. (2018) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-017-0088-4 · Cited by 193 · cited 1× · see citing works
“Historically, CASP uses GDT_TS (total score), which averages GDT scores with distance thresholds 1, 2, 4 and 8 Å .”
Assessment of template based protein structure predictions in CASP9
Mariani, V., Kiefer, F., Schmidt, T., Haas, J., & Schwede, T. (2011) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.23177 · Cited by 185 · cited 1× · see citing works
“It was introduced in the CASP9 experiment 47 and has been used as the primary comparison score for tertiary structures in CAMEO as it allows for fully automated assessment owing to its robustness against domain movement events.”
Self-citationMukherjee, S., & Zhang, Y. (2009) · Nucleic Acids Research · doi:10.1093/nar/gkp318 · Cited by 178 · cited 1× · see citing works
“Notably, the MM-align tool 23 aims to maximize the template modeling score (TM-score) 24 between model and reference by enumerating the full solution space but omits the costly score computation for unlikely mappings.”
OpenStructure: an integrated software framework for computational structural biology
Biasini, M., Schmidt, T., Bienert, S., Mariani, V., Studer, G., Haas, J., Johner, N., Schenk, A., Philippsen, A., & Schwede, T. (2013) · Acta Crystallographica Section D: Biological Crystallography · doi:10.1107/s0907444913007051 · Cited by 178 · cited 2× · see citing works
“The OpenStructure computational structural biology framework 49 has streamlined the implementation of comparison scores addressing various aspects discussed in this manuscript, including proteins, protein complexes and protein–ligand complexes.”
Self-citationWestbrook, J., Shao, C., Feng, Z.-K., Zhuravleva, M., Velankar, S., & Young, J. Y. (2015) · Bioinform. · doi:10.1093/bioinformatics/btu789 · Cited by 177 · cited 6× · see citing works
“Shortly, internal residue connectivity is established on the basis of residue and atom names using the PDB chemical component dictionary (CCD) 60 .”
InfluentialHaas, J., Barbato, A., Behringer, D., Studer, G., Roth, S., Bertoni, M., Mostaguir, K., Gumienny, R., & Schwede, T. (2018) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.25431 · Cited by 168 · cited 1× · see citing works
“To directly compare reference and model binding sites, RMSD-LP and LDDT-LP have been introduced, where LP stands for ligand pocket, and are similar to the previously described LDDT-BS score 72 .”
Self-citationAdvances in GPCR modeling evaluated by the GPCR Dock 2013 assessment: meeting new challenges
Kufareva, I., Katritch, V., Stevens, R., & Abagyan, R. (2014) · Structure · doi:10.1016/j.str.2014.06.012 · Cited by 167 · cited 3× · see citing works
“A local superposition was needed as the reference structures in CASP were larger and more flexible than those in GPCR dock 15,16 and the first stage of the D3R Grand Challenge 3 9 .”
The Protein Data Bank archive as an open data resource
Berman, H. M., Kleywegt, G. J., Nakamura, H., & Markley, J. L. (2014) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-014-9770-y · Cited by 155 · cited 1× · see citing works
“OpenStructure accepts structural information in legacy PDB format 58 or in the preferred PDBx/mmCIF format 59 .”
CAD‐score: A new contact area difference‐based function for evaluation of protein structural models
Olechnovič, K., Kulberkytė, E., & Venclovas, Č. (2013) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.24172 · Cited by 153 · cited 1× · see citing works
“Examples include the contact area difference (CAD) score 35 , and the local distance difference test (LDDT) score 36 .”
Kryshtafovych, A., Schwede, T., Topf, M., Fidelis, K., & Moult, J. (2023) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.26617 · Cited by 148 · see citing works
Self-citationBioLiP2: an updated structure database for biologically relevant ligand–protein interactions
Zhang, C., Zhang, X., Freddolino, L., & Zhang, Y. (2023) · Nucleic Acids Research · doi:10.1093/nar/gkad630 · Cited by 142 · cited 1× · see citing works
“Other resources such as BioLiP 68 , FireDB 69 or the defunct IBIS database 70 have attempted to tackle the issue.”
The structure assessment web server: for proteins, complexes and more
Waterhouse, A. M., Studer, G., Robin, X., Bienert, S., Tauriello, G., & Schwede, T. F. (2024) · Nucleic Acids Research · doi:10.1093/nar/gkae270 · Cited by 140 · cited 1× · see citing works
“A selection of scores for polymer entities is also available in the SWISS-MODEL Structure Assessment server 50 available at https://swissmodel.expasy.org/assess.”
Self-citationAssessment of three‐dimensional RNA structure prediction in CASP15
Das, R., Kretsch, R. C., Simpkin, A. J., Mulvaney, T., Pham, P., Rangan, R., Bu, F., Keegan, R. M., Topf, M., Rigden, D. J., et al. (2023) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.26602 · Cited by 139 · cited 2× · see citing works
“https://doi.org/10.1038/s41592-025-02973-z the context of the CASP15 RNA modeling challenge 48 .”
InfluentialD3R Grand Challenge 3: blind prediction of protein–ligand poses and affinity rankings
Gaieb, Z., Parks, C. D., Chiu, M. W., Yang, H., Shao, C., Walters, W. P., Lambert, M. H., Nevins, N., Bembenek, S. D., Ameriks, M. K., et al. (2019) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-018-0180-4 · Cited by 137 · cited 3× · see citing works
“A local superposition was needed as the reference structures in CASP were larger and more flexible than those in GPCR dock 15,16 and the first stage of the D3R Grand Challenge 3 9 .”
Parks, C. D., Gaieb, Z., Chiu, M. W., Yang, H., Shao, C., Walters, W. P., Jansen, J. M., McGaughey, G. B., Lewis, R. A., Bembenek, S. D., et al. (2020) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-020-00289-y · Cited by 137 · cited 1× · see citing works
“A variety of ligand pose prediction challenges have taken place, such as D3R 7–10 , CELPP 11 and others 12–16 , and ligand assessment has been included in CASP since 2022 17 .”
Modeling protein‐protein, protein‐peptide, and protein‐oligosaccharide complexes: CAPRI 7th edition
Lensink, M. F., Nadzirin, N., Velankar, S. S., & Wodak, S. J. (2019) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.25870 · Cited by 136 · cited 1× · see citing works
“For protein–peptide interactions, the CAPRI community recommended modifying the default parameters 75 .”
Damm-Ganamet, K. L., Smith, R. D., Dunbar, J. B., Stuckey, J. A., & Carlson, H. A. (2013) · Journal of Chemical Information and Modeling · doi:10.1021/ci400025f · Cited by 133 · cited 2× · see citing works
“…to assess how well participants could model receptor–ligand complexes—a symmetry-corrected r.m.s.d. to measure the absolute accuracy of the predicted ligand within the binding site—and contact-based scores to evaluate the reproduction of native receptor–ligand noncovalent interactions 13,42,43 .”
spyrmsd: symmetry-corrected RMSD calculations in Python
Meli, R., & Biggin, P. (2020) · Journal of Cheminformatics · doi:10.1186/s13321-020-00455-2 · Cited by 112 · cited 3× · see citing works
“Furthermore, enumerating graph isomorphisms produces a list of one-to-one atom mappings between reference and model ligands, allowing us to account for potential symmetries that are chemically equivalent, following established methodologies 30 .”
Mirabello, C., & Wallner, B. (2024) · bioRxiv · doi:10.1093/bioinformatics/btae586 · Cited by 110 · cited 1× · see citing works
“1.3 74 , available from https://github.com/ bjornwallner/DockQ, with additional details provided in Supplementary Section 4.”
InfluentialPrediction of protein assemblies, the next frontier: The CASP14‐CAPRI experiment
Lensink, M. F., Brysbaert, G., Mauri, T., Nadzirin, N., Velankar, S. S., Chaleil, R. A. G., Clarence, T., Bates, P. A., Kong, R., Liu, B., et al. (2021) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.26222 · Cited by 105 · see citing works
Shoemaker, B. A., Zhang, D.-C., Tyagi, M., Thangudu, R. R., Fong, J. H., Marchler-Bauer, A., Bryant, S. H., Madej, T., & Panchenko, A. (2011) · Nucleic Acids Res. · doi:10.1093/nar/gkr997 · Cited by 89 · cited 1× · see citing works
“Other resources such as BioLiP 68 , FireDB 69 or the defunct IBIS database 70 have attempted to tackle the issue.”
Bittrich, S., Bhikadiya, C., Bi, C., Chao, H., Duarte, J. M., Dutta, S., Fayazi, M., Henry, J., Khokhriakov, I., Lowe, R., et al. (2023) · Journal of Molecular Biology · doi:10.1016/j.jmb.2023.167994 · Cited by 83 · cited 2× · see citing works
“Automated access to the full functionality available for the command line ‘actions’ is available via a REST API at https://swissmodel.expasy.org/ost.”
PLINDER: The protein-ligand interactions dataset and evaluation resourcePreprint
Durairaj, J., Adeshina, Y. O., Cao, Z., Zhang, X., Oleinikovas, V., Duignan, T. J., McClure, Z. D., Robin, X., Studer, G., Kovtun, D., et al. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.07.17.603955 · Cited by 82 · cited 2× · see citing works
“Automating structural quality validation is challenging, and we are working on incorporating corresponding checks into the benchmark dataset creation process 22,54 and separately into the scores themselves.”
Self-citationVoronota: A fast and reliable tool for computing the vertices of the Voronoi diagram of atomic balls
Olechnovič, K., & Venclovas, Č. (2014) · Journal of Computational Chemistry · doi:10.1002/jcc.23538 · Cited by 80 · see citing works
Impact of AlphaFold on structure prediction of protein complexes: The CASP15‐CAPRI experiment
Lensink, M. F., Brysbaert, G., RAOURAOUA, N., Bates, P. A., Giulini, M., Honorato, R. V., van Noort, C. W., Teixeira, J. M. C., Bonvin, A. M. J. J., Kong, R., et al. (2023) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.26609 · Cited by 79 · cited 4× · see citing works
“Historically, CASP uses GDT_TS (total score), which averages GDT scores with distance thresholds 1, 2, 4 and 8 Å .”
InfluentialRapid and sensitive protein complex alignment with Foldseek-Multimer
Kim, W., Mirdita, M., Karin, E. L., Gilchrist, C. L. M., Schweke, H., Söding, J., Levy, E. D., & Steinegger, M. (2025) · Nature Methods · doi:10.1038/s41592-025-02593-7 · Cited by 75 · cited 3× · see citing works
“In addition, we compared QSMapR with US-align 25 , Foldseek-Multimer 27 and our own implementation of the chain-mapping algorithm used in AlphaFold-Multimer 26 , all of which rely on global superposition and are optimized for assessing overall topology.”
CAMEO - Perspectives on the future of fully automated evaluation of structure prediction methods
Robin, X., Haas, J., Gumienny, R., Smolinski, A., Tauriello, G., & Schwede, T. (2021) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.26213 · Cited by 75 · cited 10× · see citing works
“36 and summarized in Supplementary Section 2 but involved technical adjustments to natively handle multi-chain complexes with model/reference chain mappings from QSMap/QSMapR.”
InfluentialSelf-citationPDBx/mmCIF Ecosystem: Foundational Semantic Tools for Structural Biology
Westbrook, J., Young, J. Y., Shao, C., Feng, Z.-K., Guranovic, V., Lawson, C., Vallat, B. K., Adams, P., Berrisford, J., Bricogne, G., et al. (2022) · Journal of Molecular Biology · doi:10.1016/j.jmb.2022.167599 · Cited by 73 · cited 1× · see citing works
“OpenStructure accepts structural information in legacy PDB format 58 or in the preferred PDBx/mmCIF format 59 .”
Assessment of protein assembly prediction in CASP12
Lafita, A., Bliven, S., Kryshtafovych, A., Bertoni, M., Monastyrskyy, B., Duarte, J. M., Schwede, T., & Capitani, G. (2017) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.25408 · Cited by 72 · cited 2× · see citing works
“The CASP assessment of oligomers primarily relies on interface contact-based scores, akin to f nat , named the interface contact similarity (ICS) and interface patch similarity (IPS) scores 6 .”
Self-citationKinch, L. N., Shi, S.-Y., Cheng, H., Cong, Q., Pei, J.-M., Mariani, V., Schwede, T., & Grishin, N. V. (2011) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.23190 · Cited by 66 · cited 2× · see citing works
“Superposition-dependent scores fail to account for this flexibility and require manual segmentation of the reference structure 51 ; in Table 1, this affects the GDT, r.m.s.d. and TM-scores.”
Self-citationNew prediction categories in <scp>CASP15</scp>
Kryshtafovych, A., Antczak, M., Szachniuk, M., Zok, T., Kretsch, R. C., Rangan, R., Pham, P., Das, R., Robin, X., Studer, G., et al. (2023) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.26515 · Cited by 54 · cited 1× · see citing works
“A variety of ligand pose prediction challenges have taken place, such as D3R 7–10 , CELPP 11 and others 12–16 , and ligand assessment has been included in CASP since 2022 17 .”
Self-citationSmith, R. D., Damm-Ganamet, K., Dunbar, J., Ahmed, A., Chinnaswamy, K., Delproposto, J., Kubish, G., Tinberg, C., Khare, S., Dou, J., et al. (2015) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.5b00387 · Cited by 54 · see citing works
Ozden, B., Kryshtafovych, A., & Karaca, E. (2023) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.26598 · Cited by 51 · see citing works
Comparative analysis of methods for evaluation of protein models against native structures
Olechnovič, K., Monastyrskyy, B., Kryshtafovych, A., & Venclovas, Č. (2018) · Bioinform. · doi:10.1093/bioinformatics/bty760 · Cited by 50 · cited 1× · see citing works
“Together, these scores paint a comprehensive view of how accurately a model represents the target structure 45 .”
Assessment of protein–ligand complexes in CASP15
Robin, X., Studer, G., Durairaj, J., Eberhardt, J., Schwede, T., & Walters, W. P. (2023) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.26601 · Cited by 47 · cited 9× · see citing works
“CASP15 presented a more complex challenge.”
InfluentialSelf-citationContinuous Evaluation of Ligand Protein Predictions: A Weekly Community Challenge for Drug Docking
Wagner, J., Churas, C., Liu, S., Swift, R. V., Chiu, M., Shao, C., Feher, V., Burley, S., Gilson, M., & Amaro, R. (2018) · bioRxiv · doi:10.1101/469940 · Cited by 45 · cited 1×
“A variety of ligand pose prediction challenges have taken place, such as D3R 7–10 , CELPP 11 and others 12–16 , and ligand assessment has been included in CASP since 2022 17 .”
CAPRI-Q: The CAPRI resource evaluating the quality of predicted structures of protein complexes
Collins, K. W., Copeland, M. M., Brysbaert, G., Wodak, S. J., Bonvin, A. M. J. J., Kundrotas, P. J., Vakser, I. A., & Lensink, M. F. (2024) · Journal of Molecular Biology · doi:10.1016/j.jmb.2024.168540 · Cited by 42 · cited 1× · see citing works
“The CAPRI community considers issues concerning L-RMSD by falling back on i-RMSD for these cases in their model quality classification rules 52 .”
Assessment of the <scp>CASP14</scp> assembly predictions
Ozden, B., Kryshtafovych, A., & Karaca, E. (2021) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.26199 · Cited by 41 · see citing works
Assessment of protein assembly prediction in CASP13
Guzenko, D., Lafita, A., Monastyrskyy, B., Kryshtafovych, A., & Duarte, J. M. (2019) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.25795 · Cited by 39 · cited 13× · see citing works
“Contact-based scores, along with GDT and TM-scores, naturally penalize incomplete models by design, but r.m.s.d.-based measures do not as no distance between model and reference can be computed for missing residues.”
InfluentialAssessing interaction recovery of predicted protein-ligand poses
Errington, D., Schneider, C., Bouysset, C., & Dreyer, F. A. (2024) · Journal of Cheminformatics · doi:10.1186/s13321-025-01011-6 · Cited by 33 · cited 1× · see citing works
“Similarly, previous fingerprint-based scores assessing protein−ligand interactions 13,42–44 are restricted to a subjective set of interactions (such as hydrogen bonds, ionic, hydrophobic or π interactions, or metal coordination) and dependent on manual preparation steps, making them difficult to…”
Breaking the conformational ensemble barrier: Ensemble structure modeling challenges in CASP15
Kryshtafovych, A., Montelione, G., Rigden, D., Mesdaghi, S., Karaca, E., & Moult, J. (2023) · doi:10.22541/au.169199504.48580490/v1 · Cited by 31 · see citing works
Assessment of the assessment—All about complexes
Studer, G., Tauriello, G., & Schwede, T. F. (2023) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.26612 · Cited by 31 · see citing works
Self-citationBreaking the conformational ensemble barrier: Ensemble structure modeling challenges in CASP15
Kryshtafovych, A., Montelione, G. T., Rigden, D. J., Mesdaghi, S., Karaca, E., & Moult, J. (2023) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.26584 · Cited by 31 · cited 1× · see citing works
“Finally, considerations about flexibility and disorder, involving considerably different sets of methods 55,56 , are out of the scope of this manuscript.”
NEW TOOLS AND EXPANDED DATA ANALYSIS CAPABILITIES AT THE PROTEIN STRUCTURE PREDICTION CENTER
Kryshtafovych, A., Prlić, A., Dmytriv, Z., Daniluk, P., Milostan, M., Eyrich, V. A., Hubbard, T., & Fidelis, K. (2007) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.21653 · Cited by 30 · cited 1× · see citing works
“The GDT_HA (high accuracy) variant, introduced in the high accuracy category of CASP7, uses lower distance thresholds of 0.5, 1, 2 and 4 Å to provide a finer-grained estimate for high-quality models 76 .”
InfluentialRepresenting structures of the multiple conformational states of proteins
Ramelot, T. A., Tejero, R., & Montelione, G. T. (2023) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2023.102703 · Cited by 26 · cited 1× · see citing works
“Finally, considerations about flexibility and disorder, involving considerably different sets of methods 55,56 , are out of the scope of this manuscript.”
FireDB: a compendium of biological and pharmacologically relevant ligands
Maietta, P., López, G., Carro, A., Pingilley, B. J., León, L., Valencia, A., & Tress, M. (2013) · Nucleic Acids Res. · doi:10.1093/nar/gkt1127 · Cited by 26 · cited 1× · see citing works
“Other resources such as BioLiP 68 , FireDB 69 or the defunct IBIS database 70 have attempted to tackle the issue.”
Automated benchmarking of combined protein structure and ligand conformation prediction
Leemann, M., Sagasta, A., Eberhardt, J., Schwede, T. F., Robin, X., & Durairaj, J. (2023) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.26605 · Cited by 23 · cited 1× · see citing works
“Automating structural quality validation is challenging, and we are working on incorporating corresponding checks into the benchmark dataset creation process 22,54 and separately into the scores themselves.”
Self-citationSimplified Quality Assessment for Small-molecule Ligands in the Protein Data Bank
Shao, C., Westbrook, J., Lu, C., Bhikadiya, C., Peisach, E., Young, J. Y., Duarte, J. M., Lowe, R., Wang, S., Rose, Y., et al. (2022) · Structure · doi:10.1016/j.str.2021.10.003 · Cited by 22 · cited 1× · see citing works
“We follow the definition of ligands from the PDB, including small molecules such as ions, solvent molecules, drugs, enzymes, co-factors, etc. associated with biological polymers 67 .”
Teach-Discover-Treat (TDT): Collaborative Computational Drug Discovery for Neglected Diseases
Jansen, J., Cornell, W. D., Tseng, Y., & Amaro, R. (2012) · Journal of Molecular Graphics and Modelling · doi:10.1016/j.jmgm.2012.07.007 · Cited by 16 · cited 2× · see citing works
“Other than LGA, which employs the longest continuous segment algorithm 33 to help identify good starting sets, OpenStructure simply uses sliding windows of sizes of 7, 9, 12, 24 and 48 on the model/reference positions.”
Karmakar, T. (2025) · Resonance · doi:10.1007/s12045-025-1804-3 · Cited by 14 · cited 1× · see citing works
“Over the past decades, objective, blind, independent benchmarking efforts have been a major driver for the development of highly accurate structure prediction methods 18 .”
Flagged first, then by citations
2 of the 80 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Evaluating generalization in protein–ligand cofolding methods
Škrinjar, P., Eberhardt, J., Studer, G., Tauriello, G., Schwede, T., & Durairaj, J. (2026) · Nature Structural & Molecular Biology · doi:10.1038/s41594-026-01797-5 · Cited by 22 · see citing works
“To evaluate the accuracy of predicted models, we use the compare-ligand-structures action of OpenStructure 34 (v.2.8.0), which returns the binding-site superposed symmetry-corrected ligand r.m.s.d. (hereafter referred as just r.m.s.d.), the LDDT-LP and the LDDT-PLI.”
InfluentialSelf-citationFast remote nucleotide sequence alignment with Riboseek
Park, S., Didi, K., Favor, A., Bushuiev, A., Kim, S., Mirdita, M., & Steinegger, M. (2026) · bioRxiv · doi:10.64898/2026.07.31.741718 · Cited by 2
Stoic: fast and accurate protein stoichiometry prediction
Litvinov, D., Pantolini, L., Škrinjar, P., Tauriello, G., McCafferty, C. L., Engel, B. D., Schwede, T., & Durairaj, J. (2026) · bioRxiv · doi:10.1093/bioinformatics/btag448 · Cited by 1 · see citing works
“Our Naive baseline simply predicts a copy number of one for all entities in the complex, the current approach used in CAMEO (Robin et al .”
InfluentialSelf-citationFu, J.-M., Milam, V., & McShan, A. C. (2026) · Nucleic Acids Research · doi:10.1093/nar/gkag999
“11.0 [ 72 ].”
InfluentialOlechnovič, K. (2026) · Bioinformatics · doi:10.1093/bioinformatics/btag756
“CAD-score has since been used in CASP and CAMEO evaluation workflows (Kryshtafovych et al. 2014, Robin et al. 2021, Studer et al. 2026), shown sensitivity to important features of protein models (Olechnovi � c et al. 2019), and supported downstream applications such as complex clustering (Dapk �…”
Targeted finetuning enables co-folding models to learn ligand-induced protein conformational states
Gorantla, R., Schleberger, C., & Sesterhenn, F. (2026) · bioRxiv · doi:10.64898/2026.09.21.752570
Benchmarking protein-ligand co-folding across Nav, Cav and Kv-family channels
Zhu, Y., & Rahman, T. (2026) · Frontiers in Biophysics · doi:10.3389/frbis.2026.1937302
How Post-Training Shapes Biological Reasoning Models
Fesser, L., Zhang, H.-L., Li, M. M., Wang, E., Perozzi, B., Azizi, S., Kakade, S., & Zitnik, M. (2026) · arXiv.org · doi:10.48550/arxiv.2606.16517
“In biology, unseen pathways, diseases, species, and perturbations often involve different mechanisms and biological processes [20, 21].”
Improving AlphaFold3 by Engineering MSA and Template InputsPreprint
Neupane, P., Liu, J., & Cheng, J. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.04.22.720119 · Cited by 3 · see citing works
“GDT-TS and GDC-SC scores were calculated using the lga tool (Local–Global Alignment) [44], following the standard CASP evaluation protocol, whereas lDDT was calculated using the implementation [45] in the OpenStructure frame-199 work [46].”
AI-Driven De Novo Binder Design: From Structure Prediction to Closed-Loop Optimization
Li, X., Fan, Z., Yang, J., Jiang, R., Cao, F., Li, J., Xie, D., Meng, X., Sun, M., & Wang, L. (2026) · Computational and Structural Biotechnology Journal · doi:10.34133/csbj.0070 · Cited by 1 · see citing works
- Authors
- Studer, G., Robin, X., Bienert, S., Durairaj, J., Škrinjar, P., Tauriello, G., Waterhouse, A. M., & Schwede, T.
- Affiliations
- SIB Swiss Institute of Bioinformatics (CH)
- Corresponding author
- Schwede, Torsten
- ORCID
- Venue
- Nature Methods
- Published
- 2025-12-22
- Volume
- 23
- Issue
- 2
- Pages
- 387-394
- ISSN
- 1548-7091 (ISSN-L)
- 1548-7105
- PMC
- PMC12904798
- OpenAlex
- W7116793565
- Cited by
- 10
- References
- 80
- Citations by year
- 2026: 6 · 2025: 4
- FWCI
- 0.89
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Nature Methods (publishedVersion, cc-by)
- Research Square (acceptedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- Funders
- SIB - Swiss Institute of Bioinformatics Biozentrum - University of Basel
- Universität Basel
- Biozentrum - University of Basel
- SIB - Swiss Institute of Bioinformatics
- Swiss National Science Foundation
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Physical Sciences › Materials Science › Materials Chemistry › Enzyme Structure and Function
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Computational Drug Discovery Methods
- Keywords
- benchmarking, quaternary structure, protein-protein interfaces, protein-ligand complex, protein structure prediction
- MeSH terms
- Algorithms, Models, Molecular, Protein Conformation, Proteins, Software, Computational Biology, Benchmarking, Databases, Protein, Macromolecular Substances
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
