Automated benchmarking of combined protein structure and ligand conformation prediction
Open accessThe DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Abstract
The prediction of protein‐ligand complexes (PLC), using both experimental and predicted structures, is an active and important area of research, underscored by the inclusion of the Protein‐Ligand Interaction category in the latest round of the Critical Assessment of Protein Structure Prediction experiment CASP15. The prediction task in CASP15 consisted of predicting both the three‐dimensional structure of the receptor protein as well as the position and conformation of the ligand. This paper addresses the challenges and proposed solutions for devising automated benchmarking techniques for PLC prediction. The reliability of experimentally solved PLC as ground truth reference structures is assessed using various validation criteria. Similarity of PLC to previously released complexes are employed to judge PLC diversity and the difficulty of a PLC as a prediction target. We show that the commonly used PDBBind time‐split test‐set is inappropriate for comprehensive PLC evaluation, with state‐of‐the‐art tools showing conflicting results on a more representative and high quality dataset constructed for benchmarking purposes. We also show that redocking on crystal structures is a much simpler task than docking into predicted protein models, demonstrated by the two PLC‐prediction‐specific scoring metrics created. Finally, we introduce a fully automated pipeline that predicts PLC and evaluates the accuracy of the protein structure, ligand pose, and protein–ligand interactions.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
Nextflow enables reproducible computational workflows
Di Tommaso, P., Chatzou, M., Floden, E. W., Barja, P. P., Palumbo, E., & Notredame, C. (2017) · Nature Biotechnology · doi:10.1038/nbt.3820 · Cited by 4,001
Singularity: Scientific containers for mobility of compute
Kurtzer, G. M., Sochat, V., & Bauer, M. W. (2017) · PLOS ONE · doi:10.1371/journal.pone.0177459 · Cited by 2,740
Protein Data Bank: the single global archive for 3D macromolecular structure data
wwPDB consortium, Burley, S. K., Berman, H. M., Bhikadiya, C., Bi, C., Chen, L., Costanzo, L. D., Christie, C., Duarte, J. M., Dutta, S., et al. (2019) · Nucleic Acids Research · doi:10.1093/nar/gky949 · Cited by 1,163
Clustering huge protein sequence sets in linear time
Steinegger, M., & Söding, J. (2018) · Nature Communications · doi:10.1038/s41467-018-04964-5 · Cited by 875 · cited 1×
“49 Six different sequence-based clustering patterns were obtained as a result of clustering with minimum sequence identity thresholds of 100%, 95%, 90%, 70%, 50%, and 30%, respectively.”
MethodForging the basis for developing protein-ligand interaction scoring functions.
Liu, Z., Su, M., Han, L., Liu, J., Yang, Q., Li, Y., & Wang, R. (2017) · Accounts of Chemical Research · doi:10.1021/acs.accounts.6b00491 · Cited by 493
McGovern, S. L., & Shoichet, B. K. (2003) · Journal of Medicinal Chemistry · doi:10.1021/jm0300330 · Cited by 276
D3R grand challenge 2015: evaluation of protein-ligand pose and affinity predictions.
Gathiaka, S., Liu, S., Chiu, M., Yang, H., Stuckey, J. A., Kang, Y. N., Delproposto, J., Kubish, G., Dunbar, J. B., Carlson, H. A., et al. (2016) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-016-9946-8 · Cited by 207
Impact of protein conformational diversity on AlphaFold predictions
Saldaño, T., Escobedo, N., Marchetti, J., Zea, D. J., Mac Donagh, J., Velez Rueda, A. J., Gonik, E., Melani, A. G., Novomisky Nechcoff, J., Salas, M. N., et al. (2021) · doi:10.1101/2021.10.27.466189 · Cited by 196 · cited 1×
“Despite AlphaFold being shown to prefer prediction of the holo form, 21 homology modeling was recently shown to be better at the PLC prediction task.”
Impact of protein conformational diversity on AlphaFold predictions
Saldaño, T., Escobedo, N., Marchetti, J., Zea, D. J., Mac Donagh, J., Velez Rueda, A. J., Gonik, E., García Melani, A., Novomisky Nechcoff, J., Salas, M. N., et al. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac202 · Cited by 196
Research works printed but not linked: 39
- 10.1016/j.jmgm.2012.07.007 · DOI · Paper page
- 10.1016/j.str.2019.05.012 · DOI
- 10.1007/s10822‐018‐0180‐4 · DOI · Paper page
- 10.1007/s10822‐020‐00289‐y · DOI · Paper page
- 10.1021/acs.jcim.5b00523 · DOI · Paper page
- 10.1021/acs.jcim.5b00387 · DOI · Paper page
- 10.1002/prot.26601 · DOI · Paper page
- 10.1002/prot.26213 · DOI · Paper page
- 10.1002/prot.25815 · DOI · Paper page
- 10.1093/database/bat031 · DOI · Paper page
- 10.1107/S2059798322011901 · DOI
- 10.1016/j.drudis.2012.06.011 · DOI · Paper page
- 10.1021/ci9003943 · DOI · Paper page
- 10.1016/j.isci.2022.105920 · DOI · Paper page
- 10.1093/nar/28.1.235 · DOI · Paper page
- 10.1107/S2059798318002541 · DOI · Paper page
- Stärk H, 2022, EquiBind: geometric deep learning for drug binding structure prediction, arXiv
- 10.1101/2022.06.06.495043 · DOI
- Corso G, 2022, DiffDock: diffusion steps, twists, and turns for molecular docking, arXiv
- 10.12688/f1000research.20559.1 · DOI
- 10.1021/acs.jcim.1c00203 · DOI · Paper page
- 10.1002/jcc.21334 · DOI · Paper page
- 10.1021/ci300604z · DOI · Paper page
- 10.1186/s13321‐021‐00522‐2 · DOI · Paper page
- 10.20982/tqmp.16.5.S003 · DOI
- 10.1186/s13321‐018‐0285‐8 · DOI · Paper page
- 10.3389/fbinf.2022.959160 · DOI · Paper page
- 10.1002/prot.26562 · DOI · Paper page
- 10.1038/s41592‐020‐0731‐1 · DOI · Paper page
- 10.7554/eLife.89386 · doi:10.7554/elife.89386
- 10.1093/bioinformatics/btu789 · DOI · Paper page
- 10.1016/j.str.2011.08.006 · DOI · Paper page
- 10.1107/S0907444911050359 · DOI · Paper page
- 10.1016/j.str.2017.10.009 · DOI · Paper page
- 10.1093/nar/gks966 · DOI · Paper page
- 10.1016/j.jmb.2020.11.003 · DOI · Paper page
- 10.1021/acs.jcim.5b00654 · DOI · Paper page
- 10.1006/jmbi.1998.2401 · DOI · Paper page
- 10.1107/S0907444913007051 · DOI · Paper page
Cited by
Modelling protein complexes with crosslinking mass spectrometry and deep learning
(2024) · doi:10.1038/s41467-024-51771-2 · Cited by 94
Evaluating generalization in protein–ligand cofolding methods
Škrinjar, P., Eberhardt, J., Studer, G., Tauriello, G., Schwede, T., & Durairaj, J. (2026) · Nature Structural & Molecular Biology · doi:10.1038/s41594-026-01797-5 · Cited by 22
Self-citationModelling protein complexes with crosslinking mass spectrometry and deep learning
Stahl, K., Demann, L., Bremenkamp, R., Warneke, R., Hormes, B., Stülke, J., Brock, O., & Rappsilber, J. (2023) · doi:10.1101/2023.06.07.544059 · Cited by 20
Accurate prediction of CDR-H3 loop structures of antibodies with deep learning
(2024) · doi:10.7554/elife.91512.4 · Cited by 16
A fully automated benchmarking suite to compare macromolecular complexes
Studer, G., Robin, X., Bienert, S., Durairaj, J., Škrinjar, P., Tauriello, G., Waterhouse, A. M., & Schwede, T. (2026) · Nature Methods · doi:10.1038/s41592-025-02973-z · Cited by 10
“Automating structural quality validation is challenging, and we are working on incorporating corresponding checks into the benchmark dataset creation process 22,54 and separately into the scores themselves.”
Self-citation
Metadata
- Authors
- Leemann, M., Sagasta, A., Eberhardt, J., Schwede, T., Robin, X., & Durairaj, J.
- Venue
- Proteins: Structure, Function, and Bioinformatics
- Publisher
- Wiley
- Published
- 2023-10-26
- Volume
- 91
- Issue
- 12
- Pages
- 1912-1924
- ISSN
- 0887-3585 (ISSN-L)
- 1097-0134
- OpenAlex
- W4387965042
- Cited by
- 23
- References
- 16
- FWCI
- 3.03
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Funders
- Horizon 2020 Framework Programme
- Swiss Institute of Bioinformatics
- Universität Basel
- European Commission (956137, 676559)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.