CoChat
Sign Up
PreprintCited by 28Computational Drug Discovery MethodsMachine Learning in BioinformaticsProtein Structure and Dynamics

The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.

Robustly interrogating machine learning-based scoring functions: what are they learning?

Open access
Durant, Guy, Boyles, Fergus, Birchall, Kristian, Marsden, Brian, Deane, Charlotte M.·2023·Bioinformatics
DOI 10.1101/2023.10.30.564251PMC PMC11821266

Preprint — published as 10.1093/bioinformatics/btaf040

Open this paper in CoChat — free account

Abstract

Motivation: Machine learning-based scoring functions (MLBSFs) have been found to exhibit inconsistent performance on different benchmarks and be prone to learning dataset bias. For the field to develop MLBSFs that learn a generalisable understanding of physics, a more rigorous understanding of how they perform is required. Results: In this work, we compared the performance of a diverse set of popular MLBSFs (RFScore, SIGN, OnionNet-2, Pafnucy, and PointVS) to our proposed baseline models that can only learn dataset biases on a range of benchmarks. We found that these baseline models were competitive in accuracy to these MLBSFs in almost all proposed benchmarks, indicating these models only learn dataset biases. Our tests and provided platform, ToolBoxSF, will enable researchers to robustly interrogate MLBSF performance and determine the effect of dataset biases on their predictions. Availability and Implementation: https://github.com/guydurant/toolboxsf Contact: deane@stats.ox.ac.uk Supplementary information: Supplementary data are available at Bioinformatics online.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1101/2023.10.30.564251

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 60

  1. 10.1093/bioinformatics/btq112 · DOI
  2. 10.1021/ci500091r · DOI
  3. 10.1021/jm001044l · DOI · Paper page
  4. 2020, Learning from the ligand: using ligand-based features to improve binding affinity prediction, Bioinformatics, 36, 758, 10.1093/bioinformatics/btz665 · DOI
  5. 2022, Learning from Docked Ligands: Ligand-Based Features Rescue Structure-Based Scoring Functions When Trained on Docked Poses, Journal of Chemical Information and Modeling, 62, 5329, 10.1021/acs.jcim.1c00096 · DOI
  6. 2016, CSAR 2014: a benchmark exercise using unpublished data from pharma, Journal of chemical information and modeling, 56, 1063, 10.1021/acs.jcim.5b00523 · DOI
  7. 10.1021/ci2003889 · DOI
  8. 10.1021/acs.jcim.0c00411 · DOI
  9. 10.1021/jm0306430 · DOI · Paper page
  10. 10.1006/jmbi.1999.3371 · DOI
  11. 10.1002/(SICI)1099-1352(199601)9:1<1::AID-JMR241>3.0.CO;2-6 · DOI
  12. Huang, N. , Kalyanaraman, C. , Jacobson, M. P. , and Bernacki, K. (2006). Molecular mechanics methods for predicting protein-ligand binding. Article in Physical Chemistry Chemical Physics. · DOI
  13. 10.1021/acs.jcim.7b00650 · DOI
  14. 10.1371/journal.pone.0061007 · DOI
  15. 2020, GraphDelta: MPNN Scoring Function for the Affinity Prediction of Protein-Ligand Complexes, ACS Omega, 5, 5150, 10.1021/acsomega.9b04162 · DOI
  16. 10.1021/ci300604z · DOI
  17. 2010, Leave-cluster-out cross-validation is appropriate for scoring functions derived from diverse protein data sets, Journal of Chemical Information and Modeling, 50, 1961, 10.1021/ci100264e · DOI
  18. 10.1016/j.jmgm.2004.11.007 · DOI
  19. 2015, Improving AutoDock Vina Using Random Forest: The Growing Accuracy of Binding Affinity Prediction by the Effective Exploitation of Larger Data Sets, Molecular Informatics, 34, 115, 10.1002/minf.201400132 · DOI · Paper page
  20. Li, S. , Zhou, J. , Xu, T. , Huang, L. , Wang, F. , Xiong, H. , Huang, W. , Dou, D. , and Xiong, H. (2021). Structure-aware Interactive Graph Neural Networks for the Prediction of Protein-Ligand Binding Affinity; Structure-aware Interactive Graph Neural Networks for the Prediction of Protein-Ligand Binding Affinity. KDD, 21. · DOI
  21. 2014, PDB-wide collection of binding data: current status of the PDBbind database, Bioinformatics, 31, 405
  22. 10.1016/j.str.2009.11.012 · DOI
  23. 2021, GNINA 1.0: molecular docking with deep learning, Journal of Cheminformatics, 13, 1
  24. 2021, Learning protein-ligand binding affinity with atomic environment vectors, Journal of Cheminformatics, 13, 1
  25. Meli, R. , Morris, G. M. , and Biggin, P. C. (2022). Scoring Functions for Protein-Ligand Binding Affinity Prediction Using Structure-based Deep Learning: A Review. Frontiers in Bioinformatics, 2. · DOI
  26. 2017, Predicting Binding Free Energies: Frontiers and Benchmarks, Annual review of biophysics, 46, 531, 10.1146/annurev-biophys-070816-033654 · DOI
  27. 2022, PIGNet: a physics-informed deep learning model toward generalized drug–target interaction predictions, Chemical Science, 13, 3661, 10.1039/D1SC06946B · DOI
  28. 10.1002/jcc.21256 · DOI
  29. 2011, Open Babel: An open chemical toolbox, Journal of cheminformatics, 3, 1
  30. 2012, InChIKey collision resistance: an experimental testing, Journal of cheminformatics, 4, 1
  31. 10.1021/acs.jcim.5b00654 · DOI
  32. Scantlebury, J. , Vost, L. , Carbery, A. , Hadfield, T. E. , Turnbull, O. M. , Brown, N. , Chenthamarakshan, V. , Das, P. , Grosjean, H. , von Delft, F. , and Deane, C. M. (2023). A Step Towards Generalisability: Training a Machine Learning Scoring Function for Structure-Based Virtual Screening. bioRxiv, page 2022.10.28.511712. · DOI
  33. 2023, How good are AlphaFold models for docking-based virtual screening?, iScience, 26, 105920, 10.1016/j.isci.2022.105920 · DOI · Paper page
  34. 2021, The impact of cross-docked poses on performance of machine learning classifier for protein–ligand binding pose prediction, Journal of Cheminformatics, 13, 1
  35. 10.1093/bioinformatics/bty374 · DOI
  36. 10.1021/acs.jcim.8b00545 · DOI
  37. 10.1002/jcc.21334 · DOI · Paper page
  38. 10.1002/prot.10465 · DOI
  39. 2022, On the Frustration to Predict Binding Affinities from Protein-Ligand Structures with Deep Neural Networks, Journal of Medicinal Chemistry, 65, 7946, 10.1021/acs.jmedchem.2c00487 · DOI
  40. 10.1002/jcc.24667 · DOI
  41. Wang, C. , Wu, Q. , Weimer, M. , and Zhu, E. (2021a). FLAML: A Fast and Lightweight AutoML Library. In A. Smola , A. Dimakis , and I. Stoica , editors, Proceedings of Machine Learning and Systems, volume 3, pages 434–447.
  42. 10.1021/jm030580l · DOI
  43. 2021, OnionNet-2: A Convolutional Neural Network Model for Predicting Protein-Ligand Binding Affinity Based on Residue-Atom Contacting Shells, Frontiers in Chemistry, 9, 913
  44. 10.15252/MSB.202211081 · DOI · Paper page
  45. 2022, Assessment of the Generalization Abilities of Machine-Learning Scoring Functions for Structure-Based Virtual Screening, Journal of Chemical Information and Modeling, 62, 5485, 10.1021/acs.jcim.2c01149 · DOI
  46. 10.1021/ci400120b · DOI
  47. 10.1093/bioinformatics/btq112 · DOI
  48. Evans, R. , O’neill, M. , Pritzel, A. , Antropova, N. , Senior, A. , Green, T. , Žídek, A. , Bates, R. , Blackwell, S. , Yim, J. , Ronneberger, O. , Bodenstein, S. , Zielinski, M. , Bridgland, A. , Potapenko, A. , Cowie, A. , Tunyasuvunakool, K. , Jain, R. , Clancy, E. , Kohli, P. , Jumper, J. , and Hassabis, D. (2022). Protein complex prediction with AlphaFold-Multimer. · DOI · Paper page
  49. 10.1002/pro.3235 · DOI · Paper page
  50. Li, S. , Zhou, J. , Xu, T. , Huang, L. , Wang, F. , Xiong, H. , Huang, W. , Dou, D. , and Xiong, H. (2021). Structure-aware Interactive Graph Neural Networks for the Prediction of Protein-Ligand Binding Affinity; Structure-aware Interactive Graph Neural Networks for the Prediction of Protein-Ligand Binding Affinity. KDD, 21. · DOI
  51. 2019, MGnify: the microbiome analysis resource in 2020, Nucleic Acids Research, 48, D570
  52. 10.1524/auto.2011.0951 · DOI
  53. Scantlebury, J. , Vost, L. , Carbery, A. , Hadfield, T. E. , Turnbull, O. M. , Brown, N. , Chenthamarakshan, V. , Das, P. , Grosjean, H. , von Delft, F. , and Deane, C. M. (2023). A Step Towards Generalisab ility: Training a Machine Learning Scoring Function for Structure-Based Virtual Screening. bioRxiv, page 2022.10.28.511712. · DOI
  54. 10.1093/bioinformatics/bty374 · DOI
  55. 2010, Drawing the PDB: ProteinLigand Complexes in Two Dimensions, ACS Medicinal Chemistry Letters, 1, 540, 10.1021/ml100164p · DOI
  56. 10.1093/bioinformatics/btm098 · DOI · Paper page
  57. 2021, OnionNet-2: A Convolutional Neural Network Model for Predicting Protein-Ligand Binding Affinity Based on Residue-Atom Contacting Shells, Frontiers in Chemistry, 9, 913
  58. 2015, Open Drug Discovery Toolkit (ODDT): a new open-source player in the drug discovery field, Journal of cheminformatics, 7, 1
  59. 2004, Scoring function for automated assessment of protein structure template quality, Proteins: Structure, Function, and Bioinformatics, 57, 702, 10.1002/prot.20264 · DOI · Paper page
  60. 10.1093/nar/gki524 · DOI · Paper page

Metadata

Authors
Durant, G., Boyles, F., Birchall, K., Marsden, B., & Deane, C. M.
Affiliations
  • University of Oxford (GB)
  • LifeArc (GB)
Corresponding author
Deane, Charlotte M.
ORCID
Venue
Bioinformatics
Publisher
openRxiv
Published
2023-11-02
Volume
41
Issue
2
ISSN
1367-4803 (ISSN-L)
DOI
10.1101/2023.10.30.564251
PMC
PMC11821266
OpenAlex
W4406887026
Cited by
28
References
1
Citations by year
2026: 11 · 2025: 11 · 2024: 6
FWCI
11.33
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Bioinformatics (publishedVersion, cc-by)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
Engineering and Physical Sciences Research Council (EP/S024093, EP/S024093/1)
Topics
  • Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Drug Discovery › Computational Drug Discovery Methods
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
Keywords
protein structure prediction, data bias, benchmarking, generalizability, signs, protein structure comparison
MeSH terms
Machine Learning, Algorithms, Computational Biology

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review