CoChat
Sign Up
Cited by 143Computational Drug Discovery MethodsMachine Learning in Materials Sciencevaccines and immunoinformatics approaches

Beware of machine learning-based scoring functions-on the danger of developing black boxes.

Gabel, Joffrey, Desaphy, Jérémy, Rognan, Didier·2014·Journal of Chemical Information and Modeling
DOI 10.1021/ci500406k
Open this paper in CoChat — free account

Abstract

Training machine learning algorithms with protein-ligand descriptors has recently gained considerable attention to predict binding constants from atomic coordinates. Starting from a series of recent reports stating the advantages of this approach over empirical scoring functions, we could indeed reproduce the claimed superiority of Random Forest and Support Vector Machine-based scoring functions to predict experimental binding constants from protein-ligand X-ray structures of the PDBBind dataset. Strikingly, these scoring functions, trained on simple protein-ligand element-element distance counts, were almost unable to enrich virtual screening hit lists in true actives upon docking experiments of 10 reference DUD-E datasets; this is a a feature that, however, has been verified for an a priori less-accurate empirical scoring function (Surflex-Dock). By systematically varying ligand poses from true X-ray coordinates, we show that the Surflex-Dock scoring function is logically sensitive to the quality of docking poses. Conversely, our machine-learning based scoring functions are totally insensitive to docking poses (up to 10 Å root-mean square deviations) and just describe atomic element counts. This report does not disqualify using machine learning algorithms to design scoring functions. Protein-ligand element-element distance counts should however be used with extreme caution and only applied in a meaningful way. To avoid developing novel but meaningless scoring functions, we propose that two additional benchmarking tests must be systematically done when developing novel scoring functions: (i) sensitivity to docking pose accuracy, and (ii) ability to enrich hit lists in true actives upon structure-based (docking, receptor-ligand pharmacophore) virtual screening of reference datasets.

Integrity signals

Registry

Crossref · 10.1021/ci500406k

Cites

32 of 32 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The conjunction of nonlinear models and more straightforward atom counting descriptors is subject to significant criticisms.

    “However, the conjunction of nonlinear models and more straightforward atom counting descriptors is subjected to significant criticisms (Gabel et al., 2014).”

    From Empirical scoring functions for structure-based virtual screening: applications, critical aspects, and challenges (2018) · cites this paper as Gabel et al., 2014

  • The main drawback of nonlinear scoring functions is that they work as "black boxes" since the relationship between descriptors is often vague, requiring careful use to avoid meaningless interpretations.

    “Despite nonlinear scoring functions have the main advantage of discarding the necessity of a pre-defined functional form, their main drawback is that they work as “black boxes” since the relationship between the descriptors is often vague, requiring careful use to avoid meaningless interpretations (Gabel et al., 2014).”

    From Empirical scoring functions for structure-based virtual screening: applications, critical aspects, and challenges (2018) · cites this paper as Gabel et al., 2014

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Gabel, J., Desaphy, J., & Rognan, D.
Affiliations
Centre National de la Recherche Scientifique (FR)
Corresponding author
Rognan, Didier
ORCID
Venue
Journal of Chemical Information and Modeling
Publisher
American Chemical Society (ACS)
Published
2014-09-24
Volume
54
Issue
10
Pages
2807-2815
ISSN
  • 1549-9596 (ISSN-L)
  • 1549-960X
DOI
10.1021/ci500406k
OpenAlex
W2040990879
Cited by
143
References
32
Citations by year
2026: 5 · 2025: 11 · 2024: 17 · 2023: 15 · 2022: 17 · 2021: 15 · 2020: 11 · 2019: 15 · 2018: 7 · 2017: 10 · 2016: 12 · 2015: 8
FWCI
10.32
Type
journal article
Language
en
Open access
closed
Locations
  • Journal of Chemical Information and Modeling (publishedVersion)
  • PubMed (publishedVersion)
Funders
Agence Nationale de la Recherche
Topics
  • Physical Sciences › Computer Science › Computational Theory and Mathematics › Computational Drug Discovery Methods
  • Physical Sciences › Materials Science › Materials Chemistry › Machine Learning in Materials Science
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › vaccines and immunoinformatics approaches
Keywords
machine learning scoring functions, protein-ligand binding affinity prediction, black-box models, PDBbind, DUD-E, structure-based virtual screening, random forest, support vector machine
MeSH terms
Binding Sites, Enzyme Inhibitors, Humans, Ligands, Protein Binding, Proteins, Research Design, Thermodynamics, User-Computer Interface, Artifacts, Crystallography, X-Ray, Small Molecule Libraries, Molecular Dynamics Simulation, High-Throughput Screening Assays, Support Vector Machine, Molecular Docking Simulation, Databases, Chemical

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review