The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Evaluating Virtual Screening Methods: Good and Bad Metrics for the “Early Recognition” Problem
Abstract
Many metrics are currently used to evaluate the performance of ranking methods in virtual screening (VS), for instance, the area under the receiver operating characteristic curve (ROC), the area under the accumulation curve (AUAC), the average rank of actives, the enrichment factor (EF), and the robust initial enhancement (RIE) proposed by Sheridan et al. In this work, we show that the ROC, the AUAC, and the average rank metrics have the same inappropriate behaviors that make them poor metrics for comparing VS methods whose purpose is to rank actives early in an ordered list (the "early recognition problem"). In doing so, we derive mathematical formulas that relate those metrics together. Moreover, we show that the EF metric is not sensitive to ranking performance before and after the cutoff. Instead, we formally generalize the ROC metric to the early recognition problem which leads us to propose a novel metric called the Boltzmann-enhanced discrimination of receiver operating characteristic that turns out to contain the discrimination power of the RIE metric but incorporates the statistical significance from ROC and its well-behaved boundaries. Finally, two major sources of errors, namely, the statistical error and the "saturation effects", are examined. This leads to practical recommendations for the number of actives, the number of inactives, and the "early recognition" importance parameter that one should use when comparing ranking methods. Although this work is applied specifically to VS, it is general and can be used to analyze any method that needs to segregate actives toward the front of a rank-ordered list.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856
Measuring the Accuracy of Diagnostic Systems
Swets, J. A. (1988) · Science · doi:10.1126/science.3287615 · Cited by 10,058
Halgren, T. A., Murphy, R. B., Friesner, R. A., Beard, H. S., Frye, L. L., Pollard, W. T., & Banks, J. L. (2004) · Journal of Medicinal Chemistry · doi:10.1021/jm030644s · Cited by 5,177
Bamber, D. (1975) · Journal of Mathematical Psychology · doi:10.1016/0022-2496(75)90001-2 · Cited by 1,439
Triballeau, N., Acher, F., Brabet, I., Pin, J.-P., & Bertrand, H.-O. (2005) · Journal of Medicinal Chemistry · doi:10.1021/jm049092j · Cited by 631
Better Decisions through Science
Swets, J. A., Dawes, R. M., & Monahan, J. (2000) · Scientific American · doi:10.1038/scientificamerican1000-82 · Cited by 567
Assessing Scoring Functions for Protein−Ligand Interactions
Ferrara, P., Gohlke, H., Price, D. J., Klebe, G., & Brooks, C. L. (2004) · Journal of Medicinal Chemistry · doi:10.1021/jm030489h · Cited by 481
A detailed comparison of current docking and scoring methods on systems of pharmaceutical relevance
Perola, E., Walters, W. P., & Charifson, P. S. (2004) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.20088 · Cited by 446
Annual Reports in Computational Chemistry
(2009) · Annual Reports in Computational Chemistry · doi:10.1016/s1574-1400(09)00513-1 · Cited by 381
Cost curves: An improved method for visualizing classifier performance
Drummond, C., & Holte, R. C. (2006) · Machine Learning · doi:10.1007/s10994-006-8199-5 · Cited by 326
Cited by
Saito, T., & Rehmsmeier, M. (2015) · PLOS ONE · doi:10.1371/journal.pone.0118432 · Cited by 5,309
Molecular graph convolutions: moving beyond fingerprints
Kearnes, S., McCloskey, K., Berndl, M., Pande, V., & Riley, P. (2016) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-016-9938-8 · Cited by 1,631
Automated design of ligands to polypharmacological profiles
Besnard, J., Ruda, G. F., Setola, V., Abecassis, K., Rodriguiz, R. M., Huang, X.-P., Norval, S., Sassano, M. F., Shin, A. I., Webster, L. A., et al. (2012) · Nature · doi:10.1038/nature11691 · Cited by 832
Validation of Molecular Docking Programs for Virtual Screening against Dihydropteroate Synthase
Hevener, K. E., Zhao, W., Ball, D. M., Babaoglu, K., Qi, J., White, S. W., & Lee, R. E. (2009) · Journal of Chemical Information and Modeling · doi:10.1021/ci800293n · Cited by 668
Key Topics in Molecular Docking for Drug Design
Torres, P. H. M., Sodero, A. C. R., Jofily, P., & Silva-Jr, F. P. (2019) · International Journal of Molecular Sciences · doi:10.3390/ijms20184574 · Cited by 641
Moitessier, N., Englebienne, P., Lee, D., Lawandi, J., & Corbeil, C. R. (2008) · British Journal of Pharmacology · doi:10.1038/sj.bjp.0707515 · Cited by 514
Variability in docking success rates due to dataset preparation.
Corbeil, C. R., Williams, C. I., & Labute, P. (2012) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-012-9570-1 · Cited by 469
Recognizing Pitfalls in Virtual Screening: A Critical Review
Scior, T., Bender, A., Tresadern, G., Medina-Franco, J. L., Martínez-Mayorga, K., Langer, T., Cuanalo-Contreras, K., & Agrafiotis, D. K. (2012) · Journal of Chemical Information and Modeling · doi:10.1021/ci200528d · Cited by 438
Ain, Q. U., Aleksandrova, A., Roessler, F. D., & Ballester, P. J. (2015) · WIREs Computational Molecular Science · doi:10.1002/wcms.1225 · Cited by 357
Lenselink, E. B., ten Dijke, N., Bongers, B., Papadatos, G., van Vlijmen, H. W. T., Kowalczyk, W., IJzerman, A. P., & van Westen, G. J. P. (2017) · Journal of Cheminformatics · doi:10.1186/s13321-017-0232-0 · Cited by 334
Sastry, M., Lowrie, J. F., Dixon, S. L., & Sherman, W. (2010) · Journal of Chemical Information and Modeling · doi:10.1021/ci100062n · Cited by 332
Beware of machine learning-based scoring functions-on the danger of developing black boxes.
Gabel, J., Desaphy, J., & Rognan, D. (2014) · Journal of Chemical Information and Modeling · doi:10.1021/ci500406k · Cited by 143
Benchmarking Refined and Unrefined AlphaFold2 Structures for Hit Discovery
(2023) · doi:10.1021/acs.jcim.2c01219 · Cited by 136
Systematic exploitation of multiple receptor conformations for virtual ligand screening.
Bottegoni, G., Rocchia, W., Rueda, M., Abagyan, R., & Cavalli, A. (2011) · PLoS ONE · doi:10.1371/journal.pone.0018845 · Cited by 96
WScore: a flexible and accurate treatment of explicit water molecules in ligand-receptor docking.
Murphy, R. B., Repasky, M. P., Greenwood, J. R., Tubert-Brohman, I., Jerome, S., Annabhimoju, R., Boyles, N. A., Schmitz, C. D., Abel, R., Farid, R., et al. (2016) · Journal of Medicinal Chemistry · doi:10.1021/acs.jmedchem.6b00131 · Cited by 89
How to Achieve Better Results Using PASS-Based Virtual Screening: Case Study for Kinase Inhibitors
Pogodin, P. V., Lagunin, A. A., Rudik, A. V., Filimonov, D. A., Druzhilovskiy, D. S., Nicklaus, M. C., & Poroikov, V. V. (2018) · Frontiers in Chemistry · doi:10.3389/fchem.2018.00133 · Cited by 40
Barsottini, M. R., Pires, B. A., Vieira, M. L., Pereira, J. G., Costa, P. C., Sanitá, J., Coradini, A., Mello, F., Marschalk, C., Silva, E. M., et al. (2019) · Pest Management Science · doi:10.1002/ps.5243 · Cited by 19
A meta learning and task adaptive approach for drug target affinity prediction
Wan, M., Zhao, Y., Zhang, Y., Xu, H., Yi, D., Zan, P., He, S., & Bo, X. (2026) · Nature Communications · doi:10.1038/s41467-026-70554-5 · Cited by 4
Deep learning molecular interaction motifs from receptor structures alone
Kim, S., Oh, S., Woo, H., Sim, J., Seok, C., & Park, H. (2025) · Journal of Cheminformatics · doi:10.1186/s13321-025-01055-8 · Cited by 3
Shin, W.-H., Kagaya, Y., Jang, W., & Kihara, D. (2026) · Journal of Cheminformatics · doi:10.1186/s13321-026-01224-3 · Cited by 3
Zhang, Y., Ishitani, R., Takemoto, M., & Tomita, A. (2025) · Communications Chemistry · doi:10.1038/s42004-025-01844-0 · Cited by 1
Metadata
- Authors
- Truchon, J.-F., & Bayly, C. I.
- Venue
- Journal of Chemical Information and Modeling
- Publisher
- American Chemical Society (ACS)
- Published
- 2007-02-09
- Volume
- 47
- Issue
- 2
- Pages
- 488-508
- ISSN
- 1549-9596 (ISSN-L)
- 1549-960X
- OpenAlex
- W2042110087
- Cited by
- 874
- References
- 17
- FWCI
- 34.25
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
