The DOI matches its Crossref record; 81 of the 81 references it cites were found in a registry or an index.
Comparative Assessment of Scoring Functions: The CASF-2016 Update
Abstract
In structure-based drug design, scoring functions are often employed to evaluate protein-ligand interactions. A variety of scoring functions have been developed so far, and thus, some objective benchmarks are desired for assessing their strength and weakness. The comparative assessment of scoring functions (CASF) benchmark developed by us provides an answer to this demand. CASF is designed as a "scoring benchmark", where the scoring process is decoupled from the docking process to depict the performance of scoring function more precisely. Here, we describe the latest update of this benchmark, i.e., CASF-2016. Each scoring function is still evaluated by four metrics, including "scoring power", "ranking power", "docking power", and "screening power". Nevertheless, the evaluation methods have been improved considerably in several aspects. A new test set is compiled, which consists of 285 protein-ligand complexes with high-quality crystal structures and reliable binding constants. A panel of 25 scoring functions are tested on CASF-2016 as a demonstration. Our results reveal that the performance of current scoring functions is more promising in terms of docking power than scoring, ranking, and screening power. Scoring power is somewhat correlated with ranking power, so are docking power and screening power. The results obtained on CASF-2016 may provide valuable guidance for the end users to make smart choices among available scoring functions. Moreover, CASF is created as an open-access benchmark so that other researchers can utilize it to test a wider range of scoring functions. The complete CASF-2016 benchmark will be released on the PDBbind-CN web server ( http://www.pdbbind-cn.org/casf.asp/ ) once this article is published.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Trott, O., & Olson, A. J. (2010) · Journal of Computational Chemistry · doi:10.1002/jcc.21334 · Cited by 38,412
Features and development of Coot
Emsley, P., Lohkamp, B., Scott, W. G., & Cowtan, K. (2010) · Acta Crystallographica Section D Biological Crystallography · doi:10.1107/s0907444910007493 · Cited by 29,915
Bootstrap methods: another look at the jackknife
Efron, B. (1979) · The Annals of Statistics · doi:10.1214/aos/1176344552 · Cited by 17,602
Lipinski, C. A., Lombardo, F., Dominy, B. W., & Feeney, P. J. (1997) · Advanced Drug Delivery Reviews · doi:10.1016/s0169-409x(96)00423-1 · Cited by 11,385
Friesner, R. A., Banks, J. L., Murphy, R. B., Halgren, T. A., Klicic, J. J., Mainz, D. T., Repasky, M. P., Knoll, E. H., Shelley, M., Perry, J. K., et al. (2004) · Journal of Medicinal Chemistry · doi:10.1021/jm0306430 · Cited by 10,057
The proof and measurement of association between two things
Spearman, C. (1904) · The American Journal of Psychology · doi:10.2307/1412159 · Cited by 8,039
The use of ranks to avoid the assumption of normality implicit in the analysis of variance
Friedman, M. (1937) · Journal of the American Statistical Association · doi:10.1080/01621459.1937.10503522 · Cited by 7,005
Friesner, R. A., Murphy, R. B., Repasky, M. P., Frye, L. L., Greenwood, J. R., Halgren, T. A., Sanschagrin, P. C., & Mainz, D. T. (2006) · Journal of Medicinal Chemistry · doi:10.1021/jm051256o · Cited by 6,851
Development and validation of a genetic algorithm for flexible docking 1 1Edited by F. E. Cohen
Jones, G., Willett, P., Glen, R. C., Leach, A. R., & Taylor, R. (1997) · Journal of Molecular Biology · doi:10.1006/jmbi.1996.0897 · Cited by 6,763
Halgren, T. A., Murphy, R. B., Friesner, R. A., Beard, H. S., Frye, L. L., Pollard, W. T., & Banks, J. L. (2004) · Journal of Medicinal Chemistry · doi:10.1021/jm030644s · Cited by 5,177
Docking and scoring in virtual screening for drug discovery: methods and applications
Kitchen, D. B., Decornez, H., Furr, J. R., & Bajorath, J. (2004) · Nature Reviews Drug Discovery · doi:10.1038/nrd1549 · Cited by 4,150
Better Bootstrap Confidence Intervals
Efron, B. (1987) · Journal of the American Statistical Association · doi:10.1080/01621459.1987.10478410 · Cited by 3,364
OPLS3: A Force Field Providing Broad Coverage of Drug-like Small Molecules and Proteins
Harder, E., Damm, W., Maple, J., Wu, C., Reboul, M., Xiang, J. Y., Wang, L., Lupyan, D., Dahlgren, M. K., Knight, J. L., et al. (2016) · Journal of Chemical Theory and Computation · doi:10.1021/acs.jctc.5b00864 · Cited by 3,047
Validation of the general purpose tripos 5.2 force field
Clark, M., Cramer, R. D., & Van Opdenbosch, N. (1989) · Journal of Computational Chemistry · doi:10.1002/jcc.540100804 · Cited by 2,812
A Comparison of Alternative Tests of Significance for the Problem of $m$ Rankings
Friedman, M. (1940) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177731944 · Cited by 2,785
CD-HIT Suite: a web server for clustering and comparing biological sequences
Huang, Y., Niu, B., Gao, Y., Fu, L., & Li, W. (2010) · Bioinformatics · doi:10.1093/bioinformatics/btq003 · Cited by 2,607
Directory of useful decoys, enhanced (DUD-E): better ligands and decoys for better benchmarking.
Mysinger, M. M., Carchia, M., Irwin, J. J., & Shoichet, B. K. (2012) · Journal of Medicinal Chemistry · doi:10.1021/jm300687e · Cited by 2,412
Gaulton, A., Hersey, A., Nowotka, M., Bento, A. P., Chambers, J., Mendez, D., Mutowo, P., Atkinson, F., Bellis, L. J., Cibrián-Uhalte, E., et al. (2017) · Nucleic Acids Research · doi:10.1093/nar/gkw1074 · Cited by 2,330
A Critical Assessment of Docking Programs and Scoring Functions
Warren, G. L., Andrews, C. W., Capelli, A.-M., Clarke, B., LaLonde, J., Lambert, M. H., Lindvall, M., Nevins, N., Semus, S. F., Senger, S., et al. (2006) · Journal of Medicinal Chemistry · doi:10.1021/jm050362n · Cited by 1,814
Eldridge, M. D., Murray, C. W., Auton, T. R., Paolini, G. V., & Mee, R. P. (1997) · Journal of Computer-Aided Molecular Design · doi:10.1023/a:1007996124545 · Cited by 1,686
Gilson, M. K., Liu, T., Baitaluk, M., Nicola, G., Hwang, L., & Chong, J. (2016) · Nucleic Acids Research · doi:10.1093/nar/gkv1072 · Cited by 1,628
The ChEMBL bioactivity database: an update
Bento, A. P., Gaulton, A., Hersey, A., Bellis, L. J., Chambers, J., Davies, M., Krüger, F. A., Light, Y., Mak, L., McGlinchey, S., et al. (2014) · Nucleic Acids Research · doi:10.1093/nar/gkt1031 · Cited by 1,492
An Extension of Shapiro and Wilk's W Test for Normality to Large Samples
Royston, J. P. (1982) · Applied Statistics · doi:10.2307/2347973 · Cited by 1,399
Empirical Scoring Functions for Advanced Protein−Ligand Docking with PLANTS
Korb, O., Stützle, T., & Exner, T. E. (2009) · Journal of Chemical Information and Modeling · doi:10.1021/ci800298z · Cited by 1,349
Wang, R., Lai, L., & Wang, S. (2002) · Journal of Computer-Aided Molecular Design · doi:10.1023/a:1016357811882 · Cited by 1,191
Self-citationB�hm, H.-J. (1994) · Journal of Computer-Aided Molecular Design · doi:10.1007/bf00126743 · Cited by 1,011
A General and Fast Scoring Function for Protein−Ligand Interactions: A Simplified Potential Approach
Muegge, I., & Martin, Y. C. (1999) · Journal of Medicinal Chemistry · doi:10.1021/jm980536j · Cited by 1,007
Computer-based de novo design of drug-like molecules
Schneider, G., & Fechner, U. (2005) · Nature Reviews Drug Discovery · doi:10.1038/nrd1799 · Cited by 948
Automated docking with grid‐based energy evaluation
Meng, E. C., Shoichet, B. K., & Kuntz, I. D. (1992) · Journal of Computational Chemistry · doi:10.1002/jcc.540130412 · Cited by 938
Modified Sequentially Rejective Multiple Test Procedures
Shaffer, J. P. (1986) · Journal of the American Statistical Association · doi:10.1080/01621459.1986.10478341 · Cited by 806
Bissantz, C., Folkers, G., & Rognan, D. (2000) · Journal of Medicinal Chemistry · doi:10.1021/jm001044l · Cited by 740
Wang, Y., Bryant, S. H., Cheng, T., Wang, J., Gindulyte, A., Shoemaker, B. A., Thiessen, P. A., He, S., & Zhang, J. (2017) · Nucleic Acids Research · doi:10.1093/nar/gkw1118 · Cited by 671
Prediction of Protein−Ligand Interactions. Docking and Scoring: Successes and Gaps
Leach, A. R., Shoichet, B. K., & Peishoff, C. E. (2006) · Journal of Medicinal Chemistry · doi:10.1021/jm060999m · Cited by 646
PDB-wide collection of binding data: current status of the PDBbind database
Liu, Z., Li, Y., Han, L., Li, J., Liu, J., Zhao, Z., Nie, W., Liu, Y., & Wang, R. (2015) · Bioinformatics · doi:10.1093/bioinformatics/btu626 · Cited by 644
Self-citationJain, A. N. (1996) · Journal of Computer-Aided Molecular Design · doi:10.1007/bf00124474 · Cited by 606
Evaluation of Docking Performance: Comparative Data on Docking Algorithms
Kontoyianni, M., McClellan, L. M., & Sokol, G. S. (2004) · Journal of Medicinal Chemistry · doi:10.1021/jm0302997 · Cited by 600
Comparative Assessment of Scoring Functions on a Diverse Test Set
Cheng, T., Li, X., Li, Y., Liu, Z., & Wang, R. (2009) · Journal of Chemical Information and Modeling · doi:10.1021/ci9000053 · Cited by 513
Self-citationForging the basis for developing protein-ligand interaction scoring functions.
Liu, Z., Su, M., Han, L., Liu, J., Yang, Q., Li, Y., & Wang, R. (2017) · Accounts of Chemical Research · doi:10.1021/acs.accounts.6b00491 · Cited by 493
Self-citationDetailed Analysis of Scoring Functions for Virtual Screening
Stahl, M., & Rarey, M. (2001) · Journal of Medicinal Chemistry · doi:10.1021/jm0003992 · Cited by 489
Böhm, H.-J. (1998) · Journal of Computer-Aided Molecular Design · doi:10.1023/a:1007999920146 · Cited by 431
LigScore: a novel scoring function for predicting binding affinities
Krammer, A., Kirchhoff, P. D., Jiang, X., Venkatachalam, C. M., & Waldman, M. (2005) · Journal of Molecular Graphics and Modelling · doi:10.1016/j.jmgm.2004.11.007 · Cited by 382
Li, Y., Han, L., Liu, Z., & Wang, R. (2014) · Journal of Chemical Information and Modeling · doi:10.1021/ci500081m · Cited by 382
Self-citationFlexible docking using tabu search and an empirical estimate of binding affinity
Baxter, C. A., Murray, C. W., Clark, D. E., Westhead, D. R., & Eldridge, M. D. (1998) · Proteins: Structure, Function, and Genetics · doi:10.1002/(sici)1097-0134(19981115)33:3<367::aid-prot6>3.0.co;2-w · Cited by 375
General and targeted statistical potentials for protein–ligand interactions
Mooij, W. T. M., & Verdonk, M. L. (2005) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.20588 · Cited by 368
Shivakumar, D., Harder, E., Damm, W., Friesner, R. A., & Sherman, W. (2012) · Journal of Chemical Theory and Computation · doi:10.1021/ct300203w · Cited by 364
Velec, H. F. G., Gohlke, H., & Klebe, G. (2005) · Journal of Medicinal Chemistry · doi:10.1021/jm050436v · Cited by 343
Improving scoring-docking-screening powers of protein-ligand scoring functions using random forest
Wang, C., & Zhang, Y. (2017) · Journal of Computational Chemistry · doi:10.1002/jcc.24667 · Cited by 334
A Computer Method for Calculating Kendall's Tau with Ungrouped Data
Knight, W. R. (1966) · Journal of the American Statistical Association · doi:10.1080/01621459.1966.10480879 · Cited by 324
Muegge, I. (2006) · Journal of Medicinal Chemistry · doi:10.1021/jm050038s · Cited by 303
Classification of Current Scoring Functions
Liu, J., & Wang, R. (2015) · Journal of Chemical Information and Modeling · doi:10.1021/ci500731a · Cited by 303
Self-citationAdvising on Research Methods: A Consultant's Companion
Laberge, Y. (2011) · Journal of Applied Statistics · doi:10.1080/02664763.2011.559375 · Cited by 258
Deciphering common failures in molecular docking of ligand-protein complexes
Verkhivker, G. M., Bouzida, D., Gehlhaar, D. K., Rejto, P. A., Arthurs, S., Colson, A. B., Freer, S. T., Larson, V., Luty, B. A., Marrone, T., et al. (2000) · Journal of Computer-Aided Molecular Design · doi:10.1023/a:1008158231558 · Cited by 240
The Experimental Uncertainty of Heterogeneous Public Ki Data
Kramer, C., Kalliokoski, T., Gedeck, P., & Vulpetti, A. (2012) · Journal of Medicinal Chemistry · doi:10.1021/jm300131x · Cited by 233
Pearlman, D. A., & Charifson, P. S. (2001) · Journal of Medicinal Chemistry · doi:10.1021/jm0100279 · Cited by 217
Comparative Assessment of Scoring Functions on an Updated Benchmark: 1. Compilation of the Test Set
Li, Y., Liu, Z., Li, J., Han, L., Liu, J., Zhao, Z., & Wang, R. (2014) · Journal of Chemical Information and Modeling · doi:10.1021/ci500080q · Cited by 215
Self-citationD3R grand challenge 2015: evaluation of protein-ligand pose and affinity predictions.
Gathiaka, S., Liu, S., Chiu, M., Yang, H., Stuckey, J. A., Kang, Y. N., Delproposto, J., Kubish, G., Dunbar, J. B., Carlson, H. A., et al. (2016) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-016-9946-8 · Cited by 207
Comparison of Automated Docking Programs as Virtual Screening Tools
Cummings, M. D., DesJarlais, R. L., Gibbs, A. C., Mohan, V., & Jaeger, E. P. (2005) · Journal of Medicinal Chemistry · doi:10.1021/jm049798d · Cited by 207
Reviews in Computational Chemistry
(2001) · Reviews in Computational Chemistry · doi:10.1002/0471224413 · Cited by 205
Control of the negative IRES trans-acting factor KHSRP by ubiquitination
Kung, Y.-A., Hung, C.-T., Chien, K.-Y., & Shih, S.-R. (2017) · Nucleic Acids Research · doi:10.1093/nar/gkw1042 · Cited by 201
Gaieb, Z., Liu, S., Gathiaka, S., Chiu, M., Yang, H., Shao, C., Feher, V. A., Walters, W. P., Kuhn, B., Rudolph, M. G., et al. (2018) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-017-0088-4 · Cited by 193
Small Molecule Docking and Scoring
Muegge, I., & Rarey, M. (2001) · Reviews in computational chemistry · doi:10.1002/0471224413.ch1 · Cited by 175
The Roles of Water in the Protein Matrix: A Largely Untapped Resource for Drug Discovery
Spyrakis, F., Ahmed, M. H., Bayden, A. S., Cozzini, P., Mozzarelli, A., & Kellogg, G. E. (2017) · Journal of Medicinal Chemistry · doi:10.1021/acs.jmedchem.7b00057 · Cited by 166
Zhou, Z., Felts, A. K., Friesner, R. A., & Levy, R. M. (2007) · Journal of Chemical Information and Modeling · doi:10.1021/ci7000346 · Cited by 165
Verkhivker, G., Appelt, K., Freer, S. T., & Villafranca, J. E. (1995) · Protein Engineering Design and Selection · doi:10.1093/protein/8.7.677 · Cited by 161
Using reverse docking for target identification and its applications for drug discovery
Lee, A., Lee, K., & Kim, D. (2016) · Expert Opinion on Drug Discovery · doi:10.1080/17460441.2016.1190706 · Cited by 151
CSAR Benchmark Exercise of 2010: Combined Evaluation Across All Submitted Scoring Functions
Smith, R. D., Dunbar, J. B., Ung, P. M. U., Esposito, E. X., Yang, C., Wang, S., & Carlson, H. A. (2011) · Journal of Chemical Information and Modeling · doi:10.1021/ci200269q · Cited by 145
Damm-Ganamet, K. L., Smith, R. D., Dunbar, J. B., Stuckey, J. A., & Carlson, H. A. (2013) · Journal of Chemical Information and Modeling · doi:10.1021/ci400025f · Cited by 133
Scoring functions for protein–ligand interactions: a critical perspective
Schulz‐Gasch, T., & Ståhl, M. (2004) · Drug Discovery Today Technologies · doi:10.1016/j.ddtec.2004.08.004 · Cited by 128
CSAR Data Set Release 2012: Ligands, Affinities, Complexes, and Docking Decoys
Dunbar, J. B., Smith, R. D., Damm‐Ganamet, K. L., Ahmed, A., Esposito, E. X., Delproposto, J., Chinnaswamy, K., Kang, Y.-N., Kubish, G. M., Gestwicki, J. E., et al. (2013) · Journal of Chemical Information and Modeling · doi:10.1021/ci4000486 · Cited by 121
Nicholls, A. O. (2014) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-014-9753-z · Cited by 116
Extensive Consensus Docking Evaluation for Ligand Pose Prediction and Virtual Screening Studies
Tuccinardi, T., Poli, G., Romboli, V., Giordano, A., & Martinelli, A. (2014) · Journal of Chemical Information and Modeling · doi:10.1021/ci500424n · Cited by 111
Allen, W. J., & Rizzo, R. C. (2014) · Journal of Chemical Information and Modeling · doi:10.1021/ci400534h · Cited by 107
CSAR 2014: A Benchmark Exercise Using Unpublished Data from Pharma
Carlson, H. A., Smith, R. D., Damm‐Ganamet, K. L., Stuckey, J. A., Ahmed, A., Convery, M. A., Somers, D. O., Kranz, M., Elkins, P. A., Cui, G., et al. (2016) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.5b00523 · Cited by 105
Li, Y., & Yang, J. (2017) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.7b00049 · Cited by 100
Self-citationTrauma and PTSD symptoms: Does spiritual struggle mediate the link?
Wortmann, J. H., Park, C. L., & Edmondson, D. (2011) · Psychological Trauma: Theory, Research, Practice, and Policy · doi:10.1037/a0021413 · Cited by 93
De novodesign: balancing novelty and confined chemical space
Kutchukian, P. S., & Shakhnovich, E. I. (2010) · Expert Opinion on Drug Discovery · doi:10.1517/17460441.2010.497534 · Cited by 93
Estimating Electron Density Support for Individual Atoms and Molecular Fragments in X-ray Structures
Meyder, A., Nittinger, E., Lange, G., Klein, R., & Rarey, M. (2017) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.7b00391 · Cited by 86
García‐Sosa, A. T. (2013) · Journal of Chemical Information and Modeling · doi:10.1021/ci3005786 · Cited by 81
Xu, W., Lucke, A. J., & Fairlie, D. P. (2015) · Journal of Molecular Graphics and Modelling · doi:10.1016/j.jmgm.2015.01.009 · Cited by 74
Evaluation and application of multiple scoring functions for a virtual screening experiment
Xing, L., Hodgkin, E. E., Liu, Q., & Sedlock, D. M. (2004) · Journal of Computer-Aided Molecular Design · doi:10.1023/b:jcam.0000047812.39758.ab · Cited by 60
Li, H., Peng, J., Leung, Y., Leung, K., Wong, M., Lü, G., & Ballester, P. J. (2018) · Biomolecules · doi:10.3390/biom8010012 · Cited by 55
Cited by
Burley, S. K., Bhikadiya, C., Bi, C., Bittrich, S., Chao, H., Chen, L., Craig, P. A., Crichlow, G. V., Dalenberg, K., Duarte, J. M., et al. (2023) · Nucleic Acids Research · doi:10.1093/nar/gkac1077 · Cited by 887
GNINA 1.0: molecular docking with deep learning
McNutt, A. T., Francoeur, P., Aggarwal, R., Masuda, T., Meli, R., Ragoza, M., Sunseri, J., & Koes, D. R. (2021) · Journal of Cheminformatics · doi:10.1186/s13321-021-00522-2 · Cited by 861
The Art and Science of Molecular Docking
Paggi, J. M., Pandit, A., & Dror, R. O. (2024) · Annual Review of Biochemistry · doi:10.1146/annurev-biochem-030222-120000 · Cited by 459
Machine-learning methods for ligand–protein molecular docking
Crampon, K., Giorkallos, A., Deldossi, M., Baud, S., & Steffenel, L. A. (2022) · Drug Discovery Today · doi:10.1016/j.drudis.2021.09.007 · Cited by 388
Uni-Mol: A Universal 3D Molecular Representation Learning Framework
Zhou, G., Gao, Z., Ding, Q., Zheng, H., Xu, H., Wei, Z., Zhang, L., & Ke, G. (2023) · ChemRxiv · doi:10.26434/chemrxiv-2022-jjm0j-v4 · Cited by 324
Metadata
- Authors
- Su, M., Yang, Q., Du, Y., Feng, G., Liu, Z., Li, Y., & Wang, R.
- Venue
- Journal of Chemical Information and Modeling
- Publisher
- American Chemical Society (ACS)
- Published
- 2018-11-27
- Volume
- 59
- Issue
- 2
- Pages
- 895-913
- ISSN
- 1549-9596 (ISSN-L)
- 1549-960X
- OpenAlex
- W2902812092
- Cited by
- 824
- References
- 81
- FWCI
- 29.83
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.