D3R grand challenge 2: blind prediction of protein–ligand poses, affinity rankings, and relative binding free energies.
Open accessAbstract
The Drug Design Data Resource (D3R) ran Grand Challenge 2 (GC2) from September 2016 through February 2017. This challenge was based on a dataset of structures and affinities for the nuclear receptor farnesoid X receptor (FXR), contributed by F. Hoffmann-La Roche. The dataset contained 102 IC50 values, spanning six orders of magnitude, and 36 high-resolution co-crystal structures with representatives of four major ligand classes. Strong global participation was evident, with 49 participants submitting 262 prediction submission packages in total. Procedurally, GC2 mimicked Grand Challenge 2015 (GC2015), with a Stage 1 subchallenge testing ligand pose prediction methods and ranking and scoring methods, and a Stage 2 subchallenge testing only ligand ranking and scoring methods after the release of all blinded co-crystal structures. Two smaller curated sets of 18 and 15 ligands were developed to test alchemical free energy methods. This overview summarizes all aspects of GC2, including the dataset details, challenge procedures, and participant results. We also consider implications for progress in the field, while highlighting methodological areas that merit continued development. Similar to GC2015, the outcome of GC2 underscores the pressing need for methods development in pose prediction, particularly for ligand scaffolds not currently represented in the Protein Data Bank ( http://www.pdb.org ), and in affinity ranking and scoring of bound ligands.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Even with crystal structures of 36 complexes at Stage 2 of Grand Challenge 2, the maximum Kendall's tau achieved was 0.46.
“Even with the crystal structures of 36 complexes at Stage 2, the maximum Kendall’s tau achieved was 0.46, reinforcing the great deal in correctly ranking a set of compounds.”
From Empirical scoring functions for structure-based virtual screening: applications, critical aspects, and challenges (2018) · cites this paper as Gaieb et al., 2018
According to results from Grand Challenge 2, pose prediction is well performed for many methodologies, but scoring remains very challenging even when crystal structures are provided.
“According to the results obtained in the Grand Challenge 2, it is clear that the pose prediction task is well performed for many methodologies, but scoring is still a very challenging task, even when the crystal structures are provided (Gaieb et al., 2018).”
From Empirical scoring functions for structure-based virtual screening: applications, critical aspects, and challenges (2018) · cites this paper as Gaieb et al., 2018
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Trott, O., & Olson, A. J. (2010) · Journal of Computational Chemistry · doi:10.1002/jcc.21334 · Cited by 38,412 · see citing works
[20] Processing of X-ray diffraction data collected in oscillation mode
Otwinowski, Z., & Minor, W. (1997) · Methods in Enzymology · doi:10.1016/s0076-6879(97)76066-x · Cited by 33,475 · see citing works
<i>Phaser</i> crystallographic software
McCoy, A. J., Grosse-Kunstleve, R. W., Adams, P. D., Winn, M. D., Storoni, L. C., & Read, R. J. (2007) · Journal of Applied Crystallography · doi:10.1107/s0021889807021206 · Cited by 20,612 · see citing works
Kabsch, W. (2010) · Acta Crystallographica Section D Biological Crystallography · doi:10.1107/s0907444909047337 · Cited by 16,463 · see citing works
Yung-Chi, C., & Prusoff, W. H. (1973) · Biochemical Pharmacology · doi:10.1016/0006-2952(73)90196-2 · Cited by 12,922 · see citing works
Krissinel, E., & Henrick, K. (2004) · Acta Crystallographica Section D Biological Crystallography · doi:10.1107/s0907444904026460 · Cited by 3,785 · see citing works
Identification of a nuclear receptor for bile acids
Makishima, M., Okamoto, A. Y., Repa, J. J., Tu, H., Learned, R. M., Luk, A., Hull, M. V., Lustig, K. D., Mangelsdorf, D. J., & Shan, B. (1999) · Science · doi:10.1126/science.284.5418.1362 · Cited by 2,714 · see citing works
A hierarchical approach to all‐atom protein loop prediction
Jacobson, M. P., Pincus, D. L., Rapp, C. S., Day, T., Honig, B. H., Shaw, D. E., & Friesner, R. A. (2004) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.10613 · Cited by 2,541 · see citing works
Bile Acids: Natural Ligands for an Orphan Nuclear Receptor
Parks, D. J., Blanchard, S. G., Bledsoe, R. K., Chandra, G., Consler, T. G., Kliewer, S. A., Stimmel, J. B., Willson, T. M., Zavacki, A. M., Moore, D. D., et al. (1999) · Science · doi:10.1126/science.284.5418.1365 · Cited by 2,274 · see citing works
Computational Methods in Drug Discovery
Sliwoski, G. R., Kothiwale, S., Meiler, J., & Lowe, E. W. (2013) · Pharmacological Reviews · doi:10.1124/pr.112.007336 · Cited by 2,226 · see citing works
Endogenous Bile Acids Are Ligands for the Nuclear Receptor FXR/BAR
Wang, H., Chen, J., Hollister, K., Sowers, L. C., & Forman, B. M. (1999) · Molecular Cell · doi:10.1016/s1097-2765(00)80348-2 · Cited by 1,636 · see citing works
The Many Roles of Computation in Drug Discovery
Jorgensen, W. L. (2004) · Science · doi:10.1126/science.1096361 · Cited by 1,560 · see citing works
Molecular Basis for Feedback Regulation of Bile Acid Synthesis by Nuclear Receptors
Lu, T. T., Makishima, M., Repa, J. J., Schoonjans, K., Kerr, T. A., Auwerx, J., & Mangelsdorf, D. J. (2000) · Molecular Cell · doi:10.1016/s1097-2765(00)00050-2 · Cited by 1,487 · see citing works
On the Role of the Crystal Environment in Determining Protein Side-chain Conformations
Jacobson, M. P., Friesner, R. A., Xiang, Z., & Honig, B. H. (2002) · Journal of Molecular Biology · doi:10.1016/s0022-2836(02)00470-9 · Cited by 1,289 · see citing works
Viswanadhan, V. N., Ghose, A. K., Revankar, G. R., & Robins, R. K. (1989) · Journal of Chemical Information and Computer Sciences · doi:10.1021/ci00063a006 · Cited by 1,135 · see citing works
Macromolecular TLS Refinement in REFMAC at Moderate Resolutions
Winn, M., Murshudov, G. N., & Papiz, M. Z. (2003) · Methods in enzymology on CD-ROM/Methods in enzymology · doi:10.1016/s0076-6879(03)74014-2 · Cited by 804 · see citing works
Refinement of severely incomplete structures with maximum likelihood inBUSTER–TNT
Blanc, E., Roversi, P., Vonrhein, C., Flensburg, C., Lea, S. M., & Bricogne, G. (2004) · Acta Crystallographica Section D Biological Crystallography · doi:10.1107/s0907444904016427 · Cited by 719 · see citing works
Tembre, B. L., & McCammon, J. A. (1984) · Computers & Chemistry · doi:10.1016/0097-8485(84)85020-2 · Cited by 459 · see citing works
Duan, J., Dixon, S. L., Lowrie, J. F., & Sherman, W. (2010) · Journal of Molecular Graphics and Modelling · doi:10.1016/j.jmgm.2010.05.008 · Cited by 445 · see citing works
Accurate calculation of the absolute free energy of binding for drug molecules
Aldeghi, M., Heifetz, A., Bodkin, M. J., Knapp, S., & Biggin, P. C. (2016) · Chemical Science · doi:10.1039/c5sc02678d · Cited by 347 · see citing works
Sastry, M., Lowrie, J. F., Dixon, S. L., et al. (2010) · Journal of Chemical Information and Modeling · doi:10.1021/ci100062n · Cited by 333 · see citing works
An improved relaxed complex scheme for receptor flexibility in computer-aided drug design
Amaro, R. E., Baron, R., & McCammon, J. A. (2008) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-007-9159-2 · Cited by 321 · see citing works
Self-citationD3R grand challenge 2015: evaluation of protein-ligand pose and affinity predictions.
Gathiaka, S., Liu, S., Chiu, M., Yang, H., Stuckey, J. A., Kang, Y. N., Delproposto, J., Kubish, G., Dunbar, J. B., Carlson, H. A., et al. (2016) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-016-9946-8 · Cited by 207 · see citing works
Self-citationThe SAMPL4 host–guest blind prediction challenge: an overview
Muddana, H. S., Fenley, A. T., Mobley, D. L., et al. (2014) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-014-9735-1 · Cited by 183 · see citing works
Predictions of Ligand Selectivity from Absolute Binding Free Energy Calculations
Aldeghi, M., Heifetz, A., Bodkin, M. J., Knapp, S., & Biggin, P. C. (2017) · Journal of the American Chemical Society · doi:10.1021/jacs.6b11467 · Cited by 177 · see citing works
Overview of the SAMPL5 host–guest challenge: Are we doing better?
Yin, J., Henriksen, N. M., Slochower, D. R., Shirts, M. R., Chiu, M. W., Mobley, D. L., & Gilson, M. K. (2016) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-016-9974-4 · Cited by 173 · see citing works
Self-citationFrontiers in free‐energy calculations of biological systems
Chipot, C. J. (2013) · Wiley Interdisciplinary Reviews Computational Molecular Science · doi:10.1002/wcms.1157 · Cited by 170 · see citing works
Predicting Binding Affinities for GPCR Ligands Using Free-Energy Perturbation
Lenselink, E. B., Louvel, J., Forti, A. F., van Veldhoven, J. P. D., de Vries, H., Mulder‐Krieger, T., McRobb, F. M., Negri, A., Goose, J. E., Abel, R., et al. (2016) · ACS Omega · doi:10.1021/acsomega.6b00086 · Cited by 145 · see citing works
CSAR Benchmark Exercise of 2010: Combined Evaluation Across All Submitted Scoring Functions
Smith, R. D., Dunbar, J. B., Ung, P. M. U., Esposito, E. X., Yang, C., Wang, S., & Carlson, H. A. (2011) · Journal of Chemical Information and Modeling · doi:10.1021/ci200269q · Cited by 142 · see citing works
Damm-Ganamet, K. L., Smith, R. D., Dunbar, J. B., Stuckey, J. A., & Carlson, H. A. (2013) · Journal of Chemical Information and Modeling · doi:10.1021/ci400025f · Cited by 133 · see citing works
Prospective Evaluation of Free Energy Calculations for the Prioritization of Cathepsin L Inhibitors
Kuhn, B., Tichý, M., Wang, L., Robinson, S. D., Martin, R. E., Kuglstatter, A., Benz, J., Giroud, M., Schirmeister, T., Abel, R., et al. (2017) · Journal of Medicinal Chemistry · doi:10.1021/acs.jmedchem.6b01881 · Cited by 131 · see citing works
Self-citationNichols, J. S., Parks, D. J., Consler, T. G., & Blanchard, S. G. (1998) · Analytical Biochemistry · doi:10.1006/abio.1997.2557 · Cited by 110 · see citing works
Blind prediction of cyclohexane–water distribution coefficients from the SAMPL5 challenge
Bannan, C. C., Burley, K. H., Chiu, M. W., Shirts, M. R., Gilson, M. K., & Mobley, D. L. (2016) · Journal of Computer-Aided Molecular Design · doi:10.1007/s10822-016-9954-8 · Cited by 107 · see citing works
Self-citationCSAR 2014: A Benchmark Exercise Using Unpublished Data from Pharma
Carlson, H. A., Smith, R. D., Damm‐Ganamet, K. L., Stuckey, J. A., Ahmed, A., Convery, M. A., Somers, D. O., Kranz, M., Elkins, P. A., Cui, G., et al. (2016) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.5b00523 · Cited by 102 · see citing works
Richter, H. G., Benson, G. M., Bleicher, K., Blum, D., Chaput, E., Clemann, N., Feng, S., Gardès, C., Grether, U., HARTMAN, P. G., et al. (2011) · Bioorganic & Medicinal Chemistry Letters · doi:10.1016/j.bmcl.2010.12.123 · Cited by 79 · see citing works
Self-citationKeränen, H., Pérez‐Benito, L., Ciordia, M., et al. (2017) · Journal of Chemical Theory and Computation · doi:10.1021/acs.jctc.6b01141 · Cited by 76 · see citing works
Abel, R., Mondal, S., Masse, C. E., Greenwood, J. R., Harriman, G. C. B., Ashwell, M. A., Bhat, S., Wester, R. T., Frye, L. L., Kapeller, R., et al. (2016) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2016.10.007 · Cited by 68 · see citing works
Richter, H. G., Benson, G. M., Blum, D., Chaput, E., Feng, S., Gardès, C., Grether, U., HARTMAN, P. G., Kuhn, B., Martin, R. E., et al. (2010) · Bioorganic & Medicinal Chemistry Letters · doi:10.1016/j.bmcl.2010.11.039 · Cited by 60 · see citing works
Self-citationIdentification of an N-oxide pyridine GW4064 analog as a potent FXR agonist
Feng, S., Yang, M., Zhang, Z., Wang, Z., Hong, D., Richter, H. G., Benson, G. M., Bleicher, K., Grether, U., Martin, R. E., et al. (2009) · Bioorganic & Medicinal Chemistry Letters · doi:10.1016/j.bmcl.2009.03.008 · Cited by 55 · see citing works
Self-citationSmith, R. D., Damm-Ganamet, K., Dunbar, J., Ahmed, A., Chinnaswamy, K., Delproposto, J., Kubish, G., Tinberg, C., Khare, S., Dou, J., et al. (2015) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.5b00387 · Cited by 54 · see citing works
Mishra, S. K., Calabrò, G., Loeffler, H. H., et al. (2015) · Journal of Chemical Theory and Computation · doi:10.1021/acs.jctc.5b00159 · Cited by 33 · see citing works
Lessons Learned over Four Benchmark Exercises from the Community Structure–Activity Resource
Carlson, H. A. (2016) · Journal of Chemical Information and Modeling · doi:10.1021/acs.jcim.6b00182 · Cited by 32 · see citing works
The Collaborative Drug Discovery (CDD) Database
Ekins, S., & Bunin, B. A. (2013) · Methods in molecular biology · doi:10.1007/978-1-62703-342-8_10 · Cited by 30 · see citing works
A general method for hyperquenching protein crystals
Warkentin, M. A., & Thorne, R. (2007) · Journal of Structural and Functional Genomics · doi:10.1007/s10969-007-9029-0 · Cited by 17 · see citing works
Gardès, C., Blum, D., Bleicher, K., Chaput, E., Ebeling, M., HARTMAN, P. G., Handschin, C., Richter, H. G., & Benson, G. M. (2011) · Journal of Lipid Research · doi:10.1194/jlr.m012542 · Cited by 14 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Protein Data Bank: the single global archive for 3D macromolecular structure data
wwPDB consortium, Burley, S. K., Berman, H. M., Bhikadiya, C., Bi, C., Chen, L., Costanzo, L. D., Christie, C., Duarte, J. M., Dutta, S., et al. (2019) · Nucleic Acids Research · doi:10.1093/nar/gky949 · Cited by 1,163 · see citing works
Self-citationGuedes, I. A., Pereira, F. S. S., & Dardenne, L. E. (2018) · Frontiers in Pharmacology · doi:10.3389/fphar.2018.01089 · Cited by 331 · see citing works
Ackloo, S., Al-awar, R., Amaro, R. E., Arrowsmith, C. H., Azevedo, H., Batey, R. A., Bengio, Y., Betz, U. A. K., Bologa, C. G., Chodera, J. D., et al. (2022) · Nature Reviews Chemistry · doi:10.1038/s41570-022-00363-z · Cited by 112 · see citing works
Self-citationNew prediction categories in <scp>CASP15</scp>
Kryshtafovych, A., Antczak, M., Szachniuk, M., Zok, T., Kretsch, R. C., Rangan, R., Pham, P., Das, R., Robin, X., Studer, G., et al. (2023) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.26515 · Cited by 54 · see citing works
A membrane protein complex docking benchmark
Koukos, P. I., Faro, I., van Noort, C. W., & Bonvin, A. M. J. J. (2018) · Journal of Molecular Biology · doi:10.1016/j.jmb.2018.11.005 · Cited by 23 · see citing works
A fully automated benchmarking suite to compare macromolecular complexes
Studer, G., Robin, X., Bienert, S., Durairaj, J., Škrinjar, P., Tauriello, G., Waterhouse, A. M., & Schwede, T. (2026) · Nature Methods · doi:10.1038/s41592-025-02973-z · Cited by 10 · see citing works
“Historically, CASP uses GDT_TS (total score), which averages GDT scores with distance thresholds 1, 2, 4 and 8 Å .”
- Authors
- Gaieb, Z., Liu, S., Gathiaka, S., Chiu, M., Yang, H., Shao, C., Feher, V. A., Walters, W. P., Kuhn, B., Rudolph, M. G., Burley, S. K., Gilson, M. K., & Amaro, R. E.
- Affiliations
- University of California San Diego (US)
- Merck & Co., Inc., Rahway, NJ, USA (United States) (US)
- Rutgers, The State University of New Jersey (US)
- Relay Therapeutics (United States) (US)
- F. Hoffmann-La Roche AG (CH)
- Corresponding author
- Gilson, Michael K.
- ORCID
- Venue
- Journal of Computer-Aided Molecular Design
- Published
- 2017-12-04
- Volume
- 32
- Issue
- 1
- Pages
- 1-20
- ISSN
- 0920-654X (ISSN-L)
- 1573-4951
- PMC
- PMC5767524
- OpenAlex
- W2774738748
- Cited by
- 193
- References
- 45
- Citations by year
- 2026: 14 · 2025: 11 · 2024: 12 · 2023: 18 · 2022: 12 · 2021: 28 · 2020: 28 · 2019: 45 · 2018: 24 · 2017: 1
- FWCI
- 20.11
- Type
- journal article
- Language
- en
- Open access
- green
- europepmc.org
- Locations
- Journal of Computer-Aided Molecular Design (publishedVersion)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (acceptedVersion)
- PubMed Central (acceptedVersion)
- Rutgers University Community Repository (Rutgers University) (submittedVersion)
- DORA PSI (Paul Scherrer Institute) (submittedVersion)
- Funders
- National Institute of General Medical Sciences (U01 GM111528)
- Topics
- Physical Sciences › Computer Science › Computational Theory and Mathematics › Computational Drug Discovery Methods
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › vaccines and immunoinformatics approaches
- Keywords
- farnesoid X receptor, ligand pose prediction, relative binding free energy, alchemical free energy methods, nuclear receptor ligands, co-crystal structure, blind prediction, scoring functions
- MeSH terms
- Receptor, Farnesoid X-Activated, Humans, Ligands, Protein Binding, Software, Thermodynamics, Drug Design, Computer-Aided Design, Receptors, Cytoplasmic and Nuclear, Inhibitory Concentration 50, Databases, Protein, Molecular Docking Simulation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
