Report on the sixth blind test of organic crystal structure prediction methods
Open accessThe DOI matches its Crossref record; 75 of the 75 references it cites were found in a registry or an index.
Abstract
The sixth blind test of organic crystal structure prediction (CSP) methods has been held, with five target systems: a small nearly rigid molecule, a polymorphic former drug candidate, a chloride salt hydrate, a co-crystal and a bulky flexible molecule. This blind test has seen substantial growth in the number of participants, with the broad range of prediction methods giving a unique insight into the state of the art in the field. Significant progress has been seen in treating flexible molecules, usage of hierarchical approaches to ranking structures, the application of density-functional approximations, and the establishment of new workflows and `best practices' for performing CSP calculations. All of the targets, apart from a single potentially disordered Z′ = 2 polymorph of the drug candidate, were predicted by at least one submission. Despite many remaining challenges, it is clear that CSP methods are becoming more applicable to a wider range of real systems, including salts, hydrates and larger flexible molecules. The results also highlight the potential for CSP calculations to complement and augment experimental studies of organic solid forms.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Generalized Gradient Approximation Made Simple
Perdew, J. P., Burke, K., & Ernzerhof, M. (1996) · Physical Review Letters · doi:10.1103/physrevlett.77.3865 · Cited by 215,824
Lee, C., Yang, W., & Parr, R. G. (1988) · Physical Review B · doi:10.1103/physrevb.37.785 · Cited by 100,874
Grimme, S., Antony, J., Ehrlich, S., & Krieg, H. (2010) · The Journal of Chemical Physics · doi:10.1063/1.3382344 · Cited by 59,686
Self-citationDensity-functional exchange-energy approximation with correct asymptotic behavior
Becke, A. D. (1988) · Physical Review A · doi:10.1103/physreva.38.3098 · Cited by 53,711
Semiempirical GGA‐type density functional constructed with a long‐range dispersion correction
Grimme, S. (2006) · Journal of Computational Chemistry · doi:10.1002/jcc.20495 · Cited by 31,038
Self-citationToward reliable density functional methods without adjustable parameters: The PBE0 model
Adamo, C., & Barone, V. (1999) · The Journal of Chemical Physics · doi:10.1063/1.478522 · Cited by 19,042
Structure validation in chemical crystallography
Spek, A. L. (2009) · Acta Crystallographica Section D Biological Crystallography · doi:10.1107/s090744490804362x · Cited by 15,432
Dynamical Theory of Crystal Lattices
Born, M., Huang, K., & Lax, M. (1955) · American Journal of Physics · doi:10.1119/1.1934059 · Cited by 10,501
The Cambridge Structural Database
Groom, C. R., Bruno, I. J., Lightfoot, M. P., & Ward, S. C. (2016) · Acta Crystallographica Section B Structural Science, Crystal Engineering and Materials · doi:10.1107/s2052520616003954 · Cited by 10,297 · cited 2×
“The predicted crystal structures submitted by participants were compared with the experimentally known crystal structures using the Crystal Packing Similarity Tool (Chisholm & Motherwell, 2005), as available through the CSD Python API (Groom et al., 2016) and Mercury 3.6 (Macrae et al., 2008).”
MethodInfluentialSelf-citationMercury CSD 2.0– new features for the visualization and investigation of crystal structures
Macrae, C. F., Bruno, I. J., Chisholm, J. A., Edgington, P. R., McCabe, P., Pidcock, E., Rodriguez-Monge, L., Taylor, R., van de Streek, J., & Wood, P. A. (2008) · Journal of Applied Crystallography · doi:10.1107/s0021889807067908 · Cited by 8,863
Self-citationTao, J., Perdew, J. P., Staroverov, V. N., & Scuseria, G. E. (2003) · Physical Review Letters · doi:10.1103/physrevlett.91.146401 · Cited by 7,093
Tkatchenko, A., & Scheffler, M. (2009) · Physical Review Letters · doi:10.1103/physrevlett.102.073005 · Cited by 6,286 · cited 1×
“These include the Neumann–Perrin (Neumann & Perrin, 2005), D2 (Grimme, 2006), TS (Tkatchenko & Scheffler, 2009), XDM (Becke & Johnson, 2007), D3 (Grimme et al., 2010) and MBD (Tkatchenko et al., 2012; Ambrosetti et al., 2014) methods, as well as two vdW density functionals, vdW-DF (Dion et al.,…”
Self-citationVan der Waals Density Functional for General Geometries
Dion, M., Rydberg, H., Schröder, E., Langreth, D. C., & Lundqvist, B. I. (2004) · Physical Review Letters · doi:10.1103/physrevlett.92.246401 · Cited by 5,622
Van der Waals density functionals applied to solids
Klimeš, J., Bowler, D. R., & Michaelides, A. (2011) · Physical Review B · doi:10.1103/physrevb.83.195131 · Cited by 4,624
Python Materials Genomics (pymatgen): A robust, open-source python library for materials analysis
Ong, S. P., Richards, W. D., Jain, A., Hautier, G., Kocher, M., Cholia, S., Gunter, D., Chevrier, V. L., Persson, K. A., & Ceder, G. (2013) · Computational Materials Science · doi:10.1016/j.commatsci.2012.10.028 · Cited by 4,534
On the distribution of points in a cube and the approximate evaluation of integrals
Sobol', I. M. (1967) · USSR Computational Mathematics and Mathematical Physics · doi:10.1016/0041-5553(67)90144-9 · Cited by 2,975 · cited 1×
“As in the previous blind test, the majority of methods employ some variation on random or quasi-random searches to generate trial crystal structures (Submissions 3, 5–7, 10, 11, 15, 16 and 18–20), with four submissions (3, 15, 18, 19) using lowdiscrepancy Sobol’ sequences (Sobol’, 1967).”
MethodCALYPSO: A method for crystal structure prediction
Wang, Y., Lv, J., Zhu, L., & Ma, Y. (2012) · Computer Physics Communications · doi:10.1016/j.cpc.2012.05.008 · Cited by 2,897
Self-citationGraph-set analysis of hydrogen-bond patterns in organic crystals
Etter, M. C., MacDonald, J. C., & Bernstein, J. (1990) · Acta Crystallographica Section B Structural Science · doi:10.1107/s0108768189012929 · Cited by 2,558
Challenges for Density Functional Theory
Cohen, A. J., Mori-Sánchez, P., & Yang, W. (2012) · Chemical Reviews · doi:10.1021/cr200107z · Cited by 2,541
Insights into Current Limitations of Density Functional Theory
Cohen, A. J., Mori-Sánchez, P., & Yang, W. (2008) · Science · doi:10.1126/science.1158722 · Cited by 2,527 · cited 1×
“Standard densityfunctional approximations can also struggle to deal with charged systems and charge transfer adequately due to selfinteraction errors (Cohen et al., 2008, 2012), but in the case of (XXIV), DFT provides a good basis for fitting a bespoke potential and ranking the predicted structures.”
BackgroundTheory of Intermolecular Forces
(1969) · Elsevier eBooks · doi:10.1016/c2013-0-02436-x · Cited by 1,680
Interaction of the van der Waals Type Between Three Atoms
Axilrod, B. M., & Teller, E. (1943) · The Journal of Chemical Physics · doi:10.1063/1.1723844 · Cited by 1,672
Accurate and Efficient Method for Many-Body van der Waals Interactions
Tkatchenko, A., DiStasio, R. A., Car, R., & Scheffler, M. (2012) · Physical Review Letters · doi:10.1103/physrevlett.108.236402 · Cited by 1,528
Self-citationKlimeš, J., & Michaelides, A. (2012) · The Journal of Chemical Physics · doi:10.1063/1.4754130 · Cited by 1,125
Retrieval of Crystallographically-Derived Molecular Geometry Information
Bruno, I. J., Cole, J. C., Kessler, M., Luo, J., Motherwell, W. D. S., Purkis, L. H., Smith, B. R., Taylor, R., Cooper, R. I., Harris, S. E., et al. (2004) · Journal of Chemical Information and Computer Sciences · doi:10.1021/ci049780b · Cited by 983
Self-citationDFT-D3 Study of Some Molecular Crystals
Moellmann, J., & Grimme, S. (2014) · The Journal of Physical Chemistry C · doi:10.1021/jp501237c · Cited by 925
Self-citationIntroduction to Lattice Dynamics
Dove, M. T. (1993) · Cambridge University Press eBooks · doi:10.1017/cbo9780511619885 · Cited by 827
Cabeza, A. J. C., & Bernstein, J. (2013) · Chemical Reviews · doi:10.1021/cr400249d · Cited by 826
The Theory of Intermolecular Forces
Stone, A. (2013) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780199672394.001.0001 · Cited by 817
Long-range correlation energy calculated from coupled atomic response functions
Ambrosetti, A., Reilly, A. M., DiStasio, R. A., & Tkatchenko, A. (2014) · The Journal of Chemical Physics · doi:10.1063/1.4865104 · Cited by 661
Self-citationDistributed Multipole Analysis: Stability for Large Basis Sets
Stone, A. J. (2005) · Journal of Chemical Theory and Computation · doi:10.1021/ct050190+ · Cited by 606
Long-range correlation energy calculated from coupled atomic response functions
Ambrosetti, A., Reilly, A., DiStasio, R., & Tkatchenko, A. (2014) · Max Planck Digital Library · Cited by 580
Self-citationExchange-hole dipole moment and the dispersion interaction revisited
Becke, A. D., & Johnson, E. R. (2007) · The Journal of Chemical Physics · doi:10.1063/1.2795701 · Cited by 546
Corner flapping: a simple and fast algorithm for exhaustive generation of ring conformations
Goto, H., & Osawa, E. (1989) · Journal of the American Chemical Society · doi:10.1021/ja00206a046 · Cited by 472
Self-citationStatic and lattice vibrational energy differences between polymorphs
Nyman, J., & Day, G. M. (2015) · CrystEngComm · doi:10.1039/c5ce00045a · Cited by 438
Self-citationDay, G. M., Cooper, T. G., Cruz-Cabeza, A. J., Hejczyk, K. E., Ammon, H. L., Boerrigter, S. X. M., Tan, J. S., Della Valle, R. G., Venuti, E., Jose, J., et al. (2009) · Acta Crystallographica Section B Structural Science · doi:10.1107/s0108768109004066 · Cited by 436
Self-citationStructural and Thermodynamic Explanations of Ostwald's Rule
Threlfall, T. L. (2003) · Organic Process Research & Development · doi:10.1021/op030026l · Cited by 435
A third blind test of crystal structure prediction
Day, G. M., Motherwell, W. D. S., Ammon, H. L., Boerrigter, S. X. M., Della Valle, R. G., Venuti, E., Dzyabchenko, A., Dunitz, J. D., Schweizer, B., van Eijck, B. P., et al. (2005) · Acta Crystallographica Section B Structural Science · doi:10.1107/s0108768105016563 · Cited by 430
Self-citationRisthaus, T., & Grimme, S. (2013) · Journal of Chemical Theory and Computation · doi:10.1021/ct301081n · Cited by 423
Self-citationTowards crystal structure prediction of complex organic compounds – a report on the fifth blind test
Bardwell, D. A., Adjiman, C. S., Arnautova, Y. A., Bartashevich, E., Boerrigter, S. X. M., Braun, D. E., Cruz-Cabeza, A. J., Day, G. M., Della Valle, R. G., Desiraju, G. R., et al. (2011) · Acta Crystallographica Section B Structural Science · doi:10.1107/s0108768111042868 · Cited by 422
Self-citationAn efficient algorithm for searching low-energy conformers of cyclic and acyclic molecules
Goto, M., & O̅sawa, E. (1993) · Journal of the Chemical Society Perkin Transactions 2 · doi:10.1039/p29930000187 · Cited by 413
COMPACK: a program for identifying crystal structure similarity using distances
Chisholm, J. A., & Motherwell, S. (2005) · Journal of Applied Crystallography · doi:10.1107/s0021889804027074 · Cited by 374
Price, S. (. L. (2009) · Accounts of Chemical Research · doi:10.1021/ar800147t · Cited by 360
Self-citationA Major Advance in Crystal Structure Prediction
Neumann, M. A., Leusen, F. J. J., & Kendrick, J. (2008) · Angewandte Chemie International Edition · doi:10.1002/anie.200704247 · Cited by 349
Self-citationReilly, A. M., & Tkatchenko, A. (2013) · The Journal of Chemical Physics · doi:10.1063/1.4812819 · Cited by 331
Self-citationPrice, S. L., Leslie, M., Welch, G. W. A., Habgood, M., Price, L. S., Karamertzanis, P. G., & Day, G. M. (2010) · Physical Chemistry Chemical Physics · doi:10.1039/c004164e · Cited by 312
Self-citationAb initio determination of the crystalline benzene lattice energy to sub-kilojoule/mole accuracy
Yang, J., Hu, W., Usvyat, D., Matthews, D., Schütz, M., & Chan, G. K.-L. (2014) · Science · doi:10.1126/science.1254419 · Cited by 274
Self-citationUncovering Molecular Processes in Crystal Nucleation and Growth by Using Molecular Simulation
Anwar, J., & Zahn, D. (2011) · Angewandte Chemie International Edition · doi:10.1002/anie.201000463 · Cited by 273
Van der Waals interactions in solids using the exchange-hole dipole moment model
Otero-de-la-Roza, A., & Johnson, E. R. (2012) · The Journal of Chemical Physics · doi:10.1063/1.4705760 · Cited by 253
Why don't we find more polymorphs?
Price, S. L. (2013) · Acta Crystallographica Section B Structural Science, Crystal Engineering and Materials · doi:10.1107/s2052519213018861 · Cited by 231
Self-citationvan der Waals dispersion interactions in molecular materials: beyond pairwise additivity
Reilly, A. M., & Tkatchenko, A. (2015) · Chemical Science · doi:10.1039/c5sc00410a · Cited by 220
Self-citationSimulating micrometre-scale crystal growth from solution
Piana, S., Reyhani, M., & Gale, J. D. (2005) · Nature · doi:10.1038/nature04173 · Cited by 217
How to quantify energy landscapes of solids
Oganov, A. R., & Valle, M. (2009) · The Journal of Chemical Physics · doi:10.1063/1.3079326 · Cited by 211
Self-citationBaias, M., Dumez, J.-N., Svensson, P. H., Schantz, S., Day, G. M., & Emsley, L. (2013) · Journal of the American Chemical Society · doi:10.1021/ja4088874 · Cited by 209
Self-citationUncovering Molecular Details of Urea Crystal Growth in the Presence of Additives
Salvalaglio, M., Vetter, T., Giberti, F., Mazzotti, M., & Parrinello, M. (2012) · Journal of the American Chemical Society · doi:10.1021/ja307408x · Cited by 204
Thompson, H. P. G., & Day, G. M. (2014) · Chem. Sci. · doi:10.1039/c4sc01132e · Cited by 192
Self-citationKronik, L., & Tkatchenko, A. (2014) · Accounts of Chemical Research · doi:10.1021/ar500144s · Cited by 191
Self-citationReilly, A. M., & Tkatchenko, A. (2014) · Physical Review Letters · doi:10.1103/physrevlett.113.055701 · Cited by 190
Self-citationMany‐Body Dispersion Interactions in Molecular Crystal Polymorphism
Marom, N., DiStasio, R. A., Atalla, V., Levchenko, S., Reilly, A. M., Chelikowsky, J. R., Leiserowitz, L., & Tkatchenko, A. (2013) · Angewandte Chemie International Edition · doi:10.1002/anie.201301938 · Cited by 180
Self-citationGrimme, S. (2014) · Journal of Chemical Theory and Computation · doi:10.1021/ct500573f · Cited by 180
Self-citationZhu, Q., Oganov, A. R., Glass, C. W., & Stokes, H. T. (2012) · Acta Crystallographica Section B Structural Science · doi:10.1107/s0108768112017466 · Cited by 175
Self-citationKnowledge-based model of hydrogen-bonding propensity in organic crystals
Galek, P. T. A., Fábián, L., Motherwell, W. D. S., Allen, F. H., & Feeder, N. (2007) · Acta Crystallographica Section B Structural Science · doi:10.1107/s0108768107030996 · Cited by 170
Tailor-Made Force Fields for Crystal-Structure Prediction
Neumann, M. A. (2008) · The Journal of Physical Chemistry B · doi:10.1021/jp710575h · Cited by 160
Self-citationBen, M. D., Hutter, J., & VandeVondele, J. (2012) · Journal of Chemical Theory and Computation · doi:10.1021/ct300531w · Cited by 150
Wen, S., & Beran, G. J. O. (2011) · Journal of Chemical Theory and Computation · doi:10.1021/ct200541h · Cited by 140
Exploring the Experimental and Computed Crystal Energy Landscape of Olanzapine
Bhardwaj, R. M., Price, L. S., Price, S. L., Reutzel‐Edens, S. M., Miller, G. J., Oswald, I. D. H., Johnston, B. F., & Florence, A. J. (2013) · Crystal Growth & Design · doi:10.1021/cg301826s · Cited by 137
Self-citationConvergence Properties of Crystal Structure Prediction by Quasi-Random Sampling
Case, D. H., Campbell, J. E., Bygrave, P. J., & Day, G. M. (2015) · Journal of Chemical Theory and Computation · doi:10.1021/acs.jctc.5b01112 · Cited by 135
Self-citationBrandenburg, J. G., & Grimme, S. (2013) · Topics in current chemistry · doi:10.1007/128_2013_488 · Cited by 124
Self-citationEfficient Handling of Molecular Flexibility in Lattice Energy Minimization of Organic Crystals
Kazantsev, A. V., Karamertzanis, P. G., Adjiman, C. S., & Pantelides, C. C. (2011) · Journal of Chemical Theory and Computation · doi:10.1021/ct100597e · Cited by 123
Self-citationTemperature-Accelerated Method for Exploring Polymorphism in Molecular Crystals Based on Free Energy
Yu, T.-Q., & Tuckerman, M. E. (2011) · Physical Review Letters · doi:10.1103/physrevlett.107.015701 · Cited by 116
Self-citationEfficient Handling of Molecular Flexibility in Ab Initio Generation of Crystal Structures
Habgood, M., Sugden, I. J., Kazantsev, A. V., Adjiman, C. S., & Pantelides, C. C. (2015) · Journal of Chemical Theory and Computation · doi:10.1021/ct500621v · Cited by 77
Self-citationvan Eijck, B. P., Mooij, W. T. M., & Kroon, J. (2001) · Journal of Computational Chemistry · doi:10.1002/jcc.1046 · Cited by 54
Simons, L. J., Caprathe, B. W., Callahan, M. J., Graham, J. M., Kimura, T., Lai, Y., LeVine, H., Lipinski, W. J., Sakkab-Tan, A. T., Tasaki, Y., et al. (2008) · Bioorganic & Medicinal Chemistry Letters · doi:10.1016/j.bmcl.2008.12.049 · Cited by 46
Eddleston, M. D., Hejczyk, K. E., Bithell, E. G., et al. (2013) · Chemistry - A European Journal · doi:10.1002/chem.201204368 · Cited by 37
Sixth blind test of organic crystal-structure prediction methods
Groom, C. R., & Reilly, A. M. (2014) · Acta Crystallographica Section B Structural Science Crystal Engineering and Materials · doi:10.1107/s2052520614015923 · Cited by 15
Self-citation
Cited by
Recent advances and applications of machine learning in solid-state materials science
Schmidt, J., Marques, M. R. G., Botti, S., & Marques, M. A. L. (2019) · npj Computational Materials · doi:10.1038/s41524-019-0221-0 · Cited by 2,607
A generally applicable atomic-charge dependent London dispersion correction
Caldeweyher, E., Ehlert, S., Hansen, A., Neugebauer, H., Spicher, S., Bannwarth, C., & Grimme, S. (2019) · The Journal of Chemical Physics · doi:10.1063/1.5090222 · Cited by 1,934
Self-citationGaussian Process Regression for Materials and Molecules
Deringer, V. L., Bartók, A. P., Bernstein, N., Wilkins, D. M., Ceriotti, M., & Csányi, G. (2021) · Chemical Reviews · doi:10.1021/acs.chemrev.1c00022 · Cited by 1,221
B97-3c: A revised low-cost variant of the B97-D density functional method
Brandenburg, J. G., Bannwarth, C., Hansen, A., & Grimme, S. (2018) · The Journal of Chemical Physics · doi:10.1063/1.5012601 · Cited by 760
Self-citationStructure prediction drives materials discovery
Oganov, A. R., Pickard, C. J., Zhu, Q., & Needs, R. J. (2019) · Nature Reviews Materials · doi:10.1038/s41578-019-0101-8 · Cited by 717
Self-citation
Metadata
- Authors
- Reilly, A. M., Cooper, R. I., Adjiman, C. S., Bhattacharya, S., Boese, A. D., Brandenburg, J. G., Bygrave, P. J., Bylsma, R., Campbell, J. E., Car, R., Case, D. H., Chadha, R., Cole, J. C., Cosburn, K., Cuppen, H. M., Curtis, F., Day, G. M., DiStasio Jr, R. A., Dzyabchenko, A., van Eijck, B. P., Elking, D. M., van den Ende, J. A., Facelli, J. C., Ferraro, M. B., Fusti-Molnar, L., Gatsiou, C.-A., Gee, T. S., de Gelder, R., Ghiringhelli, L. M., Goto, H., Grimme, S., Guo, R., Hofmann, D. W. M., Hoja, J., Hylton, R. K., Iuzzolino, L., Jankiewicz, W., de Jong, D. T., Kendrick, J., de Klerk, N. J. J., Ko, H.-Y., Kuleshova, L. N., Li, X., Lohani, S., Leusen, F. J. J., Lund, A. M., Lv, J., Ma, Y., Marom, N., Masunov, A. E., McCabe, P., McMahon, D. P., Meekes, H., Metz, M. P., Misquitta, A. J., Mohamed, S., Monserrat, B., Needs, R. J., Neumann, M. A., Nyman, J., Obata, S., Oberhofer, H., Oganov, A. R., Orendt, A. M., Pagola, G. I., Pantelides, C. C., Pickard, C. J., Podeszwa, R., Price, L. S., Price, S. L., Pulido, A., Read, M. G., Reuter, K., Schneider, E., Schober, C., Shields, G. P., Singh, P., Sugden, I. J., Szalewicz, K., Taylor, C. R., Tkatchenko, A., Tuckerman, M. E., Vacarro, F., Vasileiadis, M., Vazquez-Mayagoitia, A., Vogt, L., Wang, Y., Watson, R. E., de Wijs, G. A., Yang, J., Zhu, Q., & Groom, C. R.
- Venue
- Acta Crystallographica Section B Structural Science, Crystal Engineering and Materials
- Published
- 2016-08-01
- Volume
- 72
- Issue
- 4
- Pages
- 439-459
- ISSN
- 2052-5192 (ISSN-L)
- 2052-5206
- OpenAlex
- W2482115826
- Cited by
- 584
- References
- 75
- FWCI
- 46.45
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.