CoChat
Sign Up
Cited by 411Machine Learning in Materials ScienceAdvanced Chemical Physics StudiesComputational Drug Discovery Methods

Benchmarking materials property prediction methods: the Matbench test set and Automatminer reference algorithm

Open access
Dunn, Alexander, Wang, Qi, Ganose, Alex, Dopp, Daniel, Jain, Anubhav·2020·npj Computational Materials
DOI 10.1038/s41524-020-00406-3
Open this paper in CoChat — free accountOpen PDF

Abstract

We present a benchmark test suite and an automated machine learning procedure for evaluating supervised machine learning (ML) models for predicting properties of inorganic bulk materials. The test suite, Matbench, is a set of 13 ML tasks that range in size from 312 to 132k samples and contain data from 10 density functional theory-derived and experimental sources. Tasks include predicting optical, thermal, electronic, thermodynamic, tensile, and elastic properties given a material’s composition and/or crystal structure. The reference algorithm, Automatminer, is a highly-extensible, fully automated ML pipeline for predicting materials properties from materials primitives (such as composition and crystal structure) without user intervention or hyperparameter tuning. We test Automatminer on the Matbench test suite and compare its predictive power with state-of-the-art crystal graph neural networks and a traditional descriptor-based Random Forest model. We find Automatminer achieves the best performance on 8 of 13 tasks in the benchmark. We also show our test suite is capable of exposing predictive advantages of each algorithm—namely, that crystal graph methods appear to outperform traditional machine learning methods given ~10 4 or greater data points. We encourage evaluating materials ML algorithms on the Matbench benchmark and comparing them against the latest version of Automatminer.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41524-020-00406-3

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The crystal structure discovery experiment began with an initial population of 100 groups of parents (200 parent structures) randomly seeded from a reference pool of 5,000 known stable structures from MatBench-bandgapt.

    “Each experiment began with an initial population of 100 groups of parents ( 100 × 2 = 200 parent structures), randomly seeded from the reference pool, which is composed with 5,000 known stable structures from MatBench-bandgapt [103] dataset with lowest deformation energy evaluated by CHGNet [104].”

    From Evaluating Large Language Models in Scientific Discovery (2025) · cites this paper as [103]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 26

  1. Cooper, C. B. et al. Design-to-device approach affords panchromatic co-sensitized solar cells. Adv. Energy Mater. 9, 1802820 (2019).
  2. Agrawal, A. & Choudhary, A. Perspective: materials informatics and big data: realization of the "fourth paradigm" of science in materials science. APL Mater. 4, 053208 (2016).
  3. Deng, J. et al. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition 248-255 (IEEE, 2009).
  4. Rajpurkar, P., Zhang, J., Lopyrev, K. & Liang, P. SQuAD: 100,000+ Questions for Machine Comprehension of Text. Preprint at https://arxiv.org/abs/1606.05250 (2016). · arXiv
  5. Castelli, I. E. et al. New cubic perovskites for one-and two-photon water splitting using the computational materials repository. Energy Environ. Sci. 5, 9034 (2012).
  6. Hastie, T., Tibshirani, R. & Friedman, J. H. (eds) in The elements of statistical learning: data mining, inference, and prediction 2nd edn., Chapter 7, pp. 241-249 (Springer, 2009).
  7. Alexander J. Smola, Arthur Gretton, Karsten M. Borgwardt & Bernhard Scholkopf. Correcting sample selection bias by unlabeled data. In NIPS'06 Proc. 19th Inter- national Conference on Neural Information Processing Systems 601-608 (2006).
  8. Miroslav Dud ́ık, Robert E. Schapire & Steven J. Phillips. Correcting sample selection bias in maximum entropy density estimation. In NIPS'05 Proc. 18th International Conference on Neural Information Processing Systems 323-330 (2005).
  9. Schmidt, J., Marques, M. R. G., Botti, S. & Marques, M. A. L. Recent advances and applications of machine learning in solid-state materials science. Npj Comput. Mater. 5, 83 (2019).
  10. Kabiraj, A., Kumar, M. & Mahapatra, S. High-throughput discovery of high Curie point two-dimensional ferromagnetic materials. Npj Comput. Mater. 6, 35 (2020).
  11. Freedman, D., Pisani, R. & Purves, R. Statistics (international student edition) 4th edn. (W. W. Norton & Company, 2007).
  12. Olson, R. S. et al. In Applications of Evolutionary Computation (eds Squillero, G. & Burelli, P.) vol. 9597 pp. 123-137 (Springer International Publishing, 2016).
  13. Xie, T. & Grossman, J. C. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Phys. Rev. Lett. 120, 145301 (2018).
  14. Ward, L., Agrawal, A., Choudhary, A. & Wolverton, C. A general-purpose machine learning framework for predicting properties of inorganic materials. Npj Comput. Mater. 2, 16028 (2016).
  15. Zhang, Y. & Ling, C. A strategy to apply machine learning to small datasets in materials science. Npj Comput. Mater. 4, 25 (2018).
  16. Goodall, R. E. A. & Lee, A. A. Predicting materials properties without crystal structure: Deep representation learning from stoichiometry. Preprint at https:// arxiv.org/abs/1910.00617 (2019). · arXiv
  17. Xiong, Z. et al. Evaluating explorative prediction power of machine learning algorithms for materials discovery using k-fold forward cross-validation. Comput. Mater. Sci. 171, 109203 (2020).
  18. Emery, A. A. & Wolverton, C. High-throughput DFT calculations of formation energy, stability and oxygen vacancy formation energy of ABO3 perovskites. Sci. Data 4, 170153 (2017).
  19. Stein, H. S., Soedarmadji, E., Newhouse, P. F., Guevarra, Dan & Gregoire, J. M. Synthesis, optical imaging, and absorption spectroscopy data for 179072 metal oxides. Sci. Data 6, 9 (2019).
  20. Krstajic, D., Buturovic, L. J., Leahy, D. E. & Thomas, S. Cross-validation pitfalls when selecting and assessing regression and classification models. J. Cheminforma. 6, 10 (2014).
  21. de Jong, M. et al. Charting the complete elastic properties of inorganic crystalline compounds. Sci. Data 2, 150009 (2015).
  22. Kawazoe, Y., Yu, J.-Z., Tsai, A.-P. & Masumoto, T. Nonequilibrium Phase Diagrams of Ternary Amorphous Alloys (Springer, 1997).
  23. Petousis, I. et al. High-throughput screening of inorganic compounds for the discovery of novel dielectric and optical materials. Sci. Data 4, 160134 (2017).
  24. Petretto, G. et al. High-throughput density-functional perturbation theory pho- nons for inorganic materials. Sci. Data 5, 180065 (2018).
  25. Choudhary, K., Kalish, I., Beams, R. & Tavazza, F. High-throughput identification and characterization of two-dimensional materials using density functional the- ory. Sci. Rep. 7, 5179 (2017).
  26. Choudhary, K., DeCost, B. & Tavazza, F. Machine learning with force-field-inspired descriptors for materials: fast screening and mapping energy landscape. Phys. Rev. Mater. 2, 083801 (2018).

Cited by

Sorted by citations

Authors
Dunn, A., Wang, Q., Ganose, A., Dopp, D., & Jain, A.
Affiliations
  • Lawrence Berkeley National Laboratory (US)
  • University of Kentucky (US)
Corresponding author
  • Dunn, Alexander
  • Jain, Anubhav
ORCID
Venue
npj Computational Materials
Publisher
Springer Science and Business Media LLC
Published
2020-09-15
Volume
6
Issue
1
ISSN
2057-3960 (ISSN-L)
DOI
10.1038/s41524-020-00406-3
OpenAlex
W3023937119
Cited by
411
References
6
Citations by year
2026: 131 · 2025: 93 · 2024: 84 · 2023: 46 · 2022: 41 · 2021: 14 · 2020: 2
FWCI
10.02
Type
journal article
Language
en
Update notices
correction (2020-10-16)
Open access
Licence
Locations
  • npj Computational Materials (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • eScholarship (California Digital Library) (submittedVersion)
  • OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
DOE | SC | Basic Energy Sciences (DE-AC02-05CH11231)
Topics
  • Physical Sciences › Materials Science › Materials Chemistry › Machine Learning in Materials Science
  • Physical Sciences › Physics and Astronomy › Atomic and Molecular Physics, and Optics › Advanced Chemical Physics Studies
  • Physical Sciences › Computer Science › Computational Theory and Mathematics › Computational Drug Discovery Methods
Keywords
Matbench, benchmark suite, materials property prediction, automated machine learning, random forest, density functional theory, elastic properties, thermal properties, electronic properties, thermodynamic properties, tensile properties, optical properties, crystal structure

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review