CoChat

Scientific machine learning benchmarks

Open access
Thiyagalingam, Jeyan, Shankar, Mallikarjun, Fox, Geoffrey, Hey, Tony · 2022 · Nature Reviews Physics
Cited by 186

The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.

Abstract

Deep learning has transformed the use of machine learning technologies for the analysis of large experimental datasets. In science, such datasets are typically generated by large-scale experimental facilities, and machine learning focuses on the identification of patterns, trends and anomalies to extract meaningful scientific insights from the data. In upcoming experimental facilities, such as the Extreme Photonics Application Centre (EPAC) in the UK or the international Square Kilometre Array (SKA), the rate of data generation and the scale of data volumes will increasingly require the use of more automated data analysis. However, at present, identifying the most appropriate machine learning algorithm for the analysis of any given scientific dataset is a challenge due to the potential applicability of many different machine learning frameworks, computer architectures and machine learning models. Historically, for modelling and simulation on high-performance computing systems, these issues have been addressed through benchmarking computer applications, algorithms and architectures. Extending such a benchmarking approach and identifying metrics for the application of machine learning methods to open, curated scientific datasets is a new challenge for both scientists and computer scientists. Here, we introduce the concept of machine learning benchmarks for science and review existing approaches. As an example, we describe the SciMLBench suite of scientific machine learning benchmarks. Finding the most appropriate machine learning algorithm for the analysis of any given scientific dataset is currently challenging, but new machine learning benchmarks for science are being developed to help.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s42254-022-00441-7

Cites

10 of 10 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 10 references

Research works printed but not linked: 22

  1. Hey, T., Butler, K., Jackson, S. & Thiyagalingam, J. Machine learning and big scientific data. Philos. Trans. R. Soc. A Math. Phys. Eng. Sci. 378, 20190054 (2020). · DOI
  2. Greydanus, S., Dzamba, M. & Yosinski, J. in Advances in Neural Information Processing Systems Vol. 32 (eds. Wallach, H. et al.) (Curran Associates, Inc., 2019).
  3. Butler, K., Le, M., Thiyagalingam, J. & Perring, T. Interpretable, calibrated neural networks for analysis and understanding of inelastic neutron scattering data. J. Phys. Condens. Matter 33, 194006 (2021). · DOI
  4. Hartigan, J. A. & Wong, M. A. A k-means clustering algorithm. J. R. Stat. Soc. C Appl. Stat. 28, 100–108 (1979).
  5. Baldi, P. in Proceedings of ICML Workshop on Unsupervised and Transfer Learning Vol. 27 (eds Guyon, I., Dror, G., Lemaire, V., Taylor, G. & Silver, D.) 37–49 (PMLR, 2012).
  6. Sutton, R. S. & Barto, A. G. Reinforcement Learning: An Introduction (MIT Press, 2018).
  7. Dongarra, J. & Luszczek, P. in Encyclopedia of Parallel Computing (ed. Padua, D.) 844–850 (Springer, 2011).
  8. Sakalis, C., Leonardsson, C., Kaxiras, S. & Ros, A. in 2016 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) 101–111 (IEEE, 2016).
  9. Bailey, D. H. in Encyclopedia of Parallel Computing (ed. Padua, D.) 1254–1259 (Springer, 2011).
  10. Dongarra, J. & Luszczek, P. in Encyclopedia of Parallel Computing (ed. Padua, D.) 2055–2057 (Springer, 2011).
  11. Henghes, B., Pettitt, C., Thiyagalingam, J., Hey, T. & Lahav, O. Benchmarking and scalability of machine-learning methods for photometric redshift estimation. Mon. Not. R. Astron. Soc. 505, 4847–4856 (2021). · DOI
  12. Müller, A., Karathanasopoulos, N., Roth, C. C. & Mohr, D. Machine learning classifiers for surface crack detection in fracture experiments. Int. J. Mech. Sci. 209, 106698 (2021). · DOI
  13. Ede, J. M. & Beanland, R. Improving electron micrograph signal-to-noise with an atrous convolutional encoder-decoder. Ultramicroscopy 202, 18–25 (2019). · DOI
  14. Deng, J. et al. in 2009 IEEE Conference on Computer Vision and Pattern Recognition 248–255 (IEEE, 2009).
  15. Müller, M., Whitney, B., Henschel, R. & Kumaran, K. in Encyclopedia of Parallel Computing (ed. Padua, D.) 1886–1893 (Springer, 2011).
  16. Ben-Nun, T. et al. in 2019 IEEE International Parallel and Distributed Processing Symposium (IPDPS) 66–77 (IEEE, 2019). · DOI · Paper page
  17. James, S., Ma, Z., Rovick Arrojo, D. & Davison, A. J. RLBench: The robot learning benchmark & learning environment. IEEE Robot. Autom. Lett. 5, 3019–3026 (2020). · DOI · Paper page
  18. Coleman, C. A. et al. in 31st Conference on Neural Information Processing Systems (NIPS 2017) (2017).
  19. Wu, X. et al. in Proceedings of the 48th International Conference on Parallel Processing 78 (Association for Computing Machinery, 2019).
  20. Jiang, Z. et al. in 2021 IEEE International Conference on Cluster Computing (CLUSTER) 47–58 (IEEE, 2021).
  21. Rajpurkar, P., Zhang, J., Lopyrev, K. & Liang, P. in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing 2383–2392 (Association for Computational Linguistics, 2016).
  22. Cai, L. et al. Surrogate models based on machine learning methods for parameter estimation of left ventricular myocardium. R. Soc. Open Sci. 8, 201121 (2021). · DOI

Cited by

See all 186

Metadata

Authors
Thiyagalingam, J., Shankar, M., Fox, G., & Hey, T.
Venue
Nature Reviews Physics
Published
2022-04-06
Volume
4
Issue
6
Pages
413-420
ISSN
2522-5820 (ISSN-L)
OpenAlex
W3209295695
Cited by
186
References
10
FWCI
5.12
Type
journal article
Language
en
Open access
bronze
Licence

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Scientific machine learning benchmarks. https://app.cochat.ai/papers/10.1038/s42254-022-00441-7
The paper
Thiyagalingam, J., Shankar, M., Fox, G., & Hey, T. (2022). Scientific machine learning benchmarks. Nature Reviews Physics. https://doi.org/10.1038/s42254-022-00441-7

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.