Scientific machine learning benchmarks
Open accessThe DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
Deep learning has transformed the use of machine learning technologies for the analysis of large experimental datasets. In science, such datasets are typically generated by large-scale experimental facilities, and machine learning focuses on the identification of patterns, trends and anomalies to extract meaningful scientific insights from the data. In upcoming experimental facilities, such as the Extreme Photonics Application Centre (EPAC) in the UK or the international Square Kilometre Array (SKA), the rate of data generation and the scale of data volumes will increasingly require the use of more automated data analysis. However, at present, identifying the most appropriate machine learning algorithm for the analysis of any given scientific dataset is a challenge due to the potential applicability of many different machine learning frameworks, computer architectures and machine learning models. Historically, for modelling and simulation on high-performance computing systems, these issues have been addressed through benchmarking computer applications, algorithms and architectures. Extending such a benchmarking approach and identifying metrics for the application of machine learning methods to open, curated scientific datasets is a new challenge for both scientists and computer scientists. Here, we introduce the concept of machine learning benchmarks for science and review existing approaches. As an example, we describe the SciMLBench suite of scientific machine learning benchmarks. Finding the most appropriate machine learning algorithm for the analysis of any given scientific dataset is currently challenging, but new machine learning benchmarks for science are being developed to help.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2019) · Journal of Computational Physics · doi:10.1016/j.jcp.2018.10.045 · Cited by 20,384
The FAIR Guiding Principles for scientific data management and stewardship
Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., et al. (2016) · Scientific Data · doi:10.1038/sdata.2016.18 · Cited by 18,423
Algorithm AS 136: A K-Means Clustering Algorithm
Hartigan, J. A., & Wong, M. A. (1979) · Applied Statistics · doi:10.2307/2346830 · Cited by 14,523
WeatherBench: A Benchmark Data Set for Data‐Driven Weather Forecasting
Rasp, S., Dueben, P. D., Scher, S., Weyn, J. A., Mouatadid, S., & Thuerey, N. (2020) · Journal of Advances in Modeling Earth Systems · doi:10.1029/2020ms002203 · Cited by 663
‘It will change everything’: DeepMind’s AI makes gigantic leap in solving protein structures
Callaway, E. (2020) · Nature · doi:10.1038/d41586-020-03348-4 · Cited by 499
Sejnowski, T. J. (2018) · doi:10.7551/mitpress/11474.001.0001 · Cited by 438
Research works printed but not linked: 22
- Hey, T., Butler, K., Jackson, S. & Thiyagalingam, J. Machine learning and big scientific data. Philos. Trans. R. Soc. A Math. Phys. Eng. Sci. 378, 20190054 (2020). · DOI
- Greydanus, S., Dzamba, M. & Yosinski, J. in Advances in Neural Information Processing Systems Vol. 32 (eds. Wallach, H. et al.) (Curran Associates, Inc., 2019).
- Butler, K., Le, M., Thiyagalingam, J. & Perring, T. Interpretable, calibrated neural networks for analysis and understanding of inelastic neutron scattering data. J. Phys. Condens. Matter 33, 194006 (2021). · DOI
- Hartigan, J. A. & Wong, M. A. A k-means clustering algorithm. J. R. Stat. Soc. C Appl. Stat. 28, 100–108 (1979).
- Baldi, P. in Proceedings of ICML Workshop on Unsupervised and Transfer Learning Vol. 27 (eds Guyon, I., Dror, G., Lemaire, V., Taylor, G. & Silver, D.) 37–49 (PMLR, 2012).
- Sutton, R. S. & Barto, A. G. Reinforcement Learning: An Introduction (MIT Press, 2018).
- Dongarra, J. & Luszczek, P. in Encyclopedia of Parallel Computing (ed. Padua, D.) 844–850 (Springer, 2011).
- Sakalis, C., Leonardsson, C., Kaxiras, S. & Ros, A. in 2016 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS) 101–111 (IEEE, 2016).
- Bailey, D. H. in Encyclopedia of Parallel Computing (ed. Padua, D.) 1254–1259 (Springer, 2011).
- Dongarra, J. & Luszczek, P. in Encyclopedia of Parallel Computing (ed. Padua, D.) 2055–2057 (Springer, 2011).
- Henghes, B., Pettitt, C., Thiyagalingam, J., Hey, T. & Lahav, O. Benchmarking and scalability of machine-learning methods for photometric redshift estimation. Mon. Not. R. Astron. Soc. 505, 4847–4856 (2021). · DOI
- Müller, A., Karathanasopoulos, N., Roth, C. C. & Mohr, D. Machine learning classifiers for surface crack detection in fracture experiments. Int. J. Mech. Sci. 209, 106698 (2021). · DOI
- Ede, J. M. & Beanland, R. Improving electron micrograph signal-to-noise with an atrous convolutional encoder-decoder. Ultramicroscopy 202, 18–25 (2019). · DOI
- Deng, J. et al. in 2009 IEEE Conference on Computer Vision and Pattern Recognition 248–255 (IEEE, 2009).
- Müller, M., Whitney, B., Henschel, R. & Kumaran, K. in Encyclopedia of Parallel Computing (ed. Padua, D.) 1886–1893 (Springer, 2011).
- Ben-Nun, T. et al. in 2019 IEEE International Parallel and Distributed Processing Symposium (IPDPS) 66–77 (IEEE, 2019). · DOI · Paper page
- James, S., Ma, Z., Rovick Arrojo, D. & Davison, A. J. RLBench: The robot learning benchmark & learning environment. IEEE Robot. Autom. Lett. 5, 3019–3026 (2020). · DOI · Paper page
- Coleman, C. A. et al. in 31st Conference on Neural Information Processing Systems (NIPS 2017) (2017).
- Wu, X. et al. in Proceedings of the 48th International Conference on Parallel Processing 78 (Association for Computing Machinery, 2019).
- Jiang, Z. et al. in 2021 IEEE International Conference on Cluster Computing (CLUSTER) 47–58 (IEEE, 2021).
- Rajpurkar, P., Zhang, J., Lopyrev, K. & Liang, P. in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing 2383–2392 (Association for Computational Linguistics, 2016).
- Cai, L. et al. Surrogate models based on machine learning methods for parameter estimation of left ventricular myocardium. R. Soc. Open Sci. 8, 201121 (2021). · DOI
Cited by
Neural operators for accelerating scientific simulations and design
Azizzadenesheli, K., Kovachki, N., Li, Z., Liu-Schiaffini, M., Kossaifi, J., & Anandkumar, A. (2024) · Nature Reviews Physics · doi:10.1038/s42254-024-00712-5 · Cited by 314
Wang, H., Li, B., Gong, J., & Xuan, F.-Z. (2023) · Engineering Fracture Mechanics · doi:10.1016/j.engfracmech.2023.109242 · Cited by 281
Garbage in, garbage out: mitigating risks and maximizing benefits of AI in research
Hanson, B., Stall, S., Cutcher-Gershenfeld, J., Vrouwenvelder, K., Wirz, C., Rao, Y., & Peng, G. (2023) · Nature · doi:10.1038/d41586-023-03316-8 · Cited by 73
Metadata
- Authors
- Thiyagalingam, J., Shankar, M., Fox, G., & Hey, T.
- Venue
- Nature Reviews Physics
- Published
- 2022-04-06
- Volume
- 4
- Issue
- 6
- Pages
- 413-420
- ISSN
- 2522-5820 (ISSN-L)
- OpenAlex
- W3209295695
- Cited by
- 186
- References
- 10
- FWCI
- 5.12
- Type
- journal article
- Language
- en
- Open access
- bronze
- Licence
- www.springer.com/tdm (tdm)
- www.springer.com/tdm (vor)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Scientific machine learning benchmarks. https://app.cochat.ai/papers/10.1038/s42254-022-00441-7
- The paper
- Thiyagalingam, J., Shankar, M., Fox, G., & Hey, T. (2022). Scientific machine learning benchmarks. Nature Reviews Physics. https://doi.org/10.1038/s42254-022-00441-7
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.