CoChat
Sign Up
PreprintCited by 190Evolutionary Algorithms and ApplicationsNuclear physics research studiesModel Reduction and Neural Networks

Interpretable Machine Learning for Science with PySR and SymbolicRegression.jl

Open access
Cranmer, Miles·2023·arXiv
DOI 10.48550/arxiv.2305.01582arXiv 2305.01582
Open this paper in CoChat — free accountOpen PDF

Abstract

PySR is an open-source library for practical symbolic regression, a type of machine learning which aims to discover human-interpretable symbolic models. PySR was developed to democratize and popularize symbolic regression for the sciences, and is built on a high-performance distributed back-end, a flexible search algorithm, and interfaces with several deep learning packages. PySR's internal search algorithm is a multi-population evolutionary algorithm, which consists of a unique evolve-simplify-optimize loop, designed for optimization of unknown scalar constants in newly-discovered empirical expressions. PySR's backend is the extremely optimized Julia library SymbolicRegression.jl, which can be used directly from Julia. It is capable of fusing user-defined operators into SIMD kernels at runtime, performing automatic differentiation, and distributing populations of expressions to thousands of cores across a cluster. In describing this software, we also introduce a new benchmark, "EmpiricalBench," to quantify the applicability of symbolic regression algorithms in science. This benchmark measures recovery of historical empirical equations from original and synthetic datasets.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2305.01582

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 59

  1. Donald Gerwin. Information processing, data inferences, and scientific generalization. Behav- ioral Science, 19(5):314-325, 1974.
  2. Pat Langley. BACON: A production system that discovers empirical laws. In IJCAI, 1977.
  3. Pat Langley. Rediscovering physics with BA- CON.3. In Proceedings of the 6th International Joint Conference on Artificial Intelligence - Volume 1, IJCAI'79, pages 505-507, San Fran- cisco, CA, USA, 1979. Morgan Kaufmann Pub- lishers Inc.
  4. Pat Langley, Gary L. Bradshaw, and Herbert A. Simon. BACON.5: The discovery of conserva- tion laws. In IJCAI, 1981.
  5. Pat Langley and Jan M. Zytkow. Data-driven approaches to empirical discovery. Artificial In- telligence, 40(1):283-312, 1989.
  6. John R. Koza. Genetic programming as a means for programming computers by natural selection. Statistics and Computing, 4(2):87- 112, June 1994.
  7. Josh Bongard and Hod Lipson. From the Cover: Automated reverse engineering of nonlinear dy- namical systems. Proceedings of the National Academy of Science, 104(24):9943-9948, June 2007.
  8. Michael Schmidt and Hod Lipson. Distilling Free-Form Natural Laws from Experimental Data. Science, 324(5923):81-85, April 2009.
  9. M. Schmidt and H. Lipson. Symbolic regres- sion of implicit equations. Genetic Program- ming Theory and Practice VII, pages 73-85, 2010.
  10. S. Wagner and M. Affenzeller. HeuristicLab: A Generic and Extensible Optimization Envi- ronment. In Bernardete Ribeiro, Rudolf F. Al- brecht, Andrej Dobnikar, David W. Pearson, and Nigel C. Steele, editors, Adaptive and Nat- ural Computing Algorithms, pages 538-541, Vi- enna, 2005. Springer.
  11. Kalyanmoy Deb, Samir Agrawal, Amrit Pratap, and T. Meyarivan. A Fast Elitist Non- dominated Sorting Genetic Algorithm for Multi-objective Optimization: NSGA-II. In Marc Schoenauer, Kalyanmoy Deb, Günther Rudolph, Xin Yao, Evelyne Lutton, Juan Julian Merelo, and Hans-Paul Schwefel, editors, Paral- lel Problem Solving from Nature PPSN VI, Lec- ture Notes in Computer Science, pages 849-858, Berlin, Heidelberg, 2000. Springer.
  12. K. Deb, A. Pratap, S. Agarwal, and T. Meyari- van. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Transactions on Evolutionary Computation, 6(2):182-197, April 2002.
  13. J.W. Davidson, D.A. Savic, and G.A. Wal- ters. Symbolic and numerical regression: Ex- periments and applications. Information Sci- ences, 150(1):95-117, 2003.
  14. Kyle Cranmer and R. Sean Bowman. Physic- sGP: A Genetic Programming approach to event selection. Computer Physics Communi- cations, 167(3):165-176, May 2005.
  15. William La Cava, Lee Spector, and Kourosh Danai. Epsilon-Lexicase Selection for Re- gression. In Proceedings of the Genetic and Evolutionary Computation Conference 2016, GECCO '16, pages 741-748, New York, NY, USA, July 2016. Association for Computing Machinery.
  16. Marco Virgolin, Tanja Alderliesten, Cees Wit- teveen, and Peter A. N. Bosman. Scalable ge- netic programming by gene-pool optimal mix- ing and input-space entropy-based building- block learning. In Proceedings of the Ge- netic and Evolutionary Computation Confer- ence, GECCO '17, pages 1041-1048, New York, NY, USA, July 2017. Association for Comput- ing Machinery.
  17. Marco Virgolin, Tanja Alderliesten, Cees Wit- teveen, and Peter A. N. Bosman. Improving Model-based Genetic Programming for Sym- bolic Regression of Small Expressions. Evo- lutionary Computation, 29(2):211-237, June 2021.
  18. Miles D. Cranmer, Rui Xu, Peter Battaglia, and Shirley Ho. Learning Symbolic Physics with Graph Networks. ML4Physics Workshop @ NeurIPS 2019, November 2019.
  19. Miles Cranmer, Alvaro Sanchez-Gonzalez, Pe- ter Battaglia, Rui Xu, Kyle Cranmer, David Spergel, and Shirley Ho. Discovering Symbolic Models from Deep Learning with Inductive Bi- ases. NeurIPS, June 2020.
  20. Miles Cranmer, Can Cui, Drummond B Field- ing, Shirley Ho, Alvaro Sanchez-Gonzalez, Kimberly Stachenfeld, Tobias Pfaff, et al. Dis- entangled Sparsity Networks for Explainable AI. Workshop on Sparse Neural Networks, page 7, July 2021.
  21. Roger Guimerà, Ignasi Reichardt, Antoni Aguilar-Mogas, Francesco A. Massucci, Manuel Miranda, Jordi Pallarès, and Marta Sales- Pardo. A Bayesian machine scientist to aid in the solution of challenging scientific problems. Science Advances, 6(5), 2020.
  22. Silviu-Marian Udrescu, Andrew Tan, Jiahai Feng, Orisvaldo Neto, Tailin Wu, and Max Tegmark. AI Feynman 2.0: Pareto-optimal symbolic regression exploiting graph modular- ity. In Advances in Neural Information Pro- cessing Systems, volume 33, pages 4860-4871.
  23. Ziming Liu and Max Tegmark. AI poincaré: Machine learning conservation laws from trajec- tories. arXiv e-prints, page arXiv:2011.04698, November 2020. · arXiv
  24. Sebastian J. Wetzel, Roger G. Melko, Joseph Scott, Maysum Panju, and Vijay Ganesh. Dis- covering symmetry invariants and conserved quantities by interpreting siamese neural net- works. Physical Review Research, 2(3):033499, September 2020.
  25. Subham Sahoo, Christoph Lampert, and Georg Martius. Learning Equations for Extrapola- tion and Control. volume 80 of Proceedings of Machine Learning Research, pages 4442- 4450, Stockholmsmässan, Stockholm Sweden, July 2018. PMLR.
  26. M. Brameier and W. Banzhaf. A comparison of linear genetic programming and neural net- works in medical data mining. IEEE Trans- actions on Evolutionary Computation, 5(1):17- 26, February 2001.
  27. Aytac Guven. Linear genetic programming for time-series modelling of daily flow rate. Journal of Earth System Science, 118(2):137-146, April 2009.
  28. Douglas Mota Dias and Marco Aurélio C. Pacheco. Describing Quantum-Inspired Linear Genetic Programming from symbolic regression problems. In 2012 IEEE Congress on Evolu- tionary Computation, pages 1-8, June 2012.
  29. Steven L. Brunton, Joshua L. Proctor, and J. Nathan Kutz. Discovering governing equa- tions from data by sparse identification of non- linear dynamical systems. Proceedings of the National Academy of Sciences, 113(15):3932- 3937, 2016.
  30. Samuel H Rudy, Steven L Brunton, Joshua L Proctor, and J Nathan Kutz. Data-driven dis- covery of partial differential equations. Science Advances, 3(4):e1602614, 2017.
  31. Kathleen Champion, Bethany Lusch, J. Nathan Kutz, and Steven L. Brunton. Data-driven dis- covery of coordinates and governing equations. arXiv e-prints, page arXiv:1904.02107, March 2019. · arXiv
  32. Bogdan Burlacu, Gabriel Kronberger, and Michael Kommenda. Operon C++: An effi- cient genetic programming framework for sym- bolic regression. In Proceedings of the 2020 Genetic and Evolutionary Computation Con- ference Companion, GECCO '20, pages 1562- 1570, New York, NY, USA, July 2020. Associ- ation for Computing Machinery.
  33. Trent McConaghy. FFX: Fast, Scalable, Deter- ministic Symbolic Regression Technology. In Rick Riolo, Ekaterina Vladislavleva, and Ja- son H. Moore, editors, Genetic Programming Theory and Practice IX, Genetic and Evolu- tionary Computation, pages 235-260. Springer, New York, NY, 2011.
  34. Alison Cozad and Nikolaos V. Sahinidis. A global MINLP approach to symbolic regres- sion. Mathematical Programming, 170(1):97- 119, July 2018.
  35. Cristina Cornelio, Sanjeeb Dash, Vernon Aus- tel, Tyler Josephson, Joao Goncalves, Kenneth Clarkson, Nimrod Megiddo, et al. AI Descartes: Combining Data and Theory for Derivable Sci- entific Discovery. arXiv:2109.01634 [cs], Octo- ber 2021. · arXiv
  36. Astropy Collaboration, Adrian M. Price- Whelan, Pey Lian Lim, Nicholas Earl, Nathaniel Starkman, Larry Bradley, David L. Shupe, et al. The Astropy Project: Sustain- ing and Growing a Community-oriented Open- source Project and the Latest Major Release (v5.0) of the Core Package. The Astrophysical Journal, 935:167, August 2022.
  37. Anne Brindle. Genetic Algorithms for Function Optimization. PhD thesis, 1980.
  38. David E. Goldberg and Kalyanmoy Deb. A Comparative Analysis of Selection Schemes Used in Genetic Algorithms. In GREGORY J. E. Rawlins, editor, Foundations of Genetic Algorithms, volume 1, pages 69-93. Elsevier, January 1991.
  39. Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. Regularized Evolution for Im- age Classifier Architecture Search. Proceedings of the AAAI Conference on Artificial Intelli- gence, 33(01):4780-4789, July 2019.
  40. Esteban Real, Chen Liang, David So, and Quoc Le. AutoML-Zero: Evolving Machine Learn- ing Algorithms From Scratch. In International Conference on Machine Learning, pages 8007- 8019. PMLR, November 2020.
  41. Gregory S. Hornby. ALPS: The age-layered population structure for reducing the problem of premature convergence. In Proceedings of the 8th Annual Conference on Genetic and Evolu- tionary Computation, GECCO '06, pages 815- 822, New York, NY, USA, July 2006. Associa- tion for Computing Machinery.
  42. Michael D. Schmidt and Hod Lipson. Age- fitness pareto optimization. In Proceedings of the 12th Annual Conference on Genetic and Evolutionary Computation, GECCO '10, pages 543-544, New York, NY, USA, July 2010. As- sociation for Computing Machinery.
  43. C. G. Broyden. The convergence of a class of double-rank minimization algorithms 1. Gen- eral considerations. IMA Journal of Applied Mathematics, 6(1):76-90, March 1970.
  44. Patrick Kofod Mogensen and Asbjørn Nilsen Riseth. Optim: A mathematical optimization package for Julia. Journal of Open Source Soft- ware, 3(24):615, 2018.
  45. Alexander Topchy and W. F. Punch. Faster genetic programming based on local gradient search of numeric leaf values. In Proceedings of the 3rd Annual Conference on Genetic and Evolutionary Computation, GECCO'01, pages 155-162, San Francisco, CA, USA, July 2001.
  46. Charles R. Harris, K. Jarrod Millman, Sté- fan J. van der Walt, Ralf Gommers, Pauli Vir- tanen, David Cournapeau, Eric Wieser, et al. Array programming with NumPy. Nature, 585(7825):357-362, September 2020.
  47. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, et al. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035.
  48. Nguyen Quang Uy, Nguyen Xuan Hoai, Michael O'Neill, R. I. McKay, and Edgar Galván-López. Semantically-based crossover in genetic pro- gramming: Application to real-valued symbolic regression. Genetic Programming and Evolvable Machines, 12(2):91-119, June 2011.
  49. Edwin Hubble. A relation between distance and radial velocity among extra-galactic neb- ulae. Proceedings of the National Academy of Sciences, 15(3):168-173, March 1929.
  50. William H. Press and Paul Schechter. Forma- tion of Galaxies and Clusters of Galaxies by Self-Similar Gravitational Condensation. The Astrophysical Journal, 187:425-438, February 1974.
  51. Émile Clapeyron. Mémoire sur la puissance motrice de la chaleur. Journal de l'École Poly- technique, 1835.
  52. Johannes Rydberg. Researches sur la consti- tution des spectres d'émission des Éléments chimiques. Proceedings of the Royal Swedish Academy of Science, 1889.
  53. Yanzhang Li, Hongyu Wang, Yan Li, Huan Ye, Yanan Zhang, Rongzhang Yin, Haoning Jia, et al. Electron transfer rules of minerals under pressure informed by machine learning. Nature Communications, 14(1):1815, March 2023.
  54. Konstantin T. Matchev, Katia Matcheva, and Alexander Roman. Analytical Modeling of Ex- oplanet Transit Spectroscopy with Dimensional Analysis and Symbolic Regression. The Astro- physical Journal, 930(1):33, May 2022.
  55. Sergiy Verstyuk and Michael R. Douglas. Ma- chine Learning the Gravity Equation for Inter- national Trade. SSRN Electronic Journal, 2022.
  56. Kaze W. K. Wong and Miles Cranmer. Auto- mated discovery of interpretable gravitational- wave population models. ML4Astro Workshop @ ICML 2022, July 2022.
  57. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, et al. Scikit- learn: Machine Learning in Python. Journal of Machine Learning Research, 12(85):2825-2830, 2011.
  58. Wes McKinney. Data Structures for Statistical Computing in Python. In Stéfan van der Walt and Jarrod Millman, editors, Proceedings of the 9th Python in Science Conference, pages 56 - 61, 2010.
  59. Shashi Gowda, Yingbo Ma, Alessandro Cheli, Maja Gwóźzdź, Viral B. Shah, Alan Edel- man, and Christopher Rackauckas. High- performance symbolic-numerics via multiple dispatch. ACM Communications in Computer Algebra, 55(3):92-96, January 2022.

Cited by

Sorted by citations

Related

Authors
Cranmer, M.
Affiliations
Princeton University (US)
Corresponding author
Cranmer, Miles
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2305.01582
arXiv
2305.01582
OpenAlex
W4368303491
Cited by
190
References
5
Citations by year
2026: 36 · 2025: 82 · 2024: 63 · 2023: 9
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
  • Physical Sciences › Physics and Astronomy › Nuclear and High Energy Physics › Nuclear physics research studies
  • Physical Sciences › Physics and Astronomy › Statistical and Nonlinear Physics › Model Reduction and Neural Networks
Keywords
symbolic regression, PySR, neutron star matter, equation of state, relativistic mean field, empirical equations, multi-population evolutionary algorithm

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review