Integration of Neural Network-Based Symbolic Regression in Deep Learning for Scientific Discovery
Open accessAbstract
Symbolic regression is a powerful technique to discover analytic equations that describe data, which can lead to explainable models and the ability to predict unseen data. In contrast, neural networks have achieved amazing levels of accuracy on image recognition and natural language processing tasks, but they are often seen as black-box models that are difficult to interpret and typically extrapolate poorly. In this article, we use a neural network-based architecture for symbolic regression called the equation learner (EQL) network and integrate it with other deep learning architectures such that the whole system can be trained end-to-end through backpropagation. To demonstrate the power of such systems, we study their performance on several substantially different tasks. First, we show that the neural network can perform symbolic regression and learn the form of several functions. Next, we present an MNIST arithmetic task where a convolutional network extracts the digits. Finally, we demonstrate the prediction of dynamical systems where an unknown parameter is extracted through an encoder. We find that the EQL-based architecture can extrapolate quite well outside of the training data set compared with a standard neural network-based architecture, paving the way for deep learning to be applied in scientific exploration and discovery.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Symbolic regression can produce human-readable expressions and reduce the number of parameters from many trainable ANN units to a few interpretable ones.
“Symbolic regression is particularly useful for neural-network interpretation: it produces human-readable expressions40 and dramatically reduces the number of parameters (here 7,592 trainable ANN units) to a few interpretable ones.”
From Interpretable abstractions of artificial neural networks predict behavior and neural activity during human information gathering (2026) · cites this paper as [40]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Discovering governing equations from data by sparse identification of nonlinear dynamical systems
Brunton, S. L., Proctor, J. L., & Kutz, J. N. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1517384113 · Cited by 4,796 · see citing works
Distilling Free-Form Natural Laws from Experimental Data
Schmidt, M., & Lipson, H. (2009) · Science · doi:10.1126/science.1165893 · Cited by 3,327 · see citing works
Sparse Approximate Solutions to Linear Systems
Natarajan, B. K. (1995) · SIAM Journal on Computing · doi:10.1137/s0097539792240406 · Cited by 2,866 · see citing works
Genetic programming as a means for programming computers by natural selection
Koza, J. (1994) · Statistics and Computing · doi:10.1007/bf00175355 · Cited by 1,685 · see citing works
Data-driven discovery of partial differential equations
Rudy, S. H., Brunton, S. L., Proctor, J. L., & Kutz, J. N. (2017) · Science Advances · doi:10.1126/sciadv.1602614 · Cited by 1,557 · see citing works
To prune, or not to prune: exploring the efficacy of pruning for model compression
Zhu, M., & Gupta, S. (2017) · arXiv · doi:10.48550/arxiv.1710.01878 · Cited by 805 · see citing works
The Loss Surfaces of Multilayer Networks
Choromanska, A., Henaff, M., Mathieu, M., Arous, G. B., & LeCun, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.0233 · Cited by 733 · see citing works
Learning Sparse Neural Networks through $L_0$ Regularization
Louizos, C., Welling, M., & Kingma, D. P. (2017) · arXiv · doi:10.48550/arxiv.1712.01312 · Cited by 628 · see citing works
Variational Dropout Sparsifies Deep Neural NetworksPreprint
Molchanov, D., Ashukha, A., & Vetrov, D. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1701.05369 · Cited by 612 · see citing works
PDE-Net 2.0: Learning PDEs from data with a numeric-symbolic hybrid deep network
Long, Z., Lu, Y., & Dong, B. (2019) · Journal of Computational Physics · doi:10.1016/j.jcp.2019.108925 · Cited by 552 · see citing works
The State of Sparsity in Deep Neural Networks
Gale, T., Elsen, E., & Hooker, S. (2019) · arXiv · doi:10.48550/arxiv.1902.09574 · Cited by 440 · see citing works
Learning partial differential equations via data discovery and sparse optimization
Schaeffer, H. (2017) · Proceedings of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rspa.2016.0446 · Cited by 436 · see citing works
Xu, Z., Zhang, H., Wang, Y., Chang, X., & Liang, Y. (2010) · Science China Information Sciences · doi:10.1007/s11432-010-0090-0 · Cited by 425 · see citing works
Data-Driven Identification of Parametric Partial Differential Equations
Rudy, S., Alla, A., Brunton, S. L., & Kutz, J. N. (2019) · SIAM Journal on Applied Dynamical Systems · doi:10.1137/18m1191944 · Cited by 302 · see citing works
Training Sparse Neural Networks
Srinivas, S., Subramanya, A., & Babu, R. V. (2017) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.61 · Cited by 179 · see citing works
Batch gradient method with smoothing regularization for training of feedforward neural networks
Wu, W. H., Fan, Q., Żurada, J. M., et al. (2013) · Neural Networks · doi:10.1016/j.neunet.2013.11.006 · Cited by 97 · see citing works
Fan, Q., Żurada, J. M., & Wu, W. H. (2013) · Neurocomputing · doi:10.1016/j.neucom.2013.10.023 · Cited by 57 · see citing works
Unsupervised Learning of Latent Physical Properties Using Perception-Prediction NetworksPreprint
Zheng, D., Luo, V., Wu, J.-J., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.09244 · Cited by 50 · see citing works
Learning Equations for Extrapolation and ControlPreprint
Sahoo, S. S., Lampert, C. H., & Martius, G. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.07259 · Cited by 45 · see citing works
Xu, X., Zongben, Guo, et al. (2012) · Acta Scientiarum Naturalium Universitatis Sunyatseni · Cited by 38 · see citing works
Extrapolation and learning equationsPreprint
Martius, G., & Lampert, C. H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.02995 · Cited by 18 · see citing works
Visual Physics: Discovering Physical Laws from VideosPreprint
Chari, P., Talegaonkar, C., Ba, Y., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.11893 · Cited by 5 · see citing works
Flagged first, then by citations
Research works printed but not linked: 5
- Z. Long, Y. Lu, X. Ma, and B. Dong, "PDE-net: Learning PDEs from data," in Proc. Mach. Learn. Res., Jul. 2018, pp. 3208-3216. [Online]. Available: http://proceedings.mlr.press/v80/long18a.html
- A. Trask, F. Hill, S. E. Reed, J. Rae, C. Dyer, and P. Blunsom, "Neural arithmetic logic units," in Advances in Neural Informa- tion Processing Systems 31, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds. New York, NY, USA: Curran Associates, Inc., 2018, pp. 8035-8044. [Online]. Available: http://papers.nips.cc/paper/8027-neural-arithmetic-logic-units.pdf
- P. Y. Lu, S. Kim, and M. Soljačić, "Extracting interpretable physical parameters from spatiotemporal systems using unsu- pervised learning," 2019, arXiv:1907.06011. [Online]. Available: http://arxiv.org/abs/1907.06011 · arXiv
- T. Tieleman and G. Hinton, "Lecture 6.5-RMSPROP: Divide the gradient by a running average of its recent magnitude," COURSERA, Neural Netw. Mach. Learn., vol. 4, no. 2, pp. 26-31, 2012.
- S. S. Du, J. D. Lee, H. Li, L. Wang, and X. Zhai, "Gradient descent finds global minima of deep neural networks," 2018, arXiv:1811.03804. [Online]. Available: http://arxiv.org/abs/1811.03804 · arXiv
Cited by
Sorted by citations
D’Ambrogio, S., Grohn, J., Khalighinejad, N., Mattar, M. G., Hunt, L., & Rushworth, M. F. S. (2026) · Nature Neuroscience · doi:10.1038/s41593-026-02342-9 · Cited by 1 · see citing works
- Authors
- Kim, S., Lu, P. Y., Mukherjee, S., Gilbert, M., Jing, L., Ceperic, V., & Soljacic, M.
- Affiliations
- Massachusetts Institute of Technology (US)
- Meta (United States) (US)
- Venue
- IEEE Transactions on Neural Networks and Learning Systems
- Published
- 2021-09
- Volume
- 32
- Issue
- 9
- Pages
- 4166-4177
- ISSN
- 2162-237X (ISSN-L)
- 2162-2388
- OpenAlex
- W2996029397
- Cited by
- 215
- References
- 22
- Citations by year
- 2026: 30 · 2025: 56 · 2024: 50 · 2023: 34 · 2022: 23 · 2021: 18 · 2020: 4
- FWCI
- 7.76
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- IEEE Transactions on Neural Networks and Learning Systems (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
- arXiv (Cornell University)
- Bulletin of the American Physical Society
- Funders
- Army Research Office under Cooperative Agreement (W911NF-18-2-0048)
- Department of Defense through the National Defense Science and Engineering Graduate Fellowship (NDSEG) Program
- MIT–SenseTime Alliance on Artificial Intelligence
- Defense Advanced Research Projects Agency (HR00111890042)
- United States Air Force Research Laboratory (FA8750-19-2-1000)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
- Physical Sciences › Physics and Astronomy › Statistical and Nonlinear Physics › Model Reduction and Neural Networks
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- symbolic regression, end-to-end learning, explainable models, extrapolation, parameter estimation, convolutional neural networks, scientific discovery, black-box interpretability
- MeSH terms
- Deep Learning, Biomechanical Phenomena, Computer Systems, Feedback, Humans, Research Design, Linear Models, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
