CoChat

A Bayesian machine scientist to aid in the solution of challenging scientific problems

Open access
Guimerà, Roger, Reichardt, Ignasi, Aguilar-Mogas, Antoni, Massucci, Francesco A., Miranda, Manuel, Pallarès, Jordi, Sales-Pardo, Marta · 2020 · Science Advances

The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.

Abstract

A Bayesian machine scientist uncovers closed-form mathematical models from data.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1126/sciadv.aav6971

Cites

44 of 44 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 MCMC to systematically explore the space of closed-form mathematical expressions. (A) Closed-form mathematical expressions can be represented as trees, whose internal nodes are functions (operations) and whose leaves are variables and parameters. To explore the space of mathematical expressions, we implemented three move types (Materials and Methods): (i) root addition/removal—we added/removed the root of the tree; (ii) elementary tree replacement (we define an elementary tree as a subtree containing, at most, one function)—we replaced one elementary tree by another (for example, c1x by x); (iii) node replacement—we replaced one node (function, variable, or parameter) by another. Each of these moves was proposed with a certain frequency and accepted or rejected using the Metropolis’ rule (Materials and Methods) (36). (B) To validate the MCMC, we explored the space of all expressions with one variable, one parameter and using only the functions {+, sin}. For this experiment only, we restricted the size of the expression tree to be smaller than eight nodes, and we assume that all expressions are equally plausible a priori, p(fi) = const. Each node in the network represents a different expression, and links represent jumps from one expression to another resulting from the MCMC moves described in (A). The size of each node is proportional to the number of times they are visited by the MCMC. In the absence of data, the MCMC explores all possible expressions with equal probability. (C) We generated synthetic data with the expression y = a + x + sin (x), to which we added noise (circles). We show how the correct expression and two closely related expressions fit the data. (D) We explored the same space as in (B) but considering the synthetic data in (C). When data are taken into account, expressions that are more plausible given the data are visited more frequently (larger nodes), and many expressions are too implausible and, therefore, not visited at all (fig. S1). The color of each node corresponds to the color of the curves in (C).
Fig. 1 MCMC to systematically explore the space of closed-form mathematical expressions. (A) Closed-form mathematical expressions can be represented as trees, whose internal nodes are functions (operations) and whose leaves are variables and parameters. To explore the space of mathematical expressions, we implemented three move types (Materials and Methods): (i) root addition/removal—we added/removed the root of the tree; (ii) elementary tree replacement (we define an elementary tree as a subtree containing, at most, one function)—we replaced one elementary tree by another (for example, c1x by x); (iii) node replacement—we replaced one node (function, variable, or parameter) by another. Each of these moves was proposed with a certain frequency and accepted or rejected using the Metropolis’ rule (Materials and Methods) (36). (B) To validate the MCMC, we explored the space of all expressions with one variable, one parameter and using only the functions {+, sin}. For this experiment only, we restricted the size of the expression tree to be smaller than eight nodes, and we assume that all expressions are equally plausible a priori, p(fi) = const. Each node in the network represents a different expression, and links represent jumps from one expression to another resulting from the MCMC moves described in (A). The size of each node is proportional to the number of times they are visited by the MCMC. In the absence of data, the MCMC explores all possible expressions with equal probability. (C) We generated synthetic data with the expression y = a + x + sin (x), to which we added noise (circles). We show how the correct expression and two closely related expressions fit the data. (D) We explored the same space as in (B) but considering the synthetic data in (C). When data are taken into account, expressions that are more plausible given the data are visited more frequently (larger nodes), and many expressions are too implausible and, therefore, not visited at all (fig. S1). The color of each node corresponds to the color of the curves in (C).CC BY-NC · PubMed Central
Fig. 2 The prior probability distribution generates plausible expressions. (A) We extracted 4080 mathematical expressions from Wikipedia pages listed under the category “List of scientific equations named after people”, for example, the page devoted to “Newton’s law of universal gravitation” (see also table S2). (B) After parsing each expression (Materials and Methods), we counted the number of each operation in the expression and the square of these numbers. In Newton’s law of gravitation, we counted two products, one division, and one square. The complete list of operations is listed in table S1. Then, we parametrized the prior distributions p(fi) and fit the parameters so as to generate expressions that have realistic numbers (and squares) of each operation (Materials and Methods). (C to H) We ran the MCMC to draw sets of 4080 synthetic expressions from the prior distribution. We compared these sets of 4080 synthetic expressions with the 4080 empirical expressions obtained from Wikipedia. In particular, for each set of 4080 expressions, we counted the mean number 〈no〉 of each operation o per expression [(C) sine, (D) logarithm, (E) sum], or the mean 〈no²〉 of the square of those numbers (F to H) and plot their probability density function (PDF; pink). The red vertical line indicates the mean of the distribution, which is close to the empirical observation, represented by a dashed black line. The empirical value is always well within the expected ranges of the expressions generated by the MCMC. The results in the figure correspond to expressions with up to five variables and eight parameters.
Fig. 2 The prior probability distribution generates plausible expressions. (A) We extracted 4080 mathematical expressions from Wikipedia pages listed under the category “List of scientific equations named after people”, for example, the page devoted to “Newton’s law of universal gravitation” (see also table S2). (B) After parsing each expression (Materials and Methods), we counted the number of each operation in the expression and the square of these numbers. In Newton’s law of gravitation, we counted two products, one division, and one square. The complete list of operations is listed in table S1. Then, we parametrized the prior distributions p(fi) and fit the parameters so as to generate expressions that have realistic numbers (and squares) of each operation (Materials and Methods). (C to H) We ran the MCMC to draw sets of 4080 synthetic expressions from the prior distribution. We compared these sets of 4080 synthetic expressions with the 4080 empirical expressions obtained from Wikipedia. In particular, for each set of 4080 expressions, we counted the mean number 〈no〉 of each operation o per expression [(C) sine, (D) logarithm, (E) sum], or the mean 〈no²〉 of the square of those numbers (F to H) and plot their probability density function (PDF; pink). The red vertical line indicates the mean of the distribution, which is close to the empirical observation, represented by a dashed black line. The empirical value is always well within the expected ranges of the expressions generated by the MCMC. The results in the figure correspond to expressions with up to five variables and eight parameters.CC BY-NC · PubMed Central
Fig. 3 Recovery of an arbitrary expression from synthetic data. We drew an expression with two variables (x1 and x2) and two parameters (θ1 and θ2) from the prior distribution p(fi); the drawn function is F(x1,x2; θ1,θ2) = x1(θ1 + x2) cos (x1)/[θ2 log (θ2)], with θ1 = −1.19 and θ2 = 0.29. With this function, we generated 400 synthetic points y^k=F(x1^k,x2^k;θ1,θ2)+ε^k, with (x1^k,x2^k) uniformly chosen in [−2,2]2, and ε ∼ N(0,1) normally distributed [points in (A) and (D) to (G); the color of the points corresponds to the value of y as indicated by the color bar]. We fed these synthetic data to the Bayesian machine scientist and run 2500 steps of the MCMC algorithm. (A) After 2500 steps, the most plausible model identified (blue surface) is f(x1, x2; θ1, θ2) = x1(c1 + c2x2) cos x1, which coincides with the correct model up to indeterminacies in the dependency on the parameters that cannot be resolved without varying their values (see movie S1 for the complete evolution of the MCMC). (B) The blue line indicates the evolution of the description length L during the MCMC (up to an irrelevant additive constant that comes from the prior normalization and therefore affects all points equally). Lower description lengths correspond to more plausible models. The MCMC starts from an expression with high description length, but after 1000 to 1500 steps, it equilibrates and samples from the stationary distribution p(fi∣D). (C) The blue line indicates the evolution of the mean absolute error of each model sampled by the MCMC with respect to the true model. The error is calculated over a grid in [−2,2]2, that is, not at the observed points. (D to G) The MCMC is prone to getting trapped in local maxima of the plausibility (local minima of the description length). To explore the expression space more efficiently, we used parallel tempering (38), a technique used in statistical physics to study disordered systems with rugged energy landscapes (Materials and Methods). Besides the main MCMC in (A), parallel tempering keeps a number of parallel MCMCs that sample the distributions p(fi,Tk) ∝ exp [−B(fi)/2Tk + log p(fi)] parametrized by “temperatures” Tk (Materials and Methods). In our case, higher temperatures correspond to simpler expressions. At each MCMC step, we attempted a swap of expressions at contiguous Tk using Metropolis’ rule. The evolution of the description length and the mean absolute error of the parallel MCMCs are represented in (B) and (C) with the same colors as in (D) to (G).
Fig. 3 Recovery of an arbitrary expression from synthetic data. We drew an expression with two variables (x1 and x2) and two parameters (θ1 and θ2) from the prior distribution p(fi); the drawn function is F(x1,x2; θ1,θ2) = x1(θ1 + x2) cos (x1)/[θ2 log (θ2)], with θ1 = −1.19 and θ2 = 0.29. With this function, we generated 400 synthetic points y^k=F(x1^k,x2^k;θ1,θ2)+ε^k, with (x1^k,x2^k) uniformly chosen in [−2,2]2, and ε ∼ N(0,1) normally distributed [points in (A) and (D) to (G); the color of the points corresponds to the value of y as indicated by the color bar]. We fed these synthetic data to the Bayesian machine scientist and run 2500 steps of the MCMC algorithm. (A) After 2500 steps, the most plausible model identified (blue surface) is f(x1, x2; θ1, θ2) = x1(c1 + c2x2) cos x1, which coincides with the correct model up to indeterminacies in the dependency on the parameters that cannot be resolved without varying their values (see movie S1 for the complete evolution of the MCMC). (B) The blue line indicates the evolution of the description length L during the MCMC (up to an irrelevant additive constant that comes from the prior normalization and therefore affects all points equally). Lower description lengths correspond to more plausible models. The MCMC starts from an expression with high description length, but after 1000 to 1500 steps, it equilibrates and samples from the stationary distribution p(fi∣D). (C) The blue line indicates the evolution of the mean absolute error of each model sampled by the MCMC with respect to the true model. The error is calculated over a grid in [−2,2]2, that is, not at the observed points. (D to G) The MCMC is prone to getting trapped in local maxima of the plausibility (local minima of the description length). To explore the expression space more efficiently, we used parallel tempering (38), a technique used in statistical physics to study disordered systems with rugged energy landscapes (Materials and Methods). Besides the main MCMC in (A), parallel tempering keeps a number of parallel MCMCs that sample the distributions p(fi,Tk) ∝ exp [−B(fi)/2Tk + log p(fi)] parametrized by “temperatures” Tk (Materials and Methods). In our case, higher temperatures correspond to simpler expressions. At each MCMC step, we attempted a swap of expressions at contiguous Tk using Metropolis’ rule. The evolution of the description length and the mean absolute error of the parallel MCMCs are represented in (B) and (C) with the same colors as in (D) to (G).CC BY-NC · PubMed Central
Fig. 4 Recovery of the differential equations governing the Rössler system. (A) We generated synthetic data for the Rössler system (equations shown in the panels) with a = b = 0.2 and c = 5.7. We assume that (x, y, z) are measured without error, but the functions that we aim to recover (x.,y.,z.) are measured with two levels of Gaussian noise [e.g., x.^k=F(x^k,y^k,z^k)+ε^k with εk ∼ N(0, σε)]: orange, σε = 1; blue, σε = 5. (B) Trajectories in velocity space (x.,y.,z.) for the levels of noise that we considered. (C to J) For each level of noise, we sampled 10,000 models for x. using the machine scientist and the same for y. and z.. (C and G), For each level of noise, we plot the most plausible model obtained. (D to F and H to J) We plot the distribution of description lengths (up to an irrelevant additive constant) for the models sampled. (D to F) For low noise, the most plausible model among those sampled always coincides with the true model, as indicated by the dashed vertical line. (H to J) For high noise, x. was recovered exactly, whereas for y. and z., the most plausible models corresponded to regularized versions of the true models, in which terms that are comparatively small because of the factor 0.2 are dropped. In all cases, the true model (dashed vertical line) is, at least, among the most plausible ones.
Fig. 4 Recovery of the differential equations governing the Rössler system. (A) We generated synthetic data for the Rössler system (equations shown in the panels) with a = b = 0.2 and c = 5.7. We assume that (x, y, z) are measured without error, but the functions that we aim to recover (x.,y.,z.) are measured with two levels of Gaussian noise [e.g., x.^k=F(x^k,y^k,z^k)+ε^k with εk ∼ N(0, σε)]: orange, σε = 1; blue, σε = 5. (B) Trajectories in velocity space (x.,y.,z.) for the levels of noise that we considered. (C to J) For each level of noise, we sampled 10,000 models for x. using the machine scientist and the same for y. and z.. (C and G), For each level of noise, we plot the most plausible model obtained. (D to F and H to J) We plot the distribution of description lengths (up to an irrelevant additive constant) for the models sampled. (D to F) For low noise, the most plausible model among those sampled always coincides with the true model, as indicated by the dashed vertical line. (H to J) For high noise, x. was recovered exactly, whereas for y. and z., the most plausible models corresponded to regularized versions of the true models, in which terms that are comparatively small because of the factor 0.2 are dropped. In all cases, the true model (dashed vertical line) is, at least, among the most plausible ones.CC BY-NC · PubMed Central
Fig. 5 The machine scientist gives accurate closed-form models and out-of-sample predictions for the Nikuradse dataset. The dataset contains measurements of the turbulent friction λ in rough pipes as a function of the Reynolds number xR and the inverse of the relative roughness xD. (A and B) We fed the machine scientist with the Nikuradse dataset (gray) and sample more than 18,000 models using MCMC as described in the text. The two graphs show the fit of two of the most plausible models (red), indicated at the top of each panel. For comparison, we show the best models identified by alternative machine scientists Eureqa (green) and EFS (orange). The inset in each graph shows the mean absolute error (MAE) of the machine scientist (MS), Eureqa (EU), and EFS (EF) (Materials and Methods), and error bars indicate the 95% confidence interval for the mean. (C and D) We fed the machine scientist with the data points in gray and sample 20,000 models using MCMC. We made predictions (red) for the unobserved data (black) using the median predictive model (see text). For comparison, we show the predictions of Eureqa, EFS, and Gaussian processes (GP) (Materials and Methods). The inset in each graph shows the mean absolute error for all approaches. (E) We considered, again, the models obtained from the whole data as in (A) and (B) but asked what the median predictive model predicts in the fully turbulent regime, well above the highest observed Reynold number (31). Despite the fact that individual sampled models have different behaviors in this region (fig. S5), the median predictive model [obtained as in (C) and (D)] predicts that turbulent friction stays constant at large Reynolds numbers, in agreement with theory (31–33). (F) The machine scientist also predicts the correct scaling (D/r)−1/3 for the turbulent friction in this regime (dashed line). (G) Scaling of the Nikuradse dataset proposed by Prandtl (with ks⁺(Re,r/D)=Re×√(λ/32)×D/r) (30, 33) and most plausible model identified by the machine scientist for the universal scaling function. (H) When unscaled, this simple universal function provides a good description of all the data.
Fig. 5 The machine scientist gives accurate closed-form models and out-of-sample predictions for the Nikuradse dataset. The dataset contains measurements of the turbulent friction λ in rough pipes as a function of the Reynolds number xR and the inverse of the relative roughness xD. (A and B) We fed the machine scientist with the Nikuradse dataset (gray) and sample more than 18,000 models using MCMC as described in the text. The two graphs show the fit of two of the most plausible models (red), indicated at the top of each panel. For comparison, we show the best models identified by alternative machine scientists Eureqa (green) and EFS (orange). The inset in each graph shows the mean absolute error (MAE) of the machine scientist (MS), Eureqa (EU), and EFS (EF) (Materials and Methods), and error bars indicate the 95% confidence interval for the mean. (C and D) We fed the machine scientist with the data points in gray and sample 20,000 models using MCMC. We made predictions (red) for the unobserved data (black) using the median predictive model (see text). For comparison, we show the predictions of Eureqa, EFS, and Gaussian processes (GP) (Materials and Methods). The inset in each graph shows the mean absolute error for all approaches. (E) We considered, again, the models obtained from the whole data as in (A) and (B) but asked what the median predictive model predicts in the fully turbulent regime, well above the highest observed Reynold number (31). Despite the fact that individual sampled models have different behaviors in this region (fig. S5), the median predictive model [obtained as in (C) and (D)] predicts that turbulent friction stays constant at large Reynolds numbers, in agreement with theory (31–33). (F) The machine scientist also predicts the correct scaling (D/r)−1/3 for the turbulent friction in this regime (dashed line). (G) Scaling of the Nikuradse dataset proposed by Prandtl (with ks⁺(Re,r/D)=Re×√(λ/32)×D/r) (30, 33) and most plausible model identified by the machine scientist for the universal scaling function. (H) When unscaled, this simple universal function provides a good description of all the data.CC BY-NC · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 44 references

Research works printed but not linked: 3

  1. W. La Cava, L. Spector, K. Danai, Proceedings of the Genetic and Evolutionary Computation Conference 2016, GECCO ‘16 (ACM, 2016), pp. 741–748.
  2. I. Arnaldo, K. Krawiec, U.-M. O’Reilly, Proceedings of the 2014 Annual Conference on Genetic and Evolutionary Computation, GECCO ‘14 (ACM, 2014), pp. 879–886.
  3. Hoeting J. A., Madigan D., Raftery A. E., Volinsky C. T., Bayesian model averaging: A tutorial. Stat. Sci. 14, 382417 (1999).

Cited by

Metadata

Authors
Guimerà, R., Reichardt, I., Aguilar-Mogas, A., Massucci, F. A., Miranda, M., Pallarès, J., & Sales-Pardo, M.
Affiliations
  • Institució Catalana de Recerca i Estudis Avançats (ES)
  • Universitat Rovira i Virgili (ES)
  • Department of Chemical Engineering, Universitat Rovira i Virgili, Tarragona 43007, Catalonia, Spain.
  • SIRIS Academic (ES)
Corresponding author
Guimerà, Roger
Venue
Science Advances
Published
2020-01-31
Volume
6
Issue
5
Pages
eaav6971-eaav6971
ISSN
2375-2548 (ISSN-L)
OpenAlex
W3003690504
Cited by
136
References
44
Citations by year
2026: 17 · 2025: 24 · 2024: 32 · 2023: 31 · 2022: 15 · 2021: 13 · 2020: 4
FWCI
11.11
Type
journal article
Language
en
Open access
Licence
cc-by-nc
Locations
  • Science Advances (publishedVersion, cc-by-nc)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by-nc)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Rovira & Virgili repository (submittedVersion, cc-by)
  • The Scholarship East Carolina University's Institutional Repository (East Carolina University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Ministerio de Economía y Competitividad, Gobierno de España (FIS2015-71563-ERC, FIS2016-78904-C3-P-1, DPI2016-75791-C2-1-P)
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Markov Chains and Monte Carlo Methods
  • Physical Sciences › Computer Science › Artificial Intelligence › Gaussian Processes and Bayesian Inference
  • Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
Keywords
out-of-sample forecasting, interpretable models

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.