CoChat
Sign Up
Cited by 1,032Markov Chains and Monte Carlo MethodsSimulation Techniques and ApplicationsGaussian Processes and Bayesian Inference

The frontier of simulation-based inference

Open access
Cranmer, Kyle, Brehmer, Johann, Louppe, Gilles·2020·Proceedings of the National Academy of Sciences
DOI 10.1073/pnas.1912789117PMC PMC7720103
Open this paper in CoChat — free account

Abstract

Many domains of science have developed complex simulations to describe phenomena of interest. While these simulations provide high-fidelity models, they are poorly suited for inference and lead to challenging inverse problems. We review the rapidly developing field of simulation-based inference and identify the forces giving additional momentum to the field. Finally, we describe how the frontier is expanding so that a broad audience can appreciate the profound influence these developments may have on science.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1073/pnas.1912789117

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • There are inference problems in which it is not possible to specify or compute the likelihood function.

    “There are also inference problems in which it is not possible to specify or compute the likelihood function.”

    From Meta-learned models of cognition (2024) · cites this paper as Cranmer, Brehmer, & Louppe, 2020

  • In likelihood-free inference, it is typically assumed that one can sample data from the likelihood for a given parameter setting but that computing the corresponding likelihood is impossible.

    “In this setting, it is typically assumed that we can sample data from the likelihood for a given parameter setting but that computing the corresponding likelihood is impossible.”

    From Meta-learned models of cognition (2024) · cites this paper as Cranmer et al., 2020

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 73

  1. [1]Diggle PJ, Gratton RJ (1984) Monte Carlo Methods of Inference for Implicit Statistical Models in Journal of the Royal Statistical Society: Series B (Methodological) . Vol. 46, pp. 193-212.
  2. [2]Rubin DB (1984) Bayesianly Justifiable and Relevant Frequency Calculations for the Applied Statistician. The Annals of Statistics 12(4):1151-1172.
  3. [3]Beaumont MA, Zhang W, Balding DJ (2002) Approximate Bayesian computation in population genetics. Genetics 162(4):2025-2035.
  4. [4]Mohamed S, Lakshminarayanan B (2016) Learning in Implicit Generative Models. arXiv:1610.03483 . · arXiv
  5. [10]Metropolis N, Rosenbluth AW, Rosenbluth MN, Teller AH, Teller E (1953) Equation of state calculations by fast computing machines. The Journal of Chemical Physics 21(6):1087-1092.
  6. [11]Hastings WK (1970) Monte carlo sampling methods using Markov chains and their applications. Biometrika 57(1):97-109.
  7. [14]Marjoram P , Molitor J, Plagnol V, Tavaré S (2003) Markov chain Monte Carlo without likelihoods. Proceedings of the National Academy of Sciences of the United States of America 100(26):15324-15328.
  8. [15]Sisson SA, Fan Y, Tanaka MM (2007) Sequential Monte Carlo without likelihoods. Proceedings of the National Academy of Sciences of the United States of America 104(6):1760-1765.
  9. [17]Brehmer J, Cranmer K, Louppe G, Pavez J (2018) A guide to constraining effective field theories with machine learning. Physical Review D 98(5):052004.
  10. [19]Dinh L, Krueger D, Bengio Y (2015) NICE: Non-linear independent components estimation. 3rd International Conference on Learning Representations, ICLR 2015 - Workshop Track Proceedings .
  11. [20]Dinh L, Sohl-Dickstein J, Bengio S (2019) Density estimation using real NVP in 5th International Conference on Learning Representations, ICLR 2017 - Conference Track Proceedings .
  12. [21]Kingma DP, Dhariwal P (2018) Glow: Generative flow with invertible 1×1 convolutions. Advances in Neural Information Processing Systems 2018-December:10215-10224.
  13. [22]Germain M, Gregor K, Murray I, Larochelle H (2015) MADE: Masked autoencoder for distribution estimation. 32nd International Conference on Machine Learning, ICML 2015 2:881-889.
  14. [23]Uria B, Cote MA, Gregor K, Murray I, Larochelle H (2016) Neural autoregressive distribution estimation. Journal of Machine Learning Research 17:1-37.
  15. [24]Van Den Oord A, Kalchbrenner N, Kavukcuoglu K (2016) Pixel recurrent neural networks. 33rd International Conference on Machine Learning, ICML 2016 4:2611-2620.
  16. [25]Van Den Oord A, et al. (2016) Conditional image generation with PixelCNN decoders. Advances in Neural Information Processing Systems pp. 4797-4805.
  17. [27]Kingma DP, et al. (2016) Improved variational inference with inverse autoregressive flow. Advances in Neural Information Processing Systems pp. 4743-4751.
  18. [28]Papamakarios G, Pavlakou T, Murray I (2017) Masked autoregressive flow for density estimation. Advances in Neural Information Processing Systems 2017-December:2339-2348.
  19. [29]Huang CW, Krueger D, Lacoste A, Courville A (2018) Neural autoregressive flows. 35th International Conference on Machine Learning, ICML 2018 5:3309-3324.
  20. [30]Wehenkel A, Louppe G (2019) Unconstrained monotonic neural networks. CEUR Workshop Proceedings 2491:arXiv:1908.05164. · arXiv
  21. [31]Durkan C, Bekasov A, Murray I, Papamakarios G (2019) Cubic-Spline Flows. arXiv:1906.02145 . · arXiv
  22. [32]Durkan C, Bekasov A, Murray I, Papamakarios G (2019) Neural Spline Flows. arXiv:1906.04032 . · arXiv
  23. [34]Grathwohl W, Chen RT, Bettencourt J, Sutskever I, Duvenaud D (2019) Ffjord: Free-form continuous dynamics for scalable reversible generative models in 7th International Conference on Learning Representations, ICLR 2019 .
  24. [35]Meeds E, Welling M (2014) GPS-ABC: Gaussian process surrogate approximate Bayesian computation in Uncertainty in Artificial Intelligence - Proceedings of the 30th Conference, UAI 2014 , UAI'14. (AUAI Press, Arlington, Virginia, United States), pp. 593-602.
  25. [36]Gutmann MU, Corander J (2016) Bayesian optimization for likelihood-free inference of simulator-based statistical models. Journal of Machine Learning Research 17(1):1-47.
  26. [37]Meeds E, Welling M (2015) Optimization Monte Carlo: Efficient and embarrassingly parallel likelihood-free inference. Advances in Neural Information Processing Systems 2015January:2080-2088.
  27. [38]Järvenpää M, Gutmann MU, Pleska A, Vehtari A, Marttinen P (2019) Efficient acquisition rules for model-based approximate Bayesian computation. Bayesian Analysis 14(2):595-622.
  28. [39]Wang H, Li J (2018) Adaptive Gaussian process approximation for Bayesian inference with expensive likelihood functions. Neural Computation 30(11):3072-3094.
  29. [40]Lueckmann JM, et al. (2017) Flexible statistical inference for mechanistic models of neural dynamics. Advances in Neural Information Processing Systems 2017-December:1290-1300.
  30. [41]Papamakarios G, Sterratt DC, Murray I (2018) Sequential Neural Likelihood: Fast Likelihoodfree Inference with Autoregressive Flows. International Conference on Artificial Intelligence and Statistics .
  31. [42]Ranjan P , Bingham D, Michailidis G (2008) Sequential experiment design for contour estimation from complex computer codes. Technometrics 50(4):527-541.
  32. [43]Bect J, Ginsbourger D, Li L, Picheny V, Vazquez E (2012) Sequential design of computer experiments for the estimation of a probability of failure. Statistics and Computing 22(3):773793.
  33. [45]Cutler M, Walsh TJ, How JP (2014) Reinforcement learning with multi-fidelity simulators in 2014 IEEE International Conference on Robotics and Automation (ICRA) . (IEEE), pp. 38883895.
  34. [46]Hamrick JB, et al. (2019) Metacontrol for adaptive imagination-based optimization. 5th International Conference on Learning Representations, ICLR 2017 - Conference Track Proceedings .
  35. [47]Kandasamy K, Dasarathy G, Schneider J, Póczos B (2017) Multi-fidelity Bayesian optimisation with continuous approximations in 34th International Conference on Machine Learning, ICML 2017 . (JMLR. org), Vol. 4, pp. 2861-2878.
  36. [48]Gordon AD, Henzinger TA, Nori AV, Rajamani SK (2014) Probabilistic programming in Future of Software Engineering, FOSE 2014 - Proceedings . pp. 167-181.
  37. [50]Baydin AG, et al. (2019) Etalumis: Bringing probabilistic programming to scientific simulators at scale. International Conference for High Performance Computing, Networking, Storage and Analysis, SC p. arXiv:1907.03382. · arXiv
  38. [51]Graham MM, Storkey AJ (2017) Asymptotically exact inference in differentiable generative models. Electronic Journal of Statistics 11(2):5105-5164.
  39. [52]Brehmer J, Louppe G, Pavez J, Cranmer K (2020) Mining gold from implicit models to improve likelihood-free inference. Proceedings of the National Academy of Sciences of the United States of America 117(10):5242-5249.
  40. [53]Stoye M, Brehmer J, Louppe G, Pavez J, Cranmer K (2018) Likelihood-free inference with an improved cross-entropy estimator. arXiv:1808.00973 . · arXiv
  41. [54]Güne¸ s Baydin A, Pearlmutter BA, Andreyevich Radul A, Mark Siskind J (2018) Automatic differentiation in machine learning: A survey. Journal of Machine Learning Research 18(153):143.
  42. [57]Gutmann MU, Dutta R, Kaski S, Corander J (2018) Likelihood-free inference via classification. Statistics and Computing 28(2):411-425.
  43. [58]Le TA, Baydin AG, Wood F (2017) Inference compilation and universal probabilistic programming in Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS 2017 , Proceedings of Machine Learning Research. (PMLR, Fort Lauderdale, FL, USA), Vol. 54, pp. 1338-1348.
  44. [60]Putzky P , Welling M (2017) Recurrent Inference Machines for Solving Inverse Problems. arXiv:1706.04008 . · arXiv
  45. [61]Pesah A, Wehenkel A, Louppe G (2018) Recurrent machines for likelihood-free inference in arXiv:1811.12932 . · arXiv
  46. [62]Louppe G, Hermans J, Cranmer K (2019) Adversarial variational optimization of nondifferentiable simulators in CEUR Workshop Proceedings , Proceedings of Machine Learning Research, eds. Chaudhuri K, Sugiyama M. (PMLR), Vol. 2491, pp. 1438-1447.
  47. [63]Rezende DJ, Mohamed S (2015) Variational inference with normalizing flows. 32nd International Conference on Machine Learning, ICML 2015 2:1530-1538.
  48. [64]Papamakarios G, Murray I (2016) Fast e-free inference of simulation models with Bayesian conditional density estimation in Advances in Neural Information Processing Systems . pp. 1036-1044.
  49. [65]Izbicki R, Lee AB, Pospisil T (2019) ABC-CDE: Toward Approximate Bayesian Computation With Complex High-Dimensional Data and Limited Simulations. Journal of Computational and Graphical Statistics 28(3):481-492.
  50. [66]Paige B, Wood F (2016) Inference networks for sequential monte carlo in graphical models. 33rd International Conference on Machine Learning, ICML 2016 6:4434-4444.
  51. [67]Tran D, Ranganath R, Blei DM (2017) Hierarchical implicit models and likelihood-free variational inference in Advances in Neural Information Processing Systems , eds. Guyon I, et al.
  52. Vol. 2017-December, pp. 5524-5534.
  53. [68]Durkan C, Papamakarios G, Murray I (2018) Sequential Neural Methods for Likelihood-free Inference. arXiv:1811.08723 . · arXiv
  54. [69]Lueckmann JM, Bassetto G, Karaletsos T, Macke JH (2018) Likelihood-free inference with emulator networks in Proceedings of The 1st Symposium on Advances in Approximate Bayesian Inference , Proceedings of Machine Learning Research, eds. Ruiz F, Zhang C, Liang D, Bui T. (PMLR), Vol. 96, pp. 32-53.
  55. [70]Alsing J, Charnock T, Feeney S, Wandelt B (2019) Fast likelihood-free cosmology with neural density estimators and active learning. Monthly Notices of the Royal Astronomical Society 488(3):4440-4458.
  56. [71]Neal RM (2008) Computing likelihood functions for high-energy physics experiments when distributions are defined by simulators with nuisance parameters in PHYSTAT LHC Workshop on Statistical Issues for LHC Physics, PHYSTAT 2007 - Proceedings . pp. 111-118.
  57. [72]Fan Y, Nott DJ, Sisson SA (2013) Approximate Bayesian computation via regression density estimation. Stat 2(1):34-48.
  58. [73]Cranmer K, Pavez J, Louppe G (2015) Approximating Likelihood Ratios with Calibrated Discriminative Classifiers. arXiv:1506.02169 . · arXiv
  59. [74]Thomas O, Dutta R, Corander J, Kaski S, Gutmann MU (2016) Likelihood-free inference by ratio estimation. arXiv:1611.10242 . · arXiv
  60. [75]Dinev T, Gutmann MU (2018) Dynamic Likelihood-free Inference via Ratio Estimation (DIRE). arXiv:1810.09899 . · arXiv
  61. [76]Hermans J, Begy V, Louppe G (2019) Likelihood-free MCMC with Amortized Approximate Ratio Estimators. arXiv:1903.04057 . · arXiv
  62. [77]Andreassen A, Nachman B (2019) Neural Networks for Full Phase-space Reweighting and Parameter Tuning. arXiv:1907.08209 . · arXiv
  63. [78]Brehmer J, Cranmer K, Louppe G, Pavez J (2018) Constraining Effective Field Theories with Machine Learning. Physical Review Letters 121(11):111801.
  64. [79]Louppe G, Cho K, Becot C, Cranmer K (2019) QCD-aware recursive neural networks for jet physics. Journal of High Energy Physics 2019(1):57.
  65. [80]Andreassen A, Feige I, Frye C, Schwartz MD (2019) JUNIPR: a framework for unsupervised machine learning in particle physics. European Physical Journal C 79(2):102.
  66. [82]Munk A, et al. (2019) Deep Probabilistic Surrogate Networks for Universal Simulator Approximation. arXiv:1910.11950 . · arXiv
  67. [83]Alsing J, Wandelt B (2018) Generalized massive optimal data compression. Monthly Notices of the Royal Astronomical Society: Letters 476(1):L60-L64.
  68. [84]Alsing J, Wandelt B, Feeney S (2018) Massive optimal data compression and density estimation for scalable, likelihood-free inference in cosmology. Monthly Notices of the Royal Astronomical Society 477(3):2874-2885.
  69. [86]Alsing J, Wandelt B (2019) Nuisance hardened data compression for fast likelihood-free inference. Monthly Notices of the Royal Astronomical Society 488(4):5093-5103.
  70. [87]Wilks SS (1938) The Large-Sample Distribution of the Likelihood Ratio for Testing Composite Hypotheses. The Annals of Mathematical Statistics 9(1):60-62.
  71. [88]Wald A (1943) Tests of Statistical Hypotheses Concerning Several Parameters When the Number of Observations is Large. Transactions of the American Mathematical Society 54(3):426.
  72. [89]Cowan G, Cranmer K, Gross E, Vitells O (2011) Asymptotic formulae for likelihood-based tests of new physics. European Physical Journal C 71(2):1554.
  73. [90]Wood F, Van De Meent JW, Mansinghka V (2014) A new approach to probabilistic programming inference in Journal of Machine Learning Research . Vol. 33, pp. 1024-1032.

Cited by

Sorted by citations

Authors
Cranmer, K., Brehmer, J., & Louppe, G.
Affiliations
  • New York University (US)
  • University of Liège (BE)
Corresponding author
Cranmer, Kyle
ORCID
Venue
Proceedings of the National Academy of Sciences
Publisher
National Academy of Sciences
Published
2020-05-29
Volume
117
Issue
48
Pages
30055-30062
ISSN
  • 0027-8424 (ISSN-L)
  • 1091-6490
DOI
10.1073/pnas.1912789117
PMC
PMC7720103
OpenAlex
W2984417636
Cited by
1,032
References
5
Citations by year
2026: 225 · 2025: 239 · 2024: 192 · 2023: 136 · 2022: 106 · 2021: 91 · 2020: 39 · 2019: 4
FWCI
65.58
Type
journal article
Language
en
Open access
Licence
www.pnas.org/site/aboutpnas/licenses.xhtml (vor, 184 d)
Locations
  • Proceedings of the National Academy of Sciences (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Open Repository and Bibliography (University of Liège) (submittedVersion, other-oa)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (OAC-1836650, OAC-1841471)
  • Gordon and Betty Moore Foundation (MSDSE)
  • Alfred P. Sloan Foundation (MSDSE)
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Markov Chains and Monte Carlo Methods
  • Social Sciences › Decision Sciences › Management Science and Operations Research › Simulation Techniques and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Gaussian Processes and Bayesian Inference
Keywords
simulation-based inference, inverse problems, complex simulation, parameter estimation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review