The DOI matches its Crossref record; 95 of the 95 references it cites were found in a registry or an index.
Uncertainty quantification in scientific machine learning: Methods, metrics, and comparisons
Open accessAbstract
Neural networks (NNs) are currently changing the computational paradigm on how to combine data with mathematical laws in physics and engineering in a profound way, tackling challenging inverse and ill-posed problems not solvable with traditional methods. However, quantifying errors and uncertainties in NN-based inference is more complicated than in traditional methods. This is because in addition to aleatoric uncertainty associated with noisy data, there is also uncertainty due to limited data, but also due to NN hyperparameters, overparametrization, optimization and sampling errors as well as model misspecification. Although there are some recent works on uncertainty quantification (UQ) in NNs, there is no systematic investigation of suitable methods towards quantifying the total uncertainty effectively and efficiently even for function approximation, and there is even less work on solving partial differential equations and learning operator mappings between infinite-dimensional function spaces using NNs. In this work, we present a comprehensive framework that includes uncertainty modeling, new and existing solution methods, as well as evaluation metrics and post-hoc improvement approaches. To demonstrate the applicability and reliability of our framework, we present an extensive comparative study in which various methods are tested on prototype problems, including problems with mixed input-output data, and stochastic problems in high dimensions. In the Appendix, we include a comprehensive description of all the UQ methods employed, which we will make available as open-source library of all codes included in this framework.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(2014) · Cited by 33,419
Raissi, M., Perdikaris, P., & Karniadakis, G. E. (2019) · Journal of Computational Physics · doi:10.1016/j.jcp.2018.10.045 · Cited by 20,384
Self-citationAdvances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 15,539
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,887
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Gaussian Processes for Machine Learning
Rasmussen, C. E., & Williams, C. K. I. (2005) · doi:10.7551/mitpress/3206.001.0001 · Cited by 11,705
Physics-informed machine learning
Karniadakis, G. E., Kevrekidis, I. G., Lu, L., Perdikaris, P., Wang, S., & Yang, L. (2021) · Nature Reviews Physics · doi:10.1038/s42254-021-00314-5 · Cited by 8,718
Self-citationImproved training of Wasserstein GANs
Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., & Courville, A. (2017) · arXiv · doi:10.48550/arxiv.1704.00028 · Cited by 6,800
Stochastic Finite Elements: A Spectral Approach
Ghanem, R., & Spanos, P. D. (1991) · Springer eBooks · doi:10.1007/978-1-4612-3094-6 · Cited by 6,552
Bayesian Learning for Neural Networks
Neal, R. M. (1996) · Lecture Notes in Statistics · doi:10.1007/978-1-4612-0745-0 · Cited by 4,276
Variational Inference: A Review for Statisticians
Blei, D. M., Kucukelbir, A., & McAuliffe, J. D. (2017) · Journal of the American Statistical Association · doi:10.1080/01621459.2017.1285773 · Cited by 3,811
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2016) · arXiv · doi:10.48550/arxiv.1612.01474 · Cited by 3,666
Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators
Lu, L., Jin, P., Pang, G., Zhang, Z., & Karniadakis, G. E. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00302-5 · Cited by 3,418
Self-citationAn Introduction to Variational Autoencoders
Kingma, D. P., & Welling, M. (2019) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000056 · Cited by 2,609
MCMC Using Hamiltonian Dynamics
Neal, R. M. (2011) · Handbook of Markov Chain Monte Carlo · doi:10.1201/b10905-6 · Cited by 2,574
DeepXDE: A Deep Learning Library for Solving Differential Equations
Lu, L., Meng, X., Mao, Z., & Karniadakis, G. E. (2021) · SIAM Review · doi:10.1137/19m1274067 · Cited by 2,512
Self-citationArtificial neural networks for solving ordinary and partial differential equations
Lagaris, I. E., Likas, A., & Fotiadis, D. I. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/72.712178 · Cited by 2,412
DGM: A deep learning algorithm for solving partial differential equations
Sirignano, J., & Spiliopoulos, K. (2018) · Journal of Computational Physics · doi:10.1016/j.jcp.2018.08.029 · Cited by 2,208
Understanding and Mitigating Gradient Flow Pathologies in Physics-Informed Neural Networks
Wang, S., Teng, Y., & Perdikaris, P. (2021) · SIAM Journal on Scientific Computing · doi:10.1137/20m1318043 · Cited by 2,048
Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods
Hüllermeier, E., & Waegeman, W. (2021) · Machine Learning · doi:10.1007/s10994-021-05946-3 · Cited by 1,834
Probabilistic Forecasts, Calibration and Sharpness
Gneiting, T., Balabdaoui, F., & Raftery, A. E. (2007) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.1467-9868.2007.00587.x · Cited by 1,783
Proceedings of the 29th International Conference on Machine Learning (ICML-12)
Langford, J. C., & Pineau, J. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1207.4676 · Cited by 1,618
Averaging Weights Leads to Wider Optima and Better Generalization
Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., & Wilson, A. G. (2018) · arXiv · doi:10.48550/arxiv.1803.05407 · Cited by 1,409
Hewson, P. (2015) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2014.12096_1.x · Cited by 1,367
Artificial Intelligence and Statistics
Ziegel, E. (1989) · Technometrics · doi:10.1080/00401706.1989.10488504 · Cited by 1,217
A Survey of Deep Active Learning
Ren, P., Xiao, Y., Chang, X., Huang, P.-Y., Li, Z., Gupta, B. B., Chen, X., & Wang, X. (2022) · ACM Computing Surveys · doi:10.1145/3472291 · Cited by 1,169
Yang, L., Meng, X., & Karniadakis, G. E. (2021) · Journal of Computational Physics · doi:10.1016/j.jcp.2020.109913 · Cited by 1,169
Self-citationD. Jagtap, A., & Em Karniadakis, G. (2020) · Communications in Computational Physics · doi:10.4208/cicp.oa-2020-0164 · Cited by 1,130
fPINNs: Fractional Physics-Informed Neural Networks
Pang, G., Lu, L., & Karniadakis, G. E. (2019) · SIAM Journal on Scientific Computing · doi:10.1137/18m1229845 · Cited by 1,105
Self-citation2014 International Conference on Computational Science
Abramson, D., Lees, M., Krzhizhanovskaya, V. V., Dongarra, J. J., & Sloot, P. M. A. (2014) · UvA-DARE (University of Amsterdam) · Cited by 1,013
Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J. V., Lakshminarayanan, B., & Snoek, J. (2019) · arXiv · doi:10.48550/arxiv.1906.02530 · Cited by 989
Illustration of Bayesian Inference in Normal Data Models Using Gibbs Sampling
Gelfand, A. E., Hills, S. E., Racine-Poon, A., & Smith, A. F. M. (1990) · Journal of the American Statistical Association · doi:10.1080/01621459.1990.10474968 · Cited by 969
Mackay, D. J. C. (1995) · Network Computation in Neural Systems · doi:10.1088/0954-898x/6/3/011 · Cited by 915
UQLab: A Framework for Uncertainty Quantification in Matlab
Marelli, S., & Sudret, B. (2014) · Vulnerability, Uncertainty, and Risk · doi:10.1061/9780784413609.257 · Cited by 899
Meta-Learning in Neural Networks: A Survey
Hospedales, T., Antoniou, A., Micaelli, P., & Storkey, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05439 · Cited by 886
Hands-On Bayesian Neural Networks—A Tutorial for Deep Learning Users
Jospin, L. V., Laga, H., Boussaid, F., Buntine, W., & Bennamoun, M. (2022) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2022.3155327 · Cited by 879
hp-VPINNs: Variational physics-informed neural networks with domain decomposition
Kharazmi, E., Zhang, Z., & Karniadakis, G. E. M. (2021) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/j.cma.2020.113547 · Cited by 818
Self-citationConvergence of a stochastic approximation version of the EM algorithm
Delyon, B., Lavielle, M., & Moulines, E. (1999) · The Annals of Statistics · doi:10.1214/aos/1018031103 · Cited by 778
Meng, X., & Karniadakis, G. E. (2020) · Journal of Computational Physics · doi:10.1016/j.jcp.2019.109020 · Cited by 713
Self-citationDataset Shift in Machine Learning
Adams, N. (2010) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2009.00624_10.x · Cited by 694
Evidential Deep Learning to Quantify Classification Uncertainty
Sensoy, M., Kaplan, L., & Kandemir, M. (2018) · arXiv · doi:10.48550/arxiv.1806.01768 · Cited by 694
PPINN: Parareal physics-informed neural network for time-dependent PDEs
Meng, X., Li, Z., Zhang, D., & Karniadakis, G. E. (2020) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/j.cma.2020.113250 · Cited by 654
Self-citationAlber, M., Buganza Tepole, A., Cannon, W. R., De, S., Dura-Bernal, S., Garikipati, K., Karniadakis, G., Lytton, W. W., Perdikaris, P., Petzold, L., et al. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0193-y · Cited by 639
Self-citationHandbook of Uncertainty Quantification
(2016) · Springer eBooks · doi:10.1007/978-3-319-11259-6 · Cited by 581
Zhang, D., Lu, L., Guo, L., & Karniadakis, G. E. (2019) · Journal of Computational Physics · doi:10.1016/j.jcp.2019.07.048 · Cited by 513
Self-citationPhysics-Informed Generative Adversarial Networks for Stochastic Differential Equations
Yang, L., Zhang, D., & Karniadakis, G. E. (2020) · SIAM Journal on Scientific Computing · doi:10.1137/18m1225409 · Cited by 482
Self-citationTripathy, R. K., & Bilionis, I. (2018) · Journal of Computational Physics · doi:10.1016/j.jcp.2018.08.036 · Cited by 464
A Simple Baseline for Bayesian Uncertainty in Deep Learning
Maddox, W., Garipov, T., Izmailov, P., Vetrov, D., & Wilson, A. G. (2019) · arXiv · doi:10.48550/arxiv.1902.02476 · Cited by 429
Bayesian approach for neural networks—review and case studies
Lampinen, J., & Vehtari, A. (2001) · Neural Networks · doi:10.1016/s0893-6080(00)00098-8 · Cited by 392
Stochastic Gradient Descent as Approximate Bayesian Inference
Mandt, S., Hoffman, M. D., & Blei, D. M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.04289 · Cited by 369
Mishra, S., & Molinaro, R. (2022) · IMA Journal of Numerical Analysis · doi:10.1093/imanum/drab032 · Cited by 336
Zhang, D., Guo, L., & Karniadakis, G. E. (2020) · SIAM Journal on Scientific Computing · doi:10.1137/19m1260141 · Cited by 308
Self-citationA General Framework for Uncertainty Estimation in Deep Learning
Loquercio, A., Segu, M., & Scaramuzza, D. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.2974682 · Cited by 291
Shin, Y., Darbon, J., & Em Karniadakis, G. (2020) · Communications in Computational Physics · doi:10.4208/cicp.oa-2020-0193 · Cited by 284
Neural Network-Based Uncertainty Quantification: A Survey of Methodologies and Applications
Kabir, H. M. D., Khosravi, A., Hosen, M. A., & Nahavandi, S. (2018) · IEEE Access · doi:10.1109/access.2018.2836917 · Cited by 272
A General Framework for Uncertainty Estimation in Deep Learning
Loquercio, A., Segù, M., & Scaramuzza, D. (2020) · Zurich Open Repository and Archive (University of Zurich) · doi:10.5167/uzh-197704 · Cited by 261
Ensemble learning in Bayesian neural networks
Barber, D., & Bishop, C. (1998) · Cited by 256
Priors in Bayesian Deep Learning: A Review
Fortuin, V. (2022) · International Statistical Review · doi:10.1111/insr.12502 · Cited by 255
Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
Garipov, T., Izmailov, P., Podoprikhin, D., Vetrov, D., & Wilson, A. G. (2018) · arXiv · doi:10.48550/arxiv.1802.10026 · Cited by 252
(1986) · Future Generation Computer Systems · doi:10.1016/0167-739x(86)90024-5 · Cited by 247
Stochastic Gradient Markov Chain Monte Carlo
Nemeth, C., & Fearnhead, P. (2020) · Journal of the American Statistical Association · doi:10.1080/01621459.2020.1847120 · Cited by 238
Multi-fidelity Bayesian neural networks: Algorithms and applications
Meng, X., Babaee, H., & Karniadakis, G. E. (2021) · Journal of Computational Physics · doi:10.1016/j.jcp.2021.110361 · Cited by 196
Self-citationBayesian neural networks for uncertainty quantification in data-driven materials modeling
Olivier, A., Shields, M. D., & Graham-Brady, L. (2021) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/j.cma.2021.114079 · Cited by 191
Learning constitutive relations from indirect observations using deep neural networks
Huang, D. Z., Xu, K., Farhat, C., & Darve, E. (2020) · Journal of Computational Physics · doi:10.1016/j.jcp.2020.109491 · Cited by 178
Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning
Ashukha, A., Lyzhov, A., Molchanov, D., & Vetrov, D. (2020) · arXiv · doi:10.48550/arxiv.2002.06470 · Cited by 178
Tanno, R., Worrall, D. E., Kaden, E., Ghosh, A., Grussu, F., Bizzi, A., Sotiropoulos, S. N., Criminisi, A., & Alexander, D. C. (2020) · NeuroImage · doi:10.1016/j.neuroimage.2020.117366 · Cited by 173
The Information Bottleneck Problem and its Applications in Machine Learning
Goldfeld, Z., & Polyanskiy, Y. (2020) · IEEE Journal on Selected Areas in Information Theory · doi:10.1109/jsait.2020.2991561 · Cited by 165
Build, Compute, Critique, Repeat: Data Analysis with Latent Variable Models
Blei, D. M. (2014) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-022513-115657 · Cited by 163
Karumuri, S., Tripathy, R., Bilionis, I., & Panchal, J. (2020) · Journal of Computational Physics · doi:10.1016/j.jcp.2019.109120 · Cited by 159
Deeply uncertain: comparing methods of uncertainty quantification in deep learning algorithms
Caldeira, J. L., & Nord, B. (2020) · Machine Learning Science and Technology · doi:10.1088/2632-2153/aba6f3 · Cited by 124
Deep Learning: A Bayesian Perspective
Polson, N. G., & Sokolov, V. (2017) · Bayesian Analysis · doi:10.1214/17-ba1082 · Cited by 120
UQpy: A general purpose Python package and development environment for uncertainty quantification
Olivier, A., Giovanis, D. G., Aakash, B. S., Chauhan, M., Vandanapu, L., & Shields, M. D. (2020) · Journal of Computational Science · doi:10.1016/j.jocs.2020.101204 · Cited by 120
Valid prediction intervals for regression problems
Dewolf, N., Bernard De Baets, C., & Waegeman, W. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10178-5 · Cited by 97
Learning under Model Misspecification: Applications to Variational and Ensemble methodsPreprint
Masegosa, A. R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.08335 · Cited by 95
Wang, N., Chang, H., & Zhang, D. (2020) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/j.cma.2020.113492 · Cited by 91
Darbon, J., & Meng, T. (2020) · Journal of Computational Physics · doi:10.1016/j.jcp.2020.109907 · Cited by 85
On the Expressiveness of Approximate Inference in Bayesian Neural NetworksPreprint
Foong, A. Y. K., Burt, D. R., Li, Y., & Turner, R. E. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.00719 · Cited by 76
Evaluation of Uncertainty Quantification in Deep Learning
Ståhl, N., Falkman, G., Karlsson, A., & Mathiason, G. (2020) · Communications in computer and information science · doi:10.1007/978-3-030-50146-4_41 · Cited by 74
Conditional deep surrogate models for stochastic, high-dimensional, and multi-fidelity systems
Yang, Y., & Perdikaris, P. G. (2019) · Computational Mechanics · doi:10.1007/s00466-019-01718-y · Cited by 61
Multi-fidelity generative deep learning turbulent flows
Geneva, N., & Zabaras, N. J. (2020) · Foundations of Data Science · doi:10.3934/fods.2020019 · Cited by 55
(2019) · arXiv (Cornell University) · Cited by 52
Emerging Frontiers in Nonlinear Science
Kevrekidis, P. G., Cuevas–Maraver, J., & Saxena, A. (2020) · Nonlinear systems and complexity · doi:10.1007/978-3-030-44992-6 · Cited by 44
Grigo, C., & Koutsourelakis, P. (2019) · Journal of Computational Physics · doi:10.1016/j.jcp.2019.05.053 · Cited by 41
Case Studies in Applied Bayesian Data Science: CIRM Jean-Morlet Chair, Fall 2018
Centre International de Rencontres Mathématiques (Marseille), Ullah, I., & Mengersen, K. L. (2020) · Lecture notes in mathematics · doi:10.1007/978-3-030-42553-1 · Cited by 40
GINNs: Graph-Informed Neural Networks for multiscale physics
Hall, E. J., Taverniers, S., Katsoulakis, M. A., & Tartakovsky, D. M. (2021) · Journal of Computational Physics · doi:10.1016/j.jcp.2021.110192 · Cited by 38
Bayesian Deep Ensembles via the Neural Tangent Kernel
He, B., Lakshminarayanan, B., & Teh, Y. W. (2020) · arXiv · doi:10.48550/arxiv.2007.05864 · Cited by 36
Solving inverse problems in stochastic models using deep neural networks and adversarial training
Xu, K., & Darve, E. (2021) · Computer Methods in Applied Mechanics and Engineering · doi:10.1016/j.cma.2021.113976 · Cited by 34
Uncertainty Quantification in Deep Learning for Safer Neuroimage Enhancement
Tanno, R., Worrall, D., Kaden, E., Ghosh, A., Grussu, F., Bizzi, A., Sotiropoulos, S. N., Criminisi, A., & Alexander, D. C. (2019) · arXiv · doi:10.48550/arxiv.1907.13418 · Cited by 30
Probabilistic inference of Bayesian neural networks with generalized expectation propagation
Zhao, J., Liu, X., He, S., & Sun, S. (2020) · Neurocomputing · doi:10.1016/j.neucom.2020.06.060 · Cited by 20
Inspecting adversarial examples using the fisher information
Martin, J., & Elster, C. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.11.052 · Cited by 18
Yeo, K., Grullon, D. E. C., Sun, F.-K., Boning, D. S., & Kalagnanam, J. (2021) · SIAM Journal on Scientific Computing · doi:10.1137/20m1323151 · Cited by 10
A framework for benchmarking uncertainty in deep regression
Schmähling, F., Martin, J., & Elster, C. (2022) · Applied Intelligence · doi:10.1007/s10489-022-03908-3 · Cited by 9
Detecting unusual input to neural networks
Martin, J., & Elster, C. (2020) · Applied Intelligence · doi:10.1007/s10489-020-01925-8 · Cited by 6
You Can’t Fool All the Models: Detect Adversarial Samples via Pruning Models
Wang, R., Chen, Z., Dong, H. F., & Xuan, Q. (2021) · IEEE Access · doi:10.1109/access.2021.3133334 · Cited by 5
Cited by
Hu, H., Qi, L., & Chao, X. (2024) · Thin-Walled Structures · doi:10.1016/j.tws.2024.112495 · Cited by 335
Cheng, S., Quilodrán-Casas, C., Ouala, S., Farchi, A., Liu, C., Tandeo, P., Fablet, R., Lucor, D., Iooss, B., Brajard, J., et al. (2023) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2023.123537 · Cited by 303
Emotion recognition in EEG signals using deep learning methods: A review
Jafari, M., Shoeibi, A., Khodatars, M., Bagherzadeh, S., Shalbaf, A., García, D. L., Gorriz, J. M., & Acharya, U. R. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107450 · Cited by 240
Bandai, T., & Ghezzehei, T. A. (2022) · Hydrology and Earth System Sciences · doi:10.5194/hess-26-4469-2022 · Cited by 111
Metadata
- Authors
- Psaros, A. F., Meng, X., Zou, Z., Guo, L., & Karniadakis, G. E.
- Venue
- Journal of Computational Physics
- Publisher
- Elsevier BV
- Published
- 2023-03
- Volume
- 477
- Pages
- 111902
- ISSN
- 0021-9991 (ISSN-L)
- OpenAlex
- W4315628777
- Cited by
- 455
- References
- 95
- FWCI
- 43.97
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.