CoChat
Sign Up
Cited by 597Artificial Intelligence in Healthcare and EducationAdversarial Robustness in Machine LearningExplainable Artificial Intelligence (XAI)

The need for uncertainty quantification in machine-assisted medical decision making

Open access
Begoli, Edmon, Bhattacharya, Tanmoy, Kusnezov, Dimitri·2019·Nature Machine Intelligence
DOI 10.1038/s42256-018-0004-1
Open this paper in CoChat — free accountOpen PDF

AI snapshot

Written from the paper’s full text.

TL;DR

This perspective argues that machine-assisted medical decision making, especially deep learning, requires a principled and formal discipline of uncertainty quantification (UQ) comparable to that used in nuclear stockpile stewardship and risk management. It maps where uncertainty enters the DL pipeline, identifies four overlapping challenge areas, and sketches research directions for building UQ for DL.

Key findings

  • The paper contends that predictions without UQ "are neither predictions nor actionable," and that automated clinical systems not analyzable via known causal connections will require formal UQ to be adopted, as happened in nuclear security.
  • It identifies four overlapping challenge groups for UQ of data-driven approaches: (1) absence of theory (no "hard laws" in medicine), (2) absence of causal models (DL exploits correlations without causal links), (3) sensitivity to imperfect data (missing elements and errors with patterned confounds), and (4) computational expense of recomputation for uncertainty bounds.
  • It proposes that uncertainty must be quantified across at least four pipeline steps: collection/selection of training data and its coverage of the medical phenomenon, accuracy and completeness of labelling, selection and understanding of the DL model and its performance bounds, and model performance against operational (clinical inference) data.
  • It highlights three research directions: quantifying and limiting overfitting (noting classical information criteria fail given DL's parameter counts and capacity to memorize random noise), understanding DL internals, and training DL to provide its own uncertainty estimates—suggesting GANs for detecting out-of-sample cases as "particularly fruitful."
  • Cited (not original) performance claims: DL transfer-learning applications reported performance comparable to human experts or better, and DL function estimators reported performance better than state-of-the-art classical predictive models.

Methods

  • Design: a conceptual/perspective article (no empirical study, dataset, or experiment conducted by the authors).
  • Content: narrative discussion of the role of DL in clinical decision making, the proposed place of UQ, anticipated challenges, and foreseen research directions, illustrated by a figure of uncertainty sources in the DL pipeline.
  • Analysis: argumentative synthesis drawing analogies to UQ practice in nuclear security and risk management; no statistical analysis is performed.

Limitations

  • Stated by the paper: the domains where DL is applied lack underlying mechanistic theory to ground uncertainties; DL lacks causal models, limiting conclusions; real-world data is imperfect with patterned errors that can confound prediction; and recomputation for uncertainty bounds may currently be prohibitively expensive.
  • Evident in the text: the article is a position/perspective piece with no empirical validation, no concrete UQ framework or method proposed, and no quantitative results of its own; the authors explicitly note that "it is possible that the entire new field of UQ for DL might need to be developed" and that a review of ongoing research is beyond its scope.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s42256-018-0004-1

Cites

29 of 29 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 20 cited sources checked · 0 supported · 0 flagged

  • Interpretationreasoning

    In applying DL we are ultimately operating without the fundamental underlying mathematical model that could otherwise be used to ground uncertainties and bound assumptions.

    “we are ultimately operating without the fundamental, underlying mathematical model which we could otherwise use to ground our uncertainties and to bound any assumptions we make.”
    Quote in the paperSubstantial claim
  • Premisecitation

    Advancements in UQ now provide measures of confidence necessary to inform national or international security decisions.

    “Advancements in UQ now provide measures of confidence necessary to inform national or international security decisions.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [1] Oberkampf, W. L. & Roy, C. J. Verfication and Validation in Scientific Computing (Cambridge Univ. Press, Cambridge, 2010). source

      Not checked in the free run.

    • cites [2] National Research Council Evaluation of Quantification of Margins and Uncertainties: Methodology for Assessing and Certifying the Reliability of the Nuclear Stockpile (National Academies Press, Washington DC, 2009).

      Not checked in the free run.

  • Interpretationreasoning

    The absence of a causal connection means garnering limited conclusions from DL models.

    “The absence of a causal connection, however, means garnering limited conclusions from DL models”
    Quote in the paperNot a substantial claim
  • Premisecitation

    Most partial theories in medicine explain an extremely small fraction of the observed phenomena and variations.

    “most partial theories in medicine still explain an extremely small fraction of the observed phenomena and variations 3,4 .”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [3] Zuk, O., Hechter, E., Sunyaev, S. R. & Lander, E. S. The mystery of missing heritability: Genetic interactions create phantom heritability. Proc. Natl Acad. Sci. USA 109, 1193–1198 (2012). source

      Not checked in the free run.

    • cites [4] Choi, J. D. & Lee, J.-S. Interplay between epigenetics and genetics in cancer. Genomics Inform. 11, 164–173 (2013). source

      Not checked in the free run.

  • Premisecitation

    Applications of DL techniques based on transfer learning have reported performance comparable to or better than that of human experts.

    “The applications of DL techniques based on transfer learning have reported performance comparable to that of the human experts 5 , , 6 or better 7 ., 8”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [5] Kermany, D. S. et al. Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell 172, 1122–1131 (2018). source

      Not checked in the free run.

    • cites [6] Esteva, A. et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 115–118 (2017). source

      Not checked in the free run.

    • cites [7] Mar, V. & Soyer, H. Artificial intelligence for melanoma diagnosis: How can we deliver on the promise? Ann. Oncol. 29, 1625–1628 (2018). source

      Not checked in the free run.

  • Classified as probably not a substantial claim. 44
  • May combine two claims. 16

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Computing capacity in support of DL is growing exponentially.

    “computing capacity in support of DL is growing exponentially”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    UQ today underpins many decision processes in nuclear security, risk management and associated investments.

    “UQ today underpins many decision processes in nuclear security, our risk management and associated investments, which can be at the scale of billions of dollars.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    All of the steps of the DL pipeline would need to be quantified in order to arrive at even crude overall measures of the uncertainty of the DL-based decision model.

    “we propose that all of these steps (illustrated in figure 1) would need to be quantified in order to arrive at even crude, overall measures of the uncertainty of the DL-based decision model.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    DL-structured neural networks are particularly suitable for recognition of visually manifested conditions such as changes in tissue, lesions and growth.

    “as DL-structured neural networks are particularly suitable for recognition of visually manifested conditions such as changes in tissue, lesions and growth, etc.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given

    Real world data is usually imperfect, typically containing missing elements and errors, and these imperfections have patterns that can confound prediction.

    “Real world data is usually imperfect-typically containing missing elements and errors-and these imperfections have patterns that can confound prediction.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisereasoning

    The domains where DL is usually applied, such as medicine, do not have well-understood hard laws.

    “the domains where the DL is usually applied, such as medicine, do not have such”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    There are at least four overlapping groups of challenges associated with the uncertainty quantification of data-driven approaches such as DL.

    “We see at least four overlapping groups of challenges associated with the uncertainty quantification of the data-driven approaches such as DL:”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Premisenone given

    UQ extends the traditional discipline of statistical error analysis to also capture uncertainties due to possibly incomplete, inaccurate, and contradictory input data, missing and undetected mechanisms and dependencies, expert judgment, and variations between reasonable model forms and modeling strategies.

    “To achieve this confidence, UQ extends the traditional discipline of statistical error analysis to also capture uncertainties due to possibly incomplete, inaccurate, and contradictory input data, missing and undetected mechanisms and dependencies, expert judgment, and variations between reasonable model forms and modeling strategies.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    Use of scientific insights and attempts at a cohesive framework incorporating the major clinical predictors are likely to be increasingly useful as predictive models efficiently summarize more complex correlations in the data.

    “use of scientific insights and attempts at a cohesive framework incorporating the major clinical predictors are likely to be increasingly useful as predictive models are able to efficiently summarize more complex correlations in the data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationreasoning

    Predictions without UQ are neither predictions nor actionable.

    “Predictions without UQ are neither predictions nor actionable.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Premisereasoning

    AI-based research in medicine has been one of the more challenging areas for AI to see an effective adoption.

    “Although AI-based research has long played an important role in medicine, it has, nevertheless, been one of the more challenging areas for AI to see an effective adoption.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The reluctance to delegate to machine intelligence a function of life-critical and patient-safety-related decision making is a primary reason for the slow adoption of AI in medicine.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given

    In most presented cases, applications of AI are based on DL neural networks trained on a very large number of labeled data sets, with learning tuned with a large number of hyperparameters.

    “In most of the presented cases, the applications of AI are based on the DL neural networks, trained on a very large number of labeled data sets, and their learning tuned with the large number of hyperparameters.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    A need to systematize a similar situation is what led to the development of the formal approach to UQ in US national security sciences.

    “A need to systematize a similar situation is what actually led us to develop the formal approach to UQ in US national security sciences.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    Medical AI, especially in its modern data-rich Deep Learning guise, needs to develop a principled and formal Uncertainty Quantification discipline, as has been seen in fields such as nuclear stockpile stewardship or risk management.

    “medical AI, especially in its modern data-rich Deep Learning guise, needs to develop a principled and formal Uncertainty Quantification (UQ) discipline, just as we have seen in fields such as nuclear stockpile stewardship or risk management.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The reluctance to delegate to machine intelligence a function of life-critical and patient-safety-related decision making is a primary reason for the slow adoption of AI in medicine.

  • May combine two claims.

    Premisecitation

    The US has supported a nuclear test moratorium since 1992, with detailed measures of confidence in the safety, security, and performance of the nuclear stockpile annually provided through virtual testing.

    “A notable example is the US support of a nuclear test moratorium since 1992, whereby we annually provide detailed measures of confidence in the safety, security, and performance of the nuclear stockpile--guaranteed through virtual testing 1,2 .”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [1] Oberkampf, W. L. & Roy, C. J. Verfication and Validation in Scientific Computing (Cambridge Univ. Press, Cambridge, 2010). source

      Not checked in the free run.

    • cites [2] National Research Council Evaluation of Quantification of Margins and Uncertainties: Methodology for Assessing and Certifying the Reliability of the Nuclear Stockpile (National Academies Press, Washington DC, 2009).

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    The data-rich world of AI-based learning and the frequent absence of a well-understood underlying theory poses unique challenges to straightforward adoption of UQ.

    “The data-rich world of AI-based learning, and the frequent absence of a well-understood underlying theory poses its own unique challenges to straightforward adoption of UQ.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    There are many steps in DL scenarios where there are uncertainties that would need to be quantified.

    “Obviously, there are many steps in such scenarios where there are uncertainties that would need to be quantified.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Ad hoc solutions such as sensitivity analysis and study of model variability have sometimes been employed to mitigate some of the outlined problems.

    “ad hoc solutions, such as sensitivity analysis and study of model variability, have sometimes been employed to mitigate some the problems we outline.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    Once a low-dimensional representation describing certain correlations is arrived at, it can raise hypotheses that can be tested.

    “once a low-dimensional representation is arrived at describing certain correlations (e.g., the difference between cancerous and matched normal cells), it can raise hypotheses that can be tested.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisereasoning

    DL universal approximators are trained to reconstruct the outcome of some generative function, without an explicit attempt to specify the exact mathematical model behind the process.

    “These are then”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Premisenone given

    Prediction does not need a causal relation.

    “This may not seem like a limitation since prediction does not need causal relation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationnone given

    Understanding prediction system structure and defensibly quantifying uncertainty is possible, and if done can significantly benefit both research and practical applications of AI in medicine.

    “Understanding prediction system structure and defensibly quantifying uncertainty is possible, and, if done, can significantly benefit both research and practical applications of AI in this critical domain.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    Automated systems adoption, especially systems not analyzable in terms of known causal connections, will require principled and formal UQ to play a transformative role.

    “Automated systems adoption, especially systems not analyzable in terms of known causal connections, will require principled and formal UQ to play a transformative role, just as we have seen in the nuclear security domain.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Techniques are being developed to approximate some UQ-relevant calculations.

    “techniques are being developed 17 to approximate some of the UQ-relevant calculations.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [17] Gal, Y. & Ghahramani, Z. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. Preprint at https://arxiv.org/abs/1506.02142 (2015).

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    Numerical results without a measure of their veracity do not provide the trust needed to inform decisions.

    “But numerical results, without a measure of their veracity do not provide the trust needed to inform decisions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given

    The training of DL models is computationally expensive, and recomputation aimed at calculating uncertainty bounds might currently be prohibitively expensive.

    “The training of the DL models is computationally expensive, and any further recomputation and re-evaluation of the models, aimed, for example, at the calculation of uncertainty bounds might currently be prohibitively expensive.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    The challenges of applying UQ to machine-assisted medical decision making present significant new research opportunities for new theoretical approaches and practical applications of UQ.

    “These challenges, while not trivial, also present significant new research opportunities for the development of new theoretical approaches, and for the practical applications of UQ in the area of machine-assisted medical decision making.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    It is imperative to understand how the training data must be similar to prediction data.

    “furthermore, it is imperative to understand how the training data must be similar to prediction data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Specific UQ methods need to be developed to quantify the sensitivity of models to imperfect data.

    “Specific UQ methods, therefore, need to be developed to quantify the sensitivity of models to imperfect data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    DL is essentially exploiting correlations in the data, without paying attention to any causal link.

    “DL is essentially exploiting correlations in the data, without paying attention to any causal link.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    To develop consequential decision-support from learned models built on complex data sets, there is an important need to co-develop UQ for this domain.

    “To develop consequential decision-support from”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given

    Unlike statistical approaches where mathematical models explain variations in data and propose margins of error on inferences, recent DL applications combine different learning architectures with a large number of network parameters to form universal approximators.

    “Unlike statistical approaches where mathematical models are used to explain variations observed in data, and to propose the margin of errors on inferences, with these recent applications, different learning architectures are combined with a large number of DL network parameters to form universal approximators.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

    Premisenone given

    Data-driven methods are emerging as the foundations of evidence-based decision making and the future of data-driven scientific discovery.

    “Data-driven methods are emerging as the foundations of evidence-based decision making, and the future of data-driven scientific discovery.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Approaches such as learning generalizable patterns with insertion of random noise, or cross-validation to determine the progression of generalizable learning, move the field forward while still carrying the problem of overfitting.

    “Some approaches, such as attempts to empirically learn generalizable patterns with insertion of random noise 19,20 , or the use of cross-validation to determine the progression of generalizable learning, move us forward in this problem space, while still carrying the problem of overfitting 21 .”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [19] Arpit, D. et al. A closer look at memorization in deep networks. Preprint at https://arxiv.org/abs/1706.05394 (2017).

      Not checked in the free run.

    • cites [20] Zhang, C., Vinyals, O., Munos, R. & Bengio, S. A study on overfitting in deep reinforcement learning. Preprint at http://arxiv.org/abs/1804.06893 (2018).

      Not checked in the free run.

    • cites [21] Cawley, G. C. & Talbot, N. L. C. On over-fitting in model selection and subsequent selection bias in performance evaluation. J. Mach. Learn. Res. 11, 2079–2107 (2010).

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationreasoning

    The wider application of DL in the biomedical field now requires an extension of UQ methods to this emerging field.

    “The wider application of DL in the biomedical field now requires an extension of these methods to this emerging field.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    An effective way of addressing some of the UQ problems might be to re-shape the DL engine itself to provide an uncertainty estimate on its predictions.

    “an effective way of addressing some of the mentioned UQ problems might be to re-shape the DL engine itself to provide an uncertainty estimate on its predictions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    Overfitting, the problem of a model performing well on the training set but generalizing poorly for unseen datasets, is one of the fundamental problems of all data-centric methods, and therefore DL.

    “Overfitting, or the problem of a model performing well on the training set, but generalizing poorly for unseen datasets, is one of the fundamental problems of all data-centric methods, and therefore DL.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

    Interpretationnone given

    The details of medical applications and DL networks are significantly different from other disciplines because the theoretical foundations are far less developed and there are deep psychological and sociological implications in delegating to machines decisions that affect life or health.

    “The details of the medical applications and DL networks are, however, significantly different since the theoretical foundations are far less developed and there are deep psychological and sociological implications in delegating to machines decisions that affect the life or health of human beings.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    Progress in understanding the structure of predictive systems, merging model and data driven approaches with strongly defensible UQ, and formalization of the UQ for DL discipline will be needed to make DL and other data-centric tools and methods practically useful.

    “progress in understanding the structure of these predictive systems, merging model and data driven approaches with strongly defensible UQ, and the formalization of the UQ for DL discipline will be needed to make DL and other data-centric tools and methods practically useful.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    The problem of uncertainty in data-driven predictions has been studied in other scientific disciplines and decision-making arenas, and we can learn from those.

    “These have been studied in other scientific disciplines and decision-making arena, and we can learn from those.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    UQ for DL will likely not be simply a set of tools or procedures to apply, but a more complex wrap-up of disparate methods that in total help bound the overall confidence in predictions.

    “UQ for DL will likely not be simply a set of tools or procedures to apply, but a more complex wrap-up of disparate methods that in total help bound the overall confidence in predictions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given

    In high-dimensional problems typical to the medical setting, every point can be isolated in some other dimension, and a density of points makes sense only after irrelevant dimensions are projected out.

    “In high-dimensional problems, typical to the medical setting (images, large numbers of phenotypes, etc.), every point, however, can be isolated in some other dimension, and a density of points makes sense only after irrelevant dimensions, are projected out”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationcitation

    With DL, given the large number of model parameters and the capacity of DL networks to memorize random noise, classical approaches to evaluating overfitting do not work.

    “With DL, and the large number of model parameters involved as well as the capacity of DL networks to memorize random noise 18 , classical approaches do not work.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [18] Zhang, C., Bengio, S., Hardt, M., Recht, B. & Vinyals, O. Understanding deep learning requires rethinking generalization. Preprint at http://arxiv.org/abs/1611.03530 (2016).

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    Advances in understanding of how DL works internally will allow for a more effective UQ of interpreting DL.

    “Advances in understanding of how DL works internally will allow for a more effective UQ of interpreting DL.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    There is a unifying structure to the problem of understanding precision and uncertainty in purely data-driven predictions across its various guises of complex predictive correlations in large data sets and engineering black boxes.

    “there is a unifying structure to this problem in its various guises of complex predictive correlations in large data sets and engineering black boxes.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    There are ongoing studies that attempt to understand what DL does and how it learns.

    “there are ongoing studies that attempt to understand what DL does, 22 and how it learns 23”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [22] Brahma, P. P., Wu, D. & She, Y. Why deep learning works: A manifold disentanglement perspective. IEEE Trans. Neural Netw. Learn. Sys. 27, 1997–2008 (2016). source

      Not checked in the free run.

    • cites [23] Raghu, M., Gilmer, J., Yosinski, J. & Sohl-Dickstein, J. SVCCA: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Preprint at https://arxiv.org/abs/1706.05806 (2017).

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    The DL network itself can be used to study uncertainty empirically and provide uncertainty bounds.

    “the network itself can be used to study this uncertainty empirically and provide the uncertainty bounds.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationreasoning

    It is possible that the entire new field of UQ for DL might need to be developed.

    “In the end, it is possible that the entire new field of UQ for DL might need to be developed.”
    Quote in the paperNot a substantial claim
48 of 48

  • The paper relies on this without support.

    The reluctance to delegate to machine intelligence a function of life-critical and patient-safety-related decision making is a primary reason for the slow adoption of AI in medicine.

    “There are many reasons for this, primarily the reluctance to delegate to machine intelligence a function of life-critical and patient-safety-related decision making.”

  • The authors state this as an assumption.

    Fair sampling in the training data holds.

    “assuming fair sampling in the training data”
2 of 2

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • AI, ML, and DL can enable robots to make better decisions based on data analysis and pattern recognition, leading to improved performance and outcomes.

    “Better decision-making: AI, ML, and DL can enable robots to make better decisions based on data analysis and pattern recognition, leading to improved performance and outcomes [52] .”

    From Artificial intelligence, machine learning and deep learning in advanced robotics, a review (2023) · cites this paper as [52]

    Section 2.1 reports that AI and deep learning recognize patterns in medical images and that some predictive models outperform classical approaches. It does not address robots or show that robots make better decisions or achieve improved outcomes. Section 2 also cautions that adopting automated decision-making in medicine requires uncertainty quantification. Page numbers are unavailable in our copy. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • DL models have a black-box nature and lack an underlying theoretical foundation behind their decisions.

    “However, these models have black-box nature and lack underlying theoretical foundation behind their decisions [8]”

    From Explainable, trustworthy, and ethical machine learning for healthcare: A survey (2022) · cites this paper as [8]

  • Due to the unavailability of sound fundamental laws and models, the explanation of DL cannot be produced up to the required scale.

    “However, due to the unavailability of sound fundamental laws and models, the explanation of DL cannot be produced up-to the required scale.”

    From Explainable, trustworthy, and ethical machine learning for healthcare: A survey (2022) · cites this paper as [8]

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Begoli, E., Bhattacharya, T., & Kusnezov, D.
Affiliations
  • Oak Ridge National Laboratory (US)
  • United States Department of Energy (US)
Corresponding author
Begoli, Edmon
ORCID
Begoli, Edmon
Venue
Nature Machine Intelligence
Publisher
Springer Science and Business Media LLC
Published
2019-01-07
Volume
1
Issue
1
Pages
20-23
ISSN
2522-5839 (ISSN-L)
DOI
10.1038/s42256-018-0004-1
OpenAlex
W2904422658
Semantic Scholar
a73eef4e0ec8e49be1c0e6abb04cc1967a681c50
Cited by
597
References
22
Citations by year
2026: 158 · 2025: 102 · 2024: 73 · 2023: 94 · 2022: 72 · 2021: 47 · 2020: 33 · 2019: 16 · 2018: 2
FWCI
6.47
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Machine Intelligence (publishedVersion)
  • OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) (submittedVersion)
Funders
  • Battelle (DE-AC05- 00OR22725, DE-AC05)
  • UT-Battelle (DE-AC05, DEAC05-00OR22725, AC05-00OR22725)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Keywords
uncertainty quantification, artificial intelligence in healthcare, deep learning, patient safety, trust in AI, forecast uncertainty

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review