Uncertainty modelling in deep learning for safer neuroimage enhancement: Demonstration in diffusion MRI
Open accessThe DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
Abstract
Deep learning (DL) has shown great potential in medical image enhancement problems, such as super-resolution or image synthesis. However, to date, most existing approaches are based on deterministic models, neglecting the presence of different sources of uncertainty in such problems. Here we introduce methods to characterise different components of uncertainty, and demonstrate the ideas using diffusion MRI super-resolution. Specifically, we propose to account for intrinsic uncertainty through a heteroscedastic noise model and for parameter uncertainty through approximate Bayesian inference, and integrate the two to quantify predictive uncertainty over the output image. Moreover, we introduce a method to propagate the predictive uncertainty on a multi-channelled image to derived scalar parameters, and separately quantify the effects of intrinsic and parameter uncertainty therein. The methods are evaluated for super-resolution of two different signal representations of diffusion MR images-Diffusion Tensor images and Mean Apparent Propagator MRI-and their derived quantities such as mean diffusivity and fractional anisotropy, on multiple datasets of both healthy and pathological human brains. Results highlight three key potential benefits of modelling uncertainty for improving the safety of DL-based image enhancement systems. Firstly, modelling uncertainty improves the predictive performance even when test data departs from training data ("out-of-distribution" datasets). Secondly, the predictive uncertainty highly correlates with reconstruction errors, and is therefore capable of detecting predictive "failures". Results on both healthy subjects and patients with brain glioma or multiple sclerosis demonstrate that such an uncertainty measure enables subject-specific and voxel-wise risk assessment of the super-resolved images that can be accounted for in subsequent analysis. Thirdly, we show that the method for decomposing predictive uncertainty into its independent sources provides high-level "explanations" for the model performance by separately quantifying how much uncertainty arises from the inherent difficulty of the task or the limited training examples. The introduced concepts of uncertainty modelling extend naturally to many other imaging modalities and data enhancement applications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Image quality assessment: from error visibility to structural similarity
Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004) · IEEE Transactions on Image Processing · doi:10.1109/tip.2003.819861 · Cited by 57,892
A survey on deep learning in medical image analysis
Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A. W. M., van Ginneken, B., & Sánchez, C. I. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.07.005 · Cited by 15,332
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883
Perceptual Losses for Real-Time Style Transfer and Super-Resolution
Johnson, J., Alahi, A., & Fei-Fei, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46475-6_43 · Cited by 11,283
Image Super-Resolution Using Deep Convolutional Networks
Dong, C., Loy, C. C., He, K., & Tang, X. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2439281 · Cited by 10,105
Overcoming catastrophic forgetting in neural networks
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. (2017) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1611835114 · Cited by 7,899
MR diffusion tensor spectroscopy and imaging
Basser, P. J., Mattiello, J., & LeBihan, D. (1994) · Biophysical Journal · doi:10.1016/s0006-3495(94)80775-1 · Cited by 5,970
The WU-Minn Human Connectome Project: An overview
Van Essen, D. C., Smith, S. M., Barch, D. M., Behrens, T. E. J., Yacoub, E., & Ugurbil, K. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.05.041 · Cited by 5,651
The minimal preprocessing pipelines for the Human Connectome Project
Glasser, M. F., Sotiropoulos, S. N., Wilson, J. A., Coalson, T. S., Fischl, B., Andersson, J. L., Xu, J., Jbabdi, S., Webster, M., Polimeni, J. R., et al. (2013) · NeuroImage · doi:10.1016/j.neuroimage.2013.04.127 · Cited by 5,253
Self-citationSimple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2016) · arXiv · doi:10.48550/arxiv.1612.01474 · Cited by 3,666 · cited 2×
“An alternative approach is ensem-bling where the variance of the predictions of multiple networks is used to quantify the predictive uncertainty ( Lakshminarayanan et al., 2017 ).”
MethodEfficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation
Kamnitsas, K., Ledig, C., Newcombe, V. F. J., Simpson, J. P., Kane, A. D., Menon, D. K., Rueckert, D., & Glocker, B. (2017) · Medical Image Analysis · doi:10.1016/j.media.2016.10.004 · Cited by 3,498
What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
Kendall, A., & Gal, Y. (2017) · arXiv · doi:10.48550/arxiv.1703.04977 · Cited by 3,088 · cited 2×
“…failures of deep learning systems, by and large, occur due to two reasons: (i) the task itself is inherently ambiguous or (ii) the learned model is not adequate to describe the data ( Der Kiureghian and Ditlevsen, 2009; Hora, 1996; Kendall and Gal, 2017; Tanno et al., 2017 ), as illustrated in Fig.”
BackgroundSemantic Image Synthesis With Spatially-Adaptive Normalization
Park, T., Liu, M.-Y., Wang, T.-C., & Zhu, J.-Y. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00244 · Cited by 2,974 · cited 1×
“However, controllable and realistic means to edit input images ( Clatz et al., 2005; Park et al., 2019; Prastawa et al., 2009 ) (e.g., simulation of pathological structures of different controllable parameters such as size and shape) would allow ones to study in a systematic fashion what kinds of…”
BackgroundDeep Convolutional Neural Network for Inverse Problems in Imaging
Jin, K. H., McCann, M. T., Froustey, E., & Unser, M. (2017) · IEEE Transactions on Image Processing · doi:10.1109/tip.2017.2713099 · Cited by 2,529
Aleatory or epistemic? Does it matter?
Kiureghian, A. D., & Ditlevsen, O. (2009) · Structural Safety · doi:10.1016/j.strusafe.2008.06.020 · Cited by 2,454 · cited 2×
“Predictive failures of deep learning systems, by and large, occur due to two reasons: (i) the task itself is inherently ambiguous or (ii) the learned model is not adequate to describe the data ( Der Kiureghian and Ditlevsen, 2009; Hora, 1996; Kendall and Gal, 2017; Tanno et al., 2017 ), as illustrated in Fig.”
BackgroundImage reconstruction by domain-transform manifold learning
Zhu, B., Liu, J. Z., Cauley, S. F., Rosen, B. R., & Rosen, M. S. (2018) · Nature · doi:10.1038/nature25988 · Cited by 1,694
Assessment and Propagation of Model Uncertainty
Draper, D. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02015.x · Cited by 1,599
Magnetic resonance fingerprinting
Ma, D., Gulani, V., Seiberlich, N., Liu, K., Sunshine, J. L., Duerk, J. L., & Griswold, M. A. (2013) · Nature · doi:10.1038/nature11971 · Cited by 1,577
Diverse Image-to-Image Translation via Disentangled Representations
Lee, H.-Y., Tseng, H.-Y., Huang, J.-B., Singh, M., & Yang, M.-H. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01246-5_3 · Cited by 1,541
MRtrix: Diffusion tractography in crossing fiber regions
Tournier, J., Calamante, F., & Connelly, A. (2012) · International Journal of Imaging Systems and Technology · doi:10.1002/ima.22005 · Cited by 1,370
A deep cascade of convolutional neural networks for dynamic MR image reconstruction
Schlemper, J., Caballero, J., Hajnal, J. V., Price, A. N., & Rueckert, D. (2018) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2017.2760978 · Cited by 1,338
Diffusion tensor imaging of white matter tract evolution over the lifespan
Lebel, C., Gee, M., Camicioli, R., Wieler, M., Martin, W., & Beaulieu, C. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2011.11.094 · Cited by 1,100
Deep Bayesian Active Learning with Image Data
Gal, Y., Islam, R., & Ghahramani, Z. (2017) · arXiv · doi:10.48550/arxiv.1703.02910 · Cited by 900
Life-Span Changes of the Human Brain White Matter: Diffusion Tensor Imaging (DTI) and Volumetry
Westlye, L. T., Walhovd, K. B., Dale, A. M., Bjornerud, A., Due-Tonnessen, P., Engvig, A., Grydeland, H., Tamnes, C. K., Ostby, Y., & Fjell, A. M. (2010) · Cerebral Cortex · doi:10.1093/cercor/bhp280 · Cited by 765
Oktay, O., Ferrante, E., Kamnitsas, K., Heinrich, M., Bai, W., Caballero, J., Cook, S. A., de Marvao, A., Dawes, T., O'Regan, D. P., et al. (2018) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2017.2743464 · Cited by 751
The need for uncertainty quantification in machine-assisted medical decision making
Begoli, E., Bhattacharya, T., & Kusnezov, D. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-018-0004-1 · Cited by 597
Leveraging uncertainty information from deep neural networks for disease detection
Leibig, C., Allken, V., Ayhan, M. S., Berens, P., & Wahl, S. (2017) · Scientific Reports · doi:10.1038/s41598-017-17876-z · Cited by 510
Variational Continual Learning
Nguyen, C. V., Li, Y., Bui, T. D., & Turner, R. E. (2017) · arXiv · doi:10.48550/arxiv.1710.10628 · Cited by 491 · cited 1×
“…provides a natu-ral framework to use uncertainty in order to account for the difference and commonality between distributions to guide information transfer in continual learning ( Kirkpatrick et al., 2017; Nguyen et al., 2018 ) or few-shot learning ( Finn et al., 2018; Yoon et al., 2018 ) settings.”
MethodNair, T., Precup, D., Arnold, D. L., & Arbel, T. (2020) · Medical Image Analysis · doi:10.1016/j.media.2019.101557 · Cited by 451
A probabilistic u-net for segmentation of ambiguous images
Kohl, S. A. A., Romera-Paredes, B., Meyer, C., De Fauw, J., Ledsam, J. R., Maier-Hein, K. H., Eslami, S. M. A., Rezende, D. J., & Ronneberger, O. (2018) · arXiv · doi:10.48550/arxiv.1806.05034 · Cited by 439 · cited 3×
“Future work should explore the benefits of employing more complex forms of likelihood functions such as mixture models ( Bishop, 1994; Kohl et al., 2018 ), diversity losses ( Bouchacourt et al., 2016; Guzman-Rivera et al., 2012; Lee et al., 2018 ) and more powerful density estimators ( Huang et al.,…”
BackgroundMethodVariational Continual Learning
Turner, R., Bui, T., Li, Y., & Cuong, N. (2019) · Apollo - University of Cambridge Repository · doi:10.17863/cam.35471 · Cited by 438
Probabilistic Model-Agnostic Meta-Learning
Finn, C., Xu, K., & Levine, S. (2018) · arXiv · doi:10.48550/arxiv.1806.02817 · Cited by 327
Hora, S. C. (1996) · Reliability Engineering & System Safety · doi:10.1016/s0951-8320(96)00077-4 · Cited by 249
Multi-input Cardiac Image Super-Resolution Using Convolutional Neural Networks
Oktay, O., Bai, W., Lee, M., Guerrero, R., Kamnitsas, K., Caballero, J., de Marvao, A., Cook, S., O’Regan, D., & Rueckert, D. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46726-9_29 · Cited by 246
FUIQA: Fetal Ultrasound Image Quality Assessment with Deep Convolutional Networks
Wu, L., Cheng, J.-Z., Li, S., Lei, B., Wang, T., & Ni, D. (2017) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2017.2671898 · Cited by 242 · cited 1×
“( Hammernik et al., 2018; Jin et al., 2017; Schlemper et al., 2018a; Sun et al., 2016; Yang et al., 2018; Yoon et al., 2019; Zhu et al., 2018 ), registration ( Balakrishnan et al., 2018; Sokooti et al., 2017 ) and quality control ( Esses et al., 2018; Wu et al., 2017 ).”
Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors
Louizos, C., & Welling, M. (2016) · arXiv · doi:10.48550/arxiv.1603.04733 · Cited by 138 · cited 1×
“More recent advances in the Bayesian deep learning research ( Louizos and Welling, 2016, 2017; Oh et al., 2020; Pawlowski et al., 2017; Zhang et al., 2019; Krueger et al., 2017 ) could be used to enhance the quality of parameter uncertainty estimation by allowing the model to capture multi-modality…”
BackgroundRoy, A. G., Conjeti, S., Navab, N., & Wachinger, C. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.03.042 · Cited by 137
Fast Predictive Image Registration
Yang, X., Kwitt, R., & Niethammer, M. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46976-8_6 · Cited by 127
Supervised uncertainty quantification for segmentation with multiple annotations
Hu, S., Worrall, D., Knegt, S., Veeling, B., Huisman, H., & Welling, M. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32245-8_16 · Cited by 105
Self-citationBenou, A., Veksler, R., Friedman, A., & Riklin Raviv, T. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.07.006 · Cited by 91
Figini, M., Riva, M., Graham, M., Castelli, G. M., Fernandes, B., Grimaldi, M., Baselli, G., Pessina, F., Bello, L., Zhang, H., et al. (2018) · Radiology · doi:10.1148/radiol.2018180054 · Cited by 49
Self-citationPHiSeg: Capturing Uncertainty in Medical Image Segmentation
Baumgartner, C. F., Tezcan, K. C., Chaitanya, K., Hötker, A. M., Muehlematter, U. J., Schawkat, K., Becker, A. S., Donati, O., & Konukoglu, E. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32245-8_14 · Cited by 34
As Easy as 1, 2...4? Uncertainty in Counting Tasks for Medical Imaging
Eaton-Rosen, Z., Varsavsky, T., Ourselin, S., & Cardoso, M. J. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32251-9_39 · Cited by 14
Cited by
A review of uncertainty quantification in deep learning: Techniques, applications and challenges
Abdar, M., Pourpanah, F., Hussain, S., Rezazadegan, D., Liu, L., Ghavamzadeh, M., Fieguth, P., Cao, X., Khosravi, A., Acharya, U. R., et al. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.008 · Cited by 3,111
A review of medical image data augmentation techniques for deep learning applications
Chlap, P., Min, H., Vandenberg, N., Dowling, J., Holloway, L., & Haworth, A. (2021) · Journal of Medical Imaging and Radiation Oncology · doi:10.1111/1754-9485.13261 · Cited by 956
Uncertainty quantification in scientific machine learning: Methods, metrics, and comparisons
Psaros, A. F., Meng, X., Zou, Z., Guo, L., & Karniadakis, G. E. (2023) · Journal of Computational Physics · doi:10.1016/j.jcp.2022.111902 · Cited by 455
The Human Connectome Project: A retrospective
Elam, J. S., Glasser, M. F., Harms, M. P., Sotiropoulos, S. N., Andersson, J. L. R., Burgess, G. C., Curtiss, S. W., Oostenveld, R., Larson-Prior, L. J., Schoffelen, J.-M., et al. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2021.118543 · Cited by 307
Self-citationChen, H., Gomez, C., Huang, C.-M., & Unberath, M. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00699-2 · Cited by 300
Metadata
- Authors
- Tanno, R., Worrall, D. E., Kaden, E., Ghosh, A., Grussu, F., Bizzi, A., Sotiropoulos, S. N., Criminisi, A., & Alexander, D. C.
- Venue
- NeuroImage
- Publisher
- Elsevier BV
- Published
- 2021-01
- Volume
- 225
- Pages
- 117366
- ISSN
- 1053-8119 (ISSN-L)
- OpenAlex
- W3092596753
- Cited by
- 113
- References
- 44
- FWCI
- 6.23
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.