Quantifying uncertainty in deep learning approaches to radio galaxy classification
Open accessThe DOI matches its Crossref record; 48 of the 48 references it cites were found in a registry or an index.
Abstract
In this work we use variational inference to quantify the degree of uncertainty in deep learning model predictions of radio galaxy classification. We show that the level of model posterior variance for individual test samples is correlated with human uncertainty when labelling radio galaxies. We explore the model performance and uncertainty calibration for different weight priors and suggest that a sparse prior produces more well-calibrated uncertainty estimates. Using the posterior distributions for individual weights, we demonstrate that we can prune 30 per cent of the fully connected layer weights without significant loss of performance by removing the weights with the lowest signal-to-noise ratio. A larger degree of pruning can be achieved using a Fisher information based ranking, but both pruning methods affect the uncertainty calibration for Fanaroff–Riley type I and type II radio galaxies differently. Like other work in this field, we experience a cold posterior effect, whereby the posterior must be down-weighted to achieve good predictive performance. We examine whether adapting the cost function to accommodate model misspecification can compensate for this effect, but find that it does not make a significant difference. We also examine the effect of principled data augmentation and find that this improves upon the baseline but also does not compensate for the observed effect. We interpret this as the cold posterior effect being due to the overly effective curation of our training sample leading to likelihood misspecification, and raise this as a potential issue for Bayesian deep learning approaches to radio galaxy classification in future.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
On Information and Sufficiency
Kullback, S., & Leibler, R. A. (1951) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729694 · Cited by 20,193
Advances in Neural Information Processing Systems 14: Proceedings of the 2001 Conference
(2002) · The MIT Press eBooks · doi:10.7551/mitpress/1120.001.0001 · Cited by 8,890
Condon, J. J., Cotton, W. D., Greisen, E. W., Yin, Q. F., Perley, R. A., Taylor, G. B., & Broderick, J. J. (1998) · The Astronomical Journal · doi:10.1086/300337 · Cited by 5,760
THE SEVENTH DATA RELEASE OF THE SLOAN DIGITAL SKY SURVEY
Abazajian, K. N., Adelman-McCarthy, J. K., Agüeros, M. A., Allam, S. S., Prieto, C. A., An, D., Anderson, K. S. J., Anderson, S. F., Annis, J., Bahcall, N. A., et al. (2009) · The Astrophysical Journal Supplement Series · doi:10.1088/0067-0049/182/2/543 · Cited by 5,341
Valiant, L. G. (1984) · Communications of the ACM · doi:10.1145/1968.1972 · Cited by 3,279
A practical Bayesian framework for backpropagation networks
MacKay, D. J. C. (1992) · Neural Computation · doi:10.1162/neco.1992.4.3.448 · Cited by 2,917
The FIRST Survey: Faint Images of the Radio Sky at Twenty Centimeters
Becker, R. H., White, R. L., & Helfand, D. J. (1995) · The Astrophysical Journal · doi:10.1086/176166 · Cited by 2,911
Advances in Neural Information Processing Systems 37
(2024) · doi:10.52202/079017 · Cited by 2,791
The Morphology of Extragalactic Radio Sources of High and Low Luminosity
Fanaroff, B. L., & Riley, J. M. (1974) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/167.1.31p · Cited by 2,759
LOFAR: The LOw-Frequency ARray
van Haarlem, M. P., Wise, M. W., Gunst, A. W., Heald, G., McKean, J. P., Hessels, J. W. T., de Bruyn, A. G., Nijboer, R., Swinbank, J., Fallows, R., et al. (2013) · Astronomy & Astrophysics · doi:10.1051/0004-6361/201220873 · Cited by 2,662
Proceedings of the 23rd international conference on Machine learning
Cohen, W. W., & Moore, A. (2006) · Cited by 2,647
Aleatory or epistemic? Does it matter?
Kiureghian, A. D., & Ditlevsen, O. (2009) · Structural Safety · doi:10.1016/j.strusafe.2008.06.020 · Cited by 2,454
A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants
Neal, R. M., & Hinton, G. E. (1998) · Learning in Graphical Models · doi:10.1007/978-94-011-5014-9_12 · Cited by 2,187
International Conference on Artificial Neural Networks
(1990) · Neural Networks · doi:10.1016/0893-6080(90)90065-s · Cited by 2,088
Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods
Hüllermeier, E., & Waegeman, W. (2021) · Machine Learning · doi:10.1007/s10994-021-05946-3 · Cited by 1,834
Keeping the neural networks simple by minimizing the description length of the weights
Hinton, G. E., & van Camp, D. (1993) · Proceedings of the sixth annual conference on Computational learning theory - COLT '93 · doi:10.1145/168304.168306 · Cited by 994
Proceedings of the 28th International Conference on Neural Information Processing Systems
Cortes, C., Lee, D., Sugiyama, M., & Garnett, R. (2015) · Cited by 747
The Evidence Framework Applied to Classification Networks
MacKay, D. J. C. (1992) · Neural Computation · doi:10.1162/neco.1992.4.5.720 · Cited by 722
Best, P. N., & Heckman, T. M. (2012) · Monthly Notices of the Royal Astronomical Society · doi:10.1111/j.1365-2966.2012.20414.x · Cited by 680
Hein, M., Andriushchenko, M., & Bitterwolf, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00013 · Cited by 428
Science with ASKAP: The Australian square-kilometre-array pathfinder
Johnston, S., Taylor, R., Bailes, M., Bartel, N., Baugh, C., Bietenholz, M., Blake, C., Braun, R., Brown, J., Chatterjee, S., et al. (2008) · Experimental Astronomy · doi:10.1007/s10686-008-9124-7 · Cited by 409
Mean Field Theory for Sigmoid Belief Networks
Saul, L. K., Jaakkola, T., & Jordan, M. I. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.251 · Cited by 327
Measuring Distribution Similarities Between Samples: A Distribution-Free Overlapping Index
Pastore, M., & Calcagnì, A. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.01089 · Cited by 310
Schoenmakers, A. P., de Bruyn, A. G., Rottgering, H. J. A., van der Laan, H., & Kaiser, C. R. (2000) · Monthly Notices of the Royal Astronomical Society · doi:10.1046/j.1365-8711.2000.03430.x · Cited by 305
Mingo, B., Croston, J. H., Hardcastle, M. J., Best, P. N., Duncan, K. J., Morganti, R., Rottgering, H. J. A., Sabater, J., Shimwell, T. W., Williams, W. L., et al. (2019) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stz1901 · Cited by 212
Relativistic Jets and the Fanaroff-Riley Classification of Radio Galaxies
Bicknell, G. V. (1995) · The Astrophysical Journal Supplement Series · doi:10.1086/192232 · Cited by 187
Classifying Radio Galaxies with the Convolutional Neural Network
Aniyan, A. K., & Thorat, K. (2017) · The Astrophysical Journal Supplement Series · doi:10.3847/1538-4365/aa7333 · Cited by 162
McAllester, D. A. (1999) · Machine Learning · doi:10.1023/a:1007618624809 · Cited by 162
Particle content, radio-galaxy morphology, and jet power: all radio-loud AGN are not equal
Croston, J. H., Ineson, J., & Hardcastle, M. J. (2018) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/sty274 · Cited by 146
Luminosity function, sizes and FR dichotomy of radio-loud AGN
Kaiser, C., & Best, P. N. (2007) · Monthly Notices of the Royal Astronomical Society · doi:10.1111/j.1365-2966.2007.12350.x · Cited by 140
Radio Galaxy Zoo: host galaxies and radio morphologies derived from visual inspection
Banfield, J. K., Wong, O. I., Willett, K. W., Norris, R. P., Rudnick, L., Shabala, S. S., Simmons, B. D., Snyder, C., Garon, A., Seymour, N., et al. (2015) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stv1688 · Cited by 135
The MeerKAT International GHz Tiered Extragalactic Exploration (MIGHTEE) Survey
Jarvis, M., Taylor, R., Agudo, I., Allison, J. R., Deane, R. P., Frank, B., Gupta, N., Heywood, I., Maddox, N., McAlpine, K., et al. (2018) · Proceedings of MeerKAT Science: On the Pathway to the SKA — PoS(MeerKAT2016) · doi:10.22323/1.277.0006 · Cited by 125
Self-citationDeep Deterministic Uncertainty: A New Simple Baseline
Mukhoti, J., Kirsch, A., van Amersfoort, J., Torr, P. H. S., & Gal, Y. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02336 · Cited by 116
Radio Galaxy Zoo:Claran– a deep learning classifier for radio morphologies
Wu, C., Wong, O. I., Rudnick, L., Shabala, S. S., Alger, M. J., Banfield, J. K., Ong, C. S., White, S. V., Garon, A. F., Norris, R. P., et al. (2019) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/sty2646 · Cited by 100
Head-tail radio sources in clusters of galaxies
Rudnick, L., & Owen, F. N. (1976) · The Astrophysical Journal · doi:10.1086/182030 · Cited by 96
Bayesian prediction of tissue-regulated splicing using RNA sequence and cellular context
Xiong, H. Y., Barash, Y., & Frey, B. J. (2011) · Bioinformatics · doi:10.1093/bioinformatics/btr444 · Cited by 96
Radio Galaxy Zoo: compact and extended radio source classification with deep learning
Lukic, V, Brüggen, M., Banfield, J. K., Wong, O. I., Rudnick, L., Norris, R. P., & Simmons, B. (2018) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/sty163 · Cited by 86
Miraghaei, H., & Best, P. N. (2017) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stx007 · Cited by 84
VLA observations of 57 sources in clusters of galaxies
O’Dea, C. P., & Owen, F. N. (1985) · The Astronomical Journal · doi:10.1086/113801 · Cited by 77
A Bayesian neural network for toxicity prediction
Semenova, E., Williams, D. P., Afzal, A. M., & Lazic, S. E. (2020) · Computational Toxicology · doi:10.1016/j.comtox.2020.100133 · Cited by 57
(2019) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stz1883 · Cited by 45
Self-citationScaife, A. M. M., & Porter, F. (2021) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stab530 · Cited by 44
Self-citationScience with the Murchison Widefield Array: Phase I results and Phase II opportunities
Beardsley, A. P., Johnston-Hollitt, M., Trott, C. M., Pober, J. C., Morgan, J., Oberoi, D., Kaplan, D. L., Lynch, C. R., Anderson, G. E., McCauley, P. I., et al. (2019) · Publications of the Astronomical Society of Australia · doi:10.1017/pasa.2019.41 · Cited by 44
Attention-gating for improved radio galaxy classification
Bowles, M., Scaife, A. M. M., Porter, F., Tang, H., & Bastien, D. J. (2021) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/staa3946 · Cited by 42
Self-citationReducing the Model Order of Deep Neural Networks Using Information Theory
Tu, M., Berisha, V., Cao, Y. K., & Seo, J.-S. (2016) · IEEE Computer Society Annual Symposium on VLSI · doi:10.1109/isvlsi.2016.117 · Cited by 27
Rapid sorting of radio galaxy morphology using Haralick features
Ntwaetsile, K., & Geach, J. E. (2021) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stab271 · Cited by 18
Structured variational inference for simulating populations of radio galaxies
Bastien, D. J., Scaife, A. M. M., Tang, H., Bowles, M., & Porter, F. (2021) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stab588 · Cited by 17
Self-citation
Research works printed but not linked: 20
- Abdar, 2021, Inf. Fusion
- Aitchison, 2021, International Conference on Learning Representations
- Alquier, 2016, J. Mach. Learn. Res., 17, 8374
- Blundell, 2015, 32nd International Conference on International Conference on Machine Learning - Vol. 37, 1613
- Gal, 2016, Uncertainty in Deep Learning
- Gal, 2016, International Conference on Learning Representations
- 2000, A&A, 363, 507
- Graves, 2011, Proceedings of the 24th International Conference on Neural Information Processing Systems. NIPS’11, 2348
- Guo, 2017, International Conference on Machine Learning, 1321
- Guo, 2017, Proceedings of the 34th International Conference on Machine Learning - Vol. 70. ICML’17, 1321
- Izmailov, 2021, Proceedings of Machine Learning Research Vol. 139, Proceedings of the 38th International Conference on Machine Learning, 4629
- Kingma, 2015, Adv. Neural Inf. Process. Syst., 28, 2575
- Krishnan, 2020, Adv. in Neural Info. Proc. Syst., 18237
- Kunstner, 2019, Vol. 32, Advances in Neural Information Processing Systems
- LeCun, 1989, Advances in Neural Information Processing Systems, 598
- Masegosa, 2019, Adv. in Neural Info. Proc. Syst., 5479
- Nguyen, 2015, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 427
- Peterson, 2019, Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 9617
- Shalaeva, 2020, Proceedings of the AAAI Conference on Artificial Intelligence, 5660
- Wenzel, 2020, Proceedings of Machine Learning Research Vol. 119, Proceedings of the 37th International Conference on Machine Learning, 10248
Cited by
Slijepcevic, I. V., Scaife, A. M. M., Walmsley, M., Bowles, M., Wong, O. I., Shabala, S. S., & White, S. V. (2023) · RAS Techniques and Instruments · doi:10.1093/rasti/rzad055 · Cited by 54
Self-citationAngeloudi, E., Audenaert, J., Bowles, M., Boyd, B. M., Chemaly, D., Cherinka, B., Ciucă, I., Cranmer, M., Do, A., Grayling, M., et al. (2024) · neural information processing systems · doi:10.52202/079017-1845 · Cited by 48
Self-citationSlijepcevic, I. V., Scaife, A. M. M., Walmsley, M., Bowles, M., Wong, O. I., Shabala, S. S., & Tang, H. (2022) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stac1135 · Cited by 44
Self-citationInferring astrophysics and dark matter properties from 21 cm tomography using deep learning
Neutsch, S., Heneka, C. S., & Brüggen, M. (2022) · Monthly Notices of the Royal Astronomical Society · doi:10.1093/mnras/stac218 · Cited by 38
MiraBest: a data set of morphologically classified radio galaxies for machine learning
Porter, F., & Scaife, A. M. M. (2023) · RAS Techniques and Instruments · doi:10.1093/rasti/rzad017 · Cited by 38
Self-citation
Metadata
- Authors
- Mohan, D., Scaife, A. M. M., Porter, F., Walmsley, M., & Bowles, M.
- Affiliations
- University of Manchester (GB)
- Corresponding author
- Mohan, Devina
- Venue
- Monthly Notices of the Royal Astronomical Society
- Publisher
- Oxford University Press (OUP)
- Published
- 2022-01-29
- Volume
- 511
- Issue
- 3
- Pages
- 3722-3740
- ISSN
- 0035-8711 (ISSN-L)
- 1365-2966
- OpenAlex
- W4210818292
- Cited by
- 16
- References
- 48
- Citations by year
- 2026: 3 · 2025: 3 · 2024: 1 · 2023: 5 · 2022: 4
- FWCI
- 1.30
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Monthly Notices of the Royal Astronomical Society (publishedVersion, cc-by)
- Research Explorer (The University of Manchester) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Strathprints: The University of Strathclyde institutional repository (University of Strathclyde) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Alan Turing Institute (EP/V030302/1)
- STFC
- IBM (ST/P006795/1)
- Topics
- Physical Sciences › Physics and Astronomy › Astronomy and Astrophysics › Galaxies: Formation, Evolution, Phenomena
- Physical Sciences › Physics and Astronomy › Astronomy and Astrophysics › Radio Astronomy Observations and Technology
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- radio galaxy classification, uncertainty quantification, variational inference, Bayesian deep learning, uncertainty calibration, weight pruning, signal-to-noise ratio, Fisher information, data augmentation, model misspecification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.