Uncertainty-Aware COVID-19 Detection from Imbalanced Sound Data
Open accessAbstract
Recently, sound-based COVID-19 detection studies have shown great promise to achieve scalable and prompt digital prescreening.However, there are still two unsolved issues hindering the practice.First, collected datasets for model training are often imbalanced, with a considerably smaller proportion of users tested positive, making it harder to learn representative and robust features.Second, deep learning models are generally overconfident in their predictions.Clinically, false predictions aggravate healthcare costs.Estimation of the uncertainty of screening would aid this.To handle these issues, we propose an ensemble framework where multiple deep learning models for sound-based COVID-19 detection are developed from different but balanced subsets from original data.As such, data are utilized more effectively compared to traditional up-sampling and down-sampling approaches: an AUC of 0.74 with a sensitivity of 0.68 and a specificity of 0.69 is achieved.Simultaneously, we estimate uncertainty from the disagreement across multiple models.It is shown that false predictions often yield higher uncertainty, enabling us to suggest the users with certainty higher than a threshold to repeat the audio test on their phones or to take clinical tests if digital diagnosis still fails.This study paves the way for a more robust sound-based COVID-19 automated screening system.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Uncertainty-aware modelling approaches enable prediction of uncertainty associated with final outcome predictions, providing confidence levels for users to trust or distrust systems.
“Another potential solution for enhancing interpretability is adopting uncertainty-aware modelling approaches. These enable the prediction of uncertainty associated with the final outcome predictions, thereby providing confidence levels for users to either trust or distrust the systems.”
From Human-centred artificial intelligence for mobile health sensing: challenges and opportunities (2023) · cites this paper as [185]
Ensemble learning for sound-based COVID-19 detection can infer uncertainty of model predictions, and false predictions often yield higher uncertainty, enabling suggestions to users with certainty higher than a threshold to repeat audio test or take clinical tests if digital diagnosis fails.
“One study [185] employed ensemble learning for sound-based COVID-19 detection to infer the uncertainty of the model predictions and showed that false predictions often yield higher uncertainty, enabling us to suggest to users with certainty higher than a threshold to repeat the audio test on their phones or to take clinical tests if digital diagnosis still fails.”
From Human-centred artificial intelligence for mobile health sensing: challenges and opportunities (2023) · cites this paper as [185]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
SMOTE synthetic minority over-sampling technique
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865 · see citing works
Clinical characteristics of coronavirus disease 2019 in China
Guan, W.-J., Ni, Z.-Y., Hu, Y., Liang, W.-H., Ou, C.-Q., He, J.-X., Liu, L., Shan, H., Lei, C.-L., Hui, D. S. C., et al. (2020) · New England Journal of Medicine · doi:10.1056/nejmoa2002032 · Cited by 30,246 · see citing works
Clinical Characteristics of Coronavirus Disease 2019 in China
Eastin, C., & Eastin, T. (2020) · Journal of Emergency Medicine · doi:10.1016/j.jemermed.2020.04.004 · Cited by 24,945 · see citing works
Dropout as a bayesian approximation: Representing model uncertainty in deep learning
Gal, Y., & Ghahramani, Z. (2015) · arXiv · doi:10.48550/arxiv.1506.02142 · Cited by 6,471 · see citing works
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
Lakshminarayanan, B., Pritzel, A., & Blundell, C. (2016) · arXiv · doi:10.48550/arxiv.1612.01474 · Cited by 3,666 · see citing works
On calibration of modern neural networks
Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017) · arXiv · doi:10.48550/arxiv.1706.04599 · Cited by 3,435 · see citing works
The class imbalance problem: A systematic study1
Japkowicz, N., & Stephen, S. (2002) · Intelligent Data Analysis · doi:10.3233/ida-2002-6504 · Cited by 3,295 · see citing works
CNN architectures for large-scale audio classification
Hershey, S., Chaudhuri, S., Ellis, D. P. W., Gemmeke, J. F., Jansen, A., Moore, R. C., Plakal, M., Platt, D., Saurous, R. A., Seybold, B., et al. (2017) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2017.7952132 · Cited by 2,604 · see citing works
Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J. V., Lakshminarayanan, B., & Snoek, J. (2019) · arXiv · doi:10.48550/arxiv.1906.02530 · Cited by 989 · see citing works
Analytical sensitivity and efficiency comparisons of SARS-CoV-2 RT–qPCR primer–probe sets
Vogels, C. B. F., Brito, A. F., Wyllie, A. L., Fauver, J. R., Ott, I. M., Kalinich, C. C., Petrone, M. E., Casanovas‐Massana, A., Muenker, M. C., Moore, A. J., et al. (2020) · Nature Microbiology · doi:10.1038/s41564-020-0761-6 · Cited by 878 · see citing works
Virology, transmission, and pathogenesis of SARS-CoV-2
Çevik, M., Kuppalli, K., Kindrachuk, J., & Peiris, M. J. S. (2020) · BMJ · doi:10.1136/bmj.m3862 · Cited by 750 · see citing works
COVID-19 Artificial Intelligence Diagnosis Using Only Cough Recordings
Laguarta, J., Hueto, F., & Subirana, B. (2020) · IEEE Open Journal of Engineering in Medicine and Biology · doi:10.1109/ojemb.2020.3026928 · Cited by 589 · see citing works
On the Class Imbalance Problem
Guo, X., Yin, Y., Dong, C., Yang, G., & Zhou, G. (2008) · 2008 Fourth International Conference on Natural Computation · doi:10.1109/icnc.2008.871 · Cited by 495 · see citing works
Exploring Automatic Diagnosis of COVID-19 from Crowdsourced Respiratory Sound Data
Brown, C., Chauhan, J., Grammenos, A., Han, J., Hasthanasombat, A., Spathis, D., Xia, T., Cicuta, P., & Mascolo, C. (2020) · Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3394486.3412865 · Cited by 447 · see citing works
Self-citationCoswara — A Database of Breathing, Cough, and Voice Sounds for COVID-19 Diagnosis
Sharma, N. K., Krishnan, P., Kumar, R. V., Ramoji, S., Chetupalli, S. R., Nirmala, R., Ghosh, P., & Ganapathy, S. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2768 · Cited by 332 · see citing works
Estimating Uncertainty and Interpretability in Deep Learning for Coronavirus (COVID-19) DetectionPreprint
Ghoshal, B., & Tucker, A. (2020) · Brunel University Research Archive (BURA) (Brunel University London) · doi:10.48550/arxiv.2003.10769 · Cited by 327 · see citing works
First experience of COVID-19 screening of health-care workers in England
Hunter, E., Price, D., Murphy, E., van der Loeff, I. S., Baker, K. F., Lendrem, D. W., Lendrem, C., Schmid, M. L., Pareja-Cebrian, L., Welch, A. R., et al. (2020) · The Lancet · doi:10.1016/s0140-6736(20)30970-3 · Cited by 306 · see citing works
COVID-19 cough classification using machine learning and global smartphone recordings
Pahar, M., Klopper, M., Warren, R., & Niesler, T. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104572 · Cited by 290 · see citing works
Bayesian Uncertainty Estimation for Batch Normalized Deep NetworksPreprint
Teye, M., Azizpour, H., & Smith, K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.06455 · Cited by 189 · see citing works
Han, J., Qian, K., Song, M., Yang, Z., Ren, Z. X., Liu, S., Liu, J., Zheng, H., Ji, W., Koike, T., et al. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2223 · Cited by 113 · see citing works
Self-citationVoice Quality Evaluation in Patients With COVID-19: An Acoustic Analysis
Asiaee, M., Vahedian‐Azimi, A., Atashi, S. S., Keramatfar, A., & Nourbakhsh, M. (2020) · Journal of Voice · doi:10.1016/j.jvoice.2020.09.024 · Cited by 112 · see citing works
Schubach, M., Re, M., Robinson, P. N., & Valentini, G. (2017) · Scientific Reports · doi:10.1038/s41598-017-03011-5 · Cited by 97 · see citing works
Cough Against COVID: Evidence of COVID-19 Signature in Cough Sounds
Bagad, P., Dalmia, A., Doshi, J., Nagrani, A., Bhamare, P., Mahale, A., Rane, S., Agarwal, N., & Panicker, R. (2020) · arXiv · doi:10.48550/arxiv.2009.08790 · Cited by 91 · see citing works
Schuller, B. W., Batliner, A., Bergler, C., Mascolo, C., Han, J., Lefter, I., Kaya, H., Amiriparian, S., Baird, A., Stappen, L., et al. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-19 · Cited by 89 · see citing works
Self-citationSARS-CoV-2 Detection From Voice
Pinkas, G., Karny, Y., Malachi, A., Barkai, G., Bachar, G. Y., & Aharonson, V. (2020) · IEEE Open Journal of Engineering in Medicine and Biology · doi:10.1109/ojemb.2020.3026468 · Cited by 86 · see citing works
Exploring Automatic COVID-19 Diagnosis via Voice and Symptoms from Crowdsourced Data
Han, J., Brown, C., Chauhan, J., Grammenos, A., Hasthanasombat, A., Spathis, D., Xia, T., Cicuta, P., & Cecilia Mascolo, C. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9414576 · Cited by 80 · see citing works
Self-citationEconomic Benefits of COVID-19 Screening Tests
Atkeson, A. G., Droste, M., Mina, M. J., & Stock, J. H. (2020) · National Bureau of Economic Research · doi:10.3386/w28031 · Cited by 78 · see citing works
Leveraging uncertainty information from deep neural networks for disease detectionPreprint
Leibig, C., Allken, V., Ayhan, M. S., et al. (2016) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/084210 · Cited by 53 · see citing works
Detecting COVID-19 from Breathing and Coughing Sounds using Deep Neural Networks
Nessiem, M. A., Mohamed, M. M., Coppock, H., Gaskell, A., & Schuller, B. W. (2021) · Proceedings - IEEE Symposium on Computer-Based Medical Systems · doi:10.1109/cbms52027.2021.00069 · Cited by 45 · see citing works
Schuller, B. W., Batliner, A. M., Bergler, C., Cecilia Mascolo, C., Han, J., Lefter, I., Kaya, H., Amiriparian, S., Baird, A., Stappen, L., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.13468 · Cited by 36 · see citing works
Self-citationA Real-Time Robot-Based Auxiliary System for Risk Evaluation of COVID-19 Infection
Wei, W., Wang, J., Ma, J., et al. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2105 · Cited by 25 · see citing works
Han, J., Qian, K., Song, M., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.00096 · Cited by 22 · see citing works
Self-citationQuantifying the uncertainty of deep learning-based computer-aided diagnosis for patient safety
Laves, M.-H., Ihler, S., Ortmaier, T., et al. (2019) · Current Directions in Biomedical Engineering · doi:10.1515/cdbme-2019-0057 · Cited by 20 · see citing works
Ensemble Approaches for Uncertainty in Spoken Language Assessment
Wu, X., Knill, K., Gales, M., & Malinin, A. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-2238 · Cited by 16 · see citing works
Detecting COVID-19 from Breathing and Coughing Sounds using Deep Neural NetworksPreprint
Schuller, B. W., Coppock, H., & Gaskell, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.14553 · Cited by 16 · see citing works
A Real-time Robot-based Auxiliary System for Risk Evaluation of COVID-19 InfectionPreprint
Wei, W., Wang, J., Ma, J., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.07695 · Cited by 7 · see citing works
The benefit of the doubt: Uncertainty aware sensing for edge computing platforms
Qendro, L., Chauhan, J., Ramos, A. G. C. P., & Mascolo, C. (2021) · arXiv · doi:10.48550/arxiv.2102.05956 · Cited by 6 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 3
- B. W. Schuller, D. M. Schuller, K. Qian, J. Liu, H. Zheng, and X. Li, "COVID-19 and computer audition: An overview on what speech & sound analysis could contribute in the sars-cov-2 corona crisis," Frontiers in Digital Health, vol. 3, pp. 1-10, 2021.
- A. Imran, I. Posokhova, H. N. Qureshi, U. Masood, M. S. Riaz, K. Ali, C. N. John, M. I. Hussain, and M. Nabeel, "Ai4COVID- 19: Ai enabled preliminary diagnosis for COVID-19 from cough samples via an app," Informatics in Medicine Unlocked, vol. 20, p. 100378, 2020.
- B. W. Schuller, S. Steidl, A. Batliner, P. B. Marschik, H. Baumeis- ter, F. Dong, S. Hantke, F. B. Pokorny, E.-M. Rathner, K. D. Bartl- Pokorny et al., "The 2018 computational paralinguistics chal- lenge: Atypical self-assessed affect, crying & heart beats." in Pro- ceedings of the INTERSPEECH, 2018, pp. 122-126.
Cited by
Sorted by citations
Sounds of COVID-19: exploring realistic performance of audio-based digital testing
Han, J., Xia, T., Spathis, D., Bondareva, E., Brown, C., Chauhan, J., Dang, T., Grammenos, A., Hasthanasombat, A., Floto, A., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-021-00553-x · Cited by 90 · see citing works
Self-citationHuman-centred artificial intelligence for mobile health sensing: challenges and opportunities
Dang, T., Spathis, D., Ghosh, A., & Mascolo, C. (2023) · Royal Society Open Science · doi:10.1098/rsos.230806 · Cited by 30 · see citing works
Self-citation
- Authors
- Xia, T., Han, J., Qendro, L., Dang, T., & Mascolo, C.
- Affiliations
- University of Cambridge (GB)
- Venue
- Interspeech 2021
- Publisher
- ISCA
- Published
- 2021-08-30
- Pages
- 2951-2955
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W3196633546
- Cited by
- 20
- References
- 37
- Citations by year
- 2025: 1 · 2024: 1 · 2023: 8 · 2022: 7 · 2021: 3
- FWCI
- 17.19
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Interspeech (USB) (publishedVersion)
- Topics
- Health Sciences › Medicine › Pulmonary and Respiratory Medicine › Phonocardiography and Auscultation Techniques
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › COVID-19 diagnosis using AI
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- COVID-19 detection, class imbalance, uncertainty estimation, ensemble learning, model disagreement, overconfidence, false positive reduction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
