On modelling label uncertainty in deep neural networks: Automatic estimation of intra-observer variability in 2d echocardiography quality assessment
Open accessThe DOI matches its Crossref record; 55 of the 56 references it cites were found in a registry or an index.
Abstract
Uncertainty of labels in clinical data resulting from intra-observer variability can have direct impact on the reliability of assessments made by deep neural networks. In this paper, we propose a method for modelling such uncertainty in the context of 2D echocardiography (echo), which is a routine procedure for detecting cardiovascular disease at point-of-care. Echo imaging quality and acquisition time is highly dependent on the operator's experience level. Recent developments have shown the possibility of automating echo image quality quantification by mapping an expert's assessment of quality to the echo image via deep learning techniques. Nevertheless, the observer variability in the expert's assessment can impact the quality quantification accuracy. Here, we aim to model the intra-observer variability in echo quality assessment as an aleatoric uncertainty modelling regression problem with the introduction of a novel method that handles the regression problem with categorical labels. A key feature of our design is that only a single forward pass is sufficient to estimate the level of uncertainty for the network output. Compared to the 0.11 ± 0.09 absolute error (in a scale from 0 to 1) archived by the conventional regression method, the proposed method brings the error down to 0.09 ± 0.08, where the improvement is statistically significant and equivalents to 5.7% test accuracy improvement. The simplicity of the proposed approach means that it could be generalized to other applications of deep learning in medical imaging, where there is often uncertainty in clinical labels.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Raghu al. [688] | et | 2019 | N/A | Classification | DR | 1 | DUP (Direct Uncertainty Pre- diction )”
From A review of uncertainty quantification in deep learning: Techniques, applications and challenges (2021) · cites this paper as [688]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 6×
“In parallel to the Mean and STD modules, the LSTM features are also fed to compute an additional value lambda (softmaxed over the 10 steps) to be the weighting parameter for each component.”
MethodInfluentialAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 3×
“The Adam [52] optimizer is 1The symbols defined in Fig.”
MethodInfluentialDensely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
(2014) · Cited by 33,419
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · cited 1×
“, 3 × 3 filter size with stride 1, no padding, batchnormalized [46], and use ReLU [47] activation function.”
MethodDistilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · cited 2×
“The soft targets methods [41], [42] are used as regularization techniques for training a small model with the predictions inferred from a high complexity model or an ensemble of them (trained from the same dataset), for the purposes of capturing the knowledge of the high complexity models with fewer model parameters (…”
BackgroundMethod(2010) · International Conference on Machine Learning · Cited by 13,995
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008 · cited 1×
“performance variations may be subtle when the deployed number of units/parameters are similar [49]–[51].”
BackgroundDropout as a bayesian approximation: Representing model uncertainty in deep learning
Gal, Y., & Ghahramani, Z. (2015) · arXiv · doi:10.48550/arxiv.1506.02142 · Cited by 6,471 · cited 2×
“is one of the commonly used Bayesian inference approximation method [36], which has been used in medical imaging to derive robust solutions for disease detection, lesion/organ segmentation [37]–[39].”
MethodBayesian Learning for Neural Networks
Neal, R. M. (1996) · Lecture Notes in Statistics · doi:10.1007/978-1-4612-0745-0 · Cited by 4,276 · cited 2×
“Several Bayesian inference approaches and more recent Bayesian neural networks (BNN) [25], [26], [27], [28] are designed to address the uncertainty in the induced classifier by imposing a prior distribution over model parameters.”
Method
Cited by
A review of uncertainty quantification in deep learning: Techniques, applications and challenges
Abdar, M., Pourpanah, F., Hussain, S., Rezazadegan, D., Liu, L., Ghavamzadeh, M., Fieguth, P., Cao, X., Khosravi, A., Acharya, U. R., et al. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.05.008 · Cited by 3,111
Seoni, S., Jahmunah, V., Salvi, M., Barua, P. D., Molinari, F., & Acharya, U. R. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107441 · Cited by 239
Huang, L., Ruan, S., Xing, Y., & Feng, M. (2024) · Medical Image Analysis · doi:10.1016/j.media.2024.103223 · Cited by 121
Towards Clinical Application of Artificial Intelligence in Ultrasound Imaging
Komatsu, M., Sakai, A., Dozen, A., Shozu, K., Yasutomi, S., Machino, H., Asada, K., Kaneko, S., & Hamamoto, R. (2021) · Biomedicines · doi:10.3390/biomedicines9070720 · Cited by 114
“proposed a quality assessment method for cardiac US images through modeling the label uncertainty in CNNs resulting from intra-observer variability in the labeling [100].”
MethodUlloa, A., Jing, L., Good, C. W., vanMaanen, D. P., Raghunath, S. M., Suever, J. D., Nevius, C. D., Wehner, G. J., Hartzel, D. N., Leader, J. B., et al. (2021) · Nature Biomedical Engineering · doi:10.1038/s41551-020-00667-9 · Cited by 84
Steps to use artificial intelligence in echocardiography
Kusunose, K. (2020) · Journal of Echocardiography · doi:10.1007/s12574-020-00496-4 · Cited by 72
“[16] 2019 Quality 14,443 echo studies – Score error: 0.”
BackgroundReal-time echocardiography image analysis and quantification of cardiac indices
Zamzmi, G., Rajaraman, S., Hsu, L., Sachdev, V., & Antani, S. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102438 · Cited by 79
“Other deep learningbased approaches for echo quality assessment can be found in Nolan and Thavendiranathan (2019); Liao et al. (2019); Zamzmi et al. (2019b); Vrettos et al. (2020).”
Backgroundde Siqueira, V. S., Borges, M. M., Furtado, R. G., Dourado, C. N., & da Costa, R. M. (2021) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2021.102165 · Cited by 70
Dilated densely connected U-Net with uncertainty focus loss for 3D ABUS mass segmentation
Cao, X., Chen, H., Li, Y., Peng, Y., Wang, S., & Cheng, L. (2021) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2021.106313 · Cited by 61
Image quality assessment for machine learning tasks using meta-reinforcement learning
Saeed, S. U., Fu, Y., Stavrinides, V., Baum, Z. M. C., Yang, Q., Rusu, M., Fan, R. E., Sonn, G. A., Noble, J. A., Barratt, D. C., et al. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102427 · Cited by 58
“Most of the existing methods learn from subjective expert labels of IQA (Wu et al., 2017; Zago et al., 2018; Esses et al., 2018; Baum et al., 2021; Liao et al., 2019; Abdi et al., 2017; Lin et al., 2019; Camps et al., 2020), based on a set of pre-selected reference images to varying extent.”
MethodAssisted probe guidance in cardiac ultrasound: A review
Ferraz, S. M., Coimbra, M. T., & Pedrosa, J. (2023) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2023.1056055 · Cited by 32
Blaivas, M., Adhikari, S., Savitsky, E. A., Blaivas, L. N., & Liu, Y. T. (2020) · Journal of the American College of Emergency Physicians Open · doi:10.1002/emp2.12206 · Cited by 30
“10 Artificial intelligence is rapidly infiltrating modern medicine.”
BackgroundLi, X., Zhang, H., Yue, J., Yin, L., Li, W., Ding, G., Peng, B., & Xie, S. (2024) · Scientific Reports · doi:10.1038/s41598-024-71530-z · Cited by 43
Automated assessment of transthoracic echocardiogram image quality using deep neural networks
Labs, R. B., Vrettos, A., Loo, J. K. K., & Zolgharni, M. (2022) · Intelligent Medicine · doi:10.1016/j.imed.2022.08.001 · Cited by 21
“However, echocardiogram quality assessment still exists in subjective domain, with significant impacts, as documented in [4], [5], [6], ultimately pointed to the issues of clinical reliability, misdiagnosis, and poor response to patient care.”
BackgroundVaseli, H., Gu, A. N., Amiri, S. N. A., Tsang, M. Y. C., Fung, A., Kondori, N., Saadat, A., Abolmaesumi, P., & Tsang, T. S. M. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43987-2_36 · Cited by 36
Self-citationZhang, Y., Zhu, H., Cheng, J., Wang, J., Gu, X., Han, J., Zhang, Y., Zhao, Y., He, Y., & Zhang, H. (2023) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2023.3303573 · Cited by 21
Uncertainty, Evidence, and the Integration of Machine Learning into Medical Practice
Grote, T., & Berens, P. (2023) · The Journal of Medicine and Philosophy A Forum for Bioethics and Philosophy of Medicine · doi:10.1093/jmp/jhac034 · Cited by 30
Elshafey, R., Hamila, R., & Bensaali, F. (2024) · Medicine in Novel Technology and Devices · doi:10.1016/j.medntd.2024.100322 · Cited by 22
Radiomics in Echocardiography: Deep Learning and Echocardiographic Analysis
Kusunose, K. (2020) · Current Cardiology Reports · doi:10.1007/s11886-020-01348-4 · Cited by 20
“This proposed approach could also be generalized to other images involving deep learning in the cardio-vascular field, where there are frequently gaps in clinical labeling [16].”
BackgroundGuo, Y., Hu, M., Min, X., Wang, Y., Dai, M., Zhai, G., Zhang, X.-P. S., & Yang, X. (2023) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2023.3282387 · Cited by 19
“There have been already many IQA and video quality assessment algorithms for screen environments [4], [5], [6], [7], [8], [9], [10], but not so many as those of immersive environments [11], [12], [13].”
BackgroundRegion Uncertainty Estimation for Medical Image Segmentation With Noisy Labels
Han, K., Wang, S., Chen, J., Qian, C., Lyu, C., Ma, S., Qiu, C., Sheng, V. S., Huang, Q., & Liu, Z. (2025) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2025.3589058 · Cited by 28
“For Monte Carlo dropout, Liao et al. [27] modeled uncertainty caused by intra-observer variability in 2D echocardiography.”
Zhao, H., Zheng, Q., Teng, C., Yasrab, R., Drukker, L., Papageorghiou, A. T., & Noble, J. A. (2023) · Medical Image Analysis · doi:10.1016/j.media.2023.102977 · Cited by 19
Deep Video Networks for Automatic Assessment of Aortic Stenosis in Echocardiography
Ginsberg, T., Tal, R.-E., Tsang, M. Y. C., MacDonald, C. A., Dezaki, F. T., van der Kuur, J., Luong, C. L., Abolmaesumi, P., & Tsang, T. S. M. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87583-1_20 · Cited by 17
Self-citationAutomated Recognition of Ultrasound Cardiac Views Based on Deep Learning with Graph Constraint
Gao, Y., Zhu, Y., Liu, B., Hu, Y., Yu, G., & Guo, Y. (2021) · Diagnostics · doi:10.3390/diagnostics11071177 · Cited by 15
“The first application is evaluation of image quality in echocardiography [17].”
BackgroundAdaptable Image Quality Assessment Using Meta-Reinforcement Learning of Task Amenability
Saeed, S. U., Fu, Y., Stavrinides, V., Baum, Z. M. C., Yang, Q., Rusu, M., Fan, R. E., Sonn, G. A., Noble, J. A., Barratt, D. C., et al. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-87583-1_19 · Cited by 14
Gu, A. N., Vaseli, H., Tsang, M., Wu, V., Amiri, S. N. A., Kondori, N., Fung, A., Tsang, T. S. M., & Abolmaesumi, P. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103600 · Cited by 15
Self-citationEchocardiographic Image Quality Assessment Using Deep Neural Networks
Labs, R. B., Zolgharni, M., & Loo, J. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-80432-9_36 · Cited by 12
Saeed, S. U., Wen, Y., Fu, Y., Giganti, F., Yang, Q., Baum, Z. M. C., Rusu, M., Fan, R. E., Sonn, G. A., Emberton, M., et al. (2022) · The Journal of Machine Learning for Biomedical Imaging · doi:10.59275/j.melba.2022-a1cc · Cited by 12
“Recent deep learning based approaches to IQA also rely on human labels of IQA and offer fast inference (Wu et al., 2017; Zago et al., 2018; Esses et al., 2018; Baum et al., 2021; Liao et al., 2019; Abdi et al., 2017; Lin et al., 2019; Oksuz et al., 2020).”
MethodJavadi, G., Bayat, S., Esfeh, M. M. K., Samadi, S., Sedghi, A., Sojoudi, S., Hurtado, A., Chang, S. D., Black, P. C., Mousavi, P., et al. (2021) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-021-02485-z · Cited by 9
Self-citationJansen, G. E., de Vos, B. D., Molenaar, M. A., Schuuring, M. J., Bouma, B. J., & Išgum, I. (2024) · Journal of medical imaging · doi:10.1117/1.jmi.11.5.054002 · Cited by 12
“Similar to Liao et al., 21 who found only 55% intra-rater agreement, we found 44.0% inter-rater agreement.”
InfluentialXu, J., Jin, Y., Martin, B., Smith, A., Wright, S. J., Stoyanov, D. V., & Mazomenos, E. B. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43996-4_15 · Cited by 7
“Deep models for AQA have shown promise in transthoracic echocardiography (TTE) and other ultrasound (US) applications [1,9,13,14].”
Wu, V., Fung, A., Khodabakhshian, B., Abdelsamad, B., Vaseli, H., Ahmadi, N., Goco, J. A., Tsang, M. Y. C., Luong, C. L., Abolmaesumi, P., et al. (2025) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2025.3609319 · Cited by 10
“Using a view detection algorithm [43] and cardiologist veri-fication, we extracted only PLAX and PSAX views without colour or spectral Doppler.”
Self-citationLiu, W., Wang, Q., Zhang, P., Deng, Y., Zhao, Y., Zhang, Y., Xu, H., Qiu, X., Xu, C., Xu, J., et al. (2025) · Applied Intelligence · doi:10.1007/s10489-025-06419-z · Cited by 8
“Consequently, the vagueness in label transitions for specific local parts leaded to ambiguous decision boundaries in the model’s quality score judgments for these images [36].”
China, D., MacLean, L. J., Wei, J., Theodore, N., Johnson, N., Crawford, N. R., Ding, K., & Uneri, A. (2025) · Medical Image Analysis · doi:10.1016/j.media.2025.103672 · Cited by 5
Robust fine-grained echocardiographic view classification with supervised contrastive learning
Naidoo, P., Fernandes, P., Serej, N. D., Stowell, C. C., Manisty, C., Francis, D. P., & Zolgharni, M. (2026) · Medical Image Analysis · doi:10.1016/j.media.2026.104006 · Cited by 3
Kucerenko, A., Buddenkotte, T., Apostolova, I. I., Klutmann, S., Ledig, C., & Buchert, R. (2024) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-024-06988-0 · Cited by 3
“RVT and AVT are easy-to-implement and require only a single forward pass to estimate uncertainty of the network prediction [43].”
Saadat, A., Hashemi, N., Vaseli, H., Tsang, M. Y., Luong, C., van de Panne, M., Tsang, T. S. M., & Abolmaesumi, P. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-05185-1_46 · Cited by 4
“A view-detection algorithm [13] automatically identified parasternal long-axis (PLAX) and short-axis (PSAX) cine clips, which were then re-screened by a Level III echocardiographer to remove misclassifications.”
Self-citationUsing ScrutinAI for visual inspection of DNN performance in a medical use case
Görge, R., Haedecke, E., & Möck, M. (2023) · AI and Ethics · doi:10.1007/s43681-023-00399-x · Cited by 3
“This results in a lack of consistency among different observers, which is defined as inter-observer variability (Liao et al. 2019).”
Gao, Q., Shan, H., & Zeng, D. (2023) · IEEE International Conference on Medical Artificial Intelligence (MedAI) · doi:10.1109/medai59581.2023.00059 · Cited by 2
“However, due to inconsistencies of scale in scoring by different human raters, the problem of inter-observer variability arises [21].”
BackgroundHsu, C. F., Wang, Y.-W., Lin, L., & Chang, R. (2025) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2025.107671 · Cited by 2
A Label Uncertainty-Guided Multi-Stream Model for Disease Screening
Liu, C., Ge, Z., He, M., & Han, X. (2022) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi52829.2022.9761483 · Cited by 4
“Previous studies exploited such information for quantifying imaging quality [7], or identifying hard patient cases that may require a medical second opinion [8].”
BackgroundArtificial Intelligence Algorithm Development for Biomedical Imaging
van Ooijen, P. M. A., & van den Oever, L. B. (2022) · Contemporary medical imaging · doi:10.1007/978-3-030-92087-6_3 · Cited by 1
Spatiotemporal Contrastive Learning for Echocardiography View Classification
Naidoo, P., Fernandes, P., Ufumaka, I., Serej, N. D., Howard, J. P., Francis, D. P., Manisty, C., & Zolgharni, M. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-00656-1_18 · Cited by 2
Deslandes, A., Petashvili, D., Wang, H., Carnerio, G., Avery, J., Condous, G., Leonardi, M., Hull, M. L., & Chen, H. (2025) · Australasian Journal of Ultrasound in Medicine · doi:10.1002/ajum.70026 · Cited by 1
“The result of this will be noisy image labelling, which is typically problematic for AI tools to manage [12, 22].”
InfluentialIslam, J., Farady, I., Kuo, C.-C., Lin, F., & Lin, C.-Y. (2026) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2026.109717 · Cited by 2
Peng, H., Chen, Z., Tan, X., Fan, X., Xu, L., Li, S., Chen, D., & Sun, F. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.131777 · Cited by 1
賢也, 楠. (2020) · Journal of JCS Cardiologists · doi:10.1253/jjcsc.29.0_8 · Cited by 1
Toward a Generic Multi-modal Medical Data Representation Model
Swaroopa, K. M., Kaur, N., & Chetty, G. (2023) · Lecture notes in networks and systems · doi:10.1007/978-981-99-5088-1_33
Li, Y., Cao, Y., Mi, J., Cui, X., Hu, X., Zhang, Y., Liu, Z., Cui, L., & Li, S. (2024) · Future Generation Computer Systems · doi:10.1016/j.future.2024.107693
No-reference image quality assessment method based on self-attention transformer encoder
Zheng, Y., Yang, F., Liu, C., Liao, K., Wang, K., Sun, B., Zhong, C., & Huo, H. (2025) · Signal Image and Video Processing · doi:10.1007/s11760-025-04382-3
Poslon, L., & Čartolovni, A. (2025) · Springer eBooks · doi:10.1007/978-3-031-91379-2_3
Guillaumot, C., Sklab, Y., Dubied, M., Laffont, R., & Navarro, N. (2026) · Evolutionary Biology · doi:10.1007/s11692-026-09669-y · Cited by 1
“As imaging technologies continue to advance and gener - ate increasingly large datasets, manual landmarking has become a major bottleneck, limiting both scalability and reproducibility (Karlsson et al., 2003; Liao et al., 2019).”
Saadat, A., Hashemi, N., Khodabakhshian, B., Tsang, M. Y., Luong, C., Tsang, T. S. M., & Abolmaesumi, P. (2026) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-026-03680-6
“In contrast, aleatoric uncertainty arises from inherent randomness or measurement noise in the data-generating process and is not reducible by increasing dataset size [14].”
Self-citationXie, D., Jie, H. U., Liang, G., Zhi, R., & Deng, Y. (2026) · Cardiology · doi:10.1159/000552266
Point tracking as a temporal Cue for robust myocardial segmentation in echocardiography videos
Khodabakhshian, B., Hashemi, N., Saadat, A., Gholami, Z., Hwang, I.-C., Sojoudi, S., Luong, C. L., Abolmaesumi, P., & Tsang, T. S. M. (2026) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-026-03645-9
“The test set includes 40 poor-quality and 694 medium/good-quality frames, with quality assessed using an automatic echo quality estimation model [24].”
Self-citationMulti-Expert Consensus as a Label-Free Quality Surrogate for Echocardiographic LV Segmentation
Ufumaka, I., Alibakhshi, A., Fernandes, P., Abdi, A., Hussain, A., Dadashiserej, N., Shun-Shin Matthew, J., Francis, D. P., & Zolgharni, M. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-35393-1_18
Self-citation
Metadata
- Authors
- Liao, Z., Girgis, H., Abdi, A., Vaseli, H., Hetherington, J., Rohling, R., Gin, K., Tsang, T., & Abolmaesumi, P.
- Affiliations
- University of British Columbia (CA)
- ORCID
- Venue
- IEEE Transactions on Medical Imaging
- Published
- 2020-06
- Volume
- 39
- Issue
- 6
- Pages
- 1868-1883
- ISSN
- 0278-0062 (ISSN-L)
- 1558-254X
- OpenAlex
- W2996089546
- Cited by
- 56
- References
- 56
- Citations by year
- 2026: 8 · 2025: 11 · 2024: 6 · 2023: 11 · 2022: 6 · 2021: 10 · 2020: 4
- FWCI
- 1.59
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Medical Imaging (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Canadian Institutes of Health Research
- Natural Sciences and Engineering Research Council of Canada
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image and Video Quality Assessment
- Keywords
- label uncertainty, intraobserver variability, two-dimensional echocardiography, aleatoric uncertainty, uncertainty estimation, categorical regression, single forward pass
- MeSH terms
- Echocardiography, Humans, Reproducibility of Results, Observer Variation, Neural Networks, Computer, Uncertainty
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.