CoChat

On modelling label uncertainty in deep neural networks: Automatic estimation of intra-observer variability in 2d echocardiography quality assessment

Open access

The DOI matches its Crossref record; 55 of the 56 references it cites were found in a registry or an index.

Abstract

Uncertainty of labels in clinical data resulting from intra-observer variability can have direct impact on the reliability of assessments made by deep neural networks. In this paper, we propose a method for modelling such uncertainty in the context of 2D echocardiography (echo), which is a routine procedure for detecting cardiovascular disease at point-of-care. Echo imaging quality and acquisition time is highly dependent on the operator's experience level. Recent developments have shown the possibility of automating echo image quality quantification by mapping an expert's assessment of quality to the echo image via deep learning techniques. Nevertheless, the observer variability in the expert's assessment can impact the quality quantification accuracy. Here, we aim to model the intra-observer variability in echo quality assessment as an aleatoric uncertainty modelling regression problem with the introduction of a novel method that handles the regression problem with categorical labels. A key feature of our design is that only a single forward pass is sufficient to estimate the level of uncertainty for the network output. Compared to the 0.11 ± 0.09 absolute error (in a scale from 0 to 1) archived by the conventional regression method, the proposed method brings the error down to 0.09 ± 0.08, where the improvement is statistically significant and equivalents to 5.7% test accuracy improvement. The simplicity of the proposed approach means that it could be generalized to other applications of deep learning in medical imaging, where there is often uncertainty in clinical labels.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tmi.2019.2959209

Cites

55 of 55 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 55 references

Research works printed but not linked: 1

  1. 2019, Echocardiography in ICU—Main Views

Cited by

See all 56

Metadata

Authors
Liao, Z., Girgis, H., Abdi, A., Vaseli, H., Hetherington, J., Rohling, R., Gin, K., Tsang, T., & Abolmaesumi, P.
Affiliations
University of British Columbia (CA)
Venue
IEEE Transactions on Medical Imaging
Published
2020-06
Volume
39
Issue
6
Pages
1868-1883
ISSN
  • 0278-0062 (ISSN-L)
  • 1558-254X
OpenAlex
W2996089546
Cited by
56
References
56
Citations by year
2026: 8 · 2025: 11 · 2024: 6 · 2023: 11 · 2022: 6 · 2021: 10 · 2020: 4
FWCI
1.59
Type
journal article
Language
en
Open access
Locations
  • IEEE Transactions on Medical Imaging (publishedVersion)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Canadian Institutes of Health Research
  • Natural Sciences and Engineering Research Council of Canada
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image and Video Quality Assessment
Keywords
label uncertainty, intraobserver variability, two-dimensional echocardiography, aleatoric uncertainty, uncertainty estimation, categorical regression, single forward pass
MeSH terms
Echocardiography, Humans, Reproducibility of Results, Observer Variation, Neural Networks, Computer, Uncertainty

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: On modelling label uncertainty in deep neural networks: Automatic estimation of intra-observer variability in 2d echocardiography quality assessment. https://app.cochat.ai/papers/10.1109/tmi.2019.2959209
The paper
Liao, Z., Girgis, H., Abdi, A., Vaseli, H., Hetherington, J., Rohling, R., Gin, K., Tsang, T., & Abolmaesumi, P. (2020). On modelling label uncertainty in deep neural networks: Automatic estimation of intra-observer variability in 2d echocardiography quality assessment. IEEE Transactions on Medical Imaging. https://doi.org/10.1109/tmi.2019.2959209

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.