Uncertainty-guided annotation enhances segmentation with the human-in-the-loop
Open accessAbstract
Deep learning algorithms, often critiqued for their 'black box' nature, traditionally fall short in providing the necessary transparency for trusted clinical use. This challenge is particularly evident when such models are deployed in local hospitals, encountering out-of-domain distributions due to varying imaging techniques and patient-specific pathologies. Yet, this limitation offers a unique avenue for continual learning. The Uncertainty-Guided Annotation (UGA) framework introduces a human-in-the-loop approach, enabling AI to convey its uncertainties to clinicians, effectively acting as an automated quality control mechanism. UGA eases this interaction by quantifying uncertainty at the pixel level, thereby revealing the model's limitations and opening the door for clinician-guided corrections. We evaluated UGA on the Camelyon dataset for lymph node metastasis segmentation which revealed that UGA improved the Dice coefficient (DC), from 0.66 to 0.76 by adding 5 patches, and further to 0.84 with 10 patches. To foster broader application and community contribution, we have made our code accessible at
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Disagreement among CV models is increasingly reused as a proxy for epistemic uncertainty in downstream applications such as quality control and human-in-the-loop review.
“Increasingly, disagreement among these CV models is reused as a proxy for epistemic uncertainty in downstream applications such as quality control and human-in-the-loop review [11,18,15].”
From Lost in the Folds: When Cross-Validation Is Not a Deep Ensemble for Uncertainty Estimation (2026) · cites this paper as [15]
Section 3 measures uncertainty by disagreement among five cross-validated models, and the Abstract describes its use for human-in-the-loop review and automated quality control. The copy does not identify that uncertainty specifically as epistemic or establish that this use is increasing across downstream applications. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 3
- S. J. Wagner, N. Khalili, R. Sharma, M. Boxberg, C. Marr, W. de Back, T. Peng, Structure-preserving multi-domain stain color augmentation us- ing style-transfer with disentangled representations, in: Medical Image Computing and Computer Assisted Intervention-MICCAI 2021: 24th International Conference, Strasbourg, France, September 27-October 1, 2021, Proceedings, Part VIII 24, Springer, 2021, pp. 257-266.
- B. Lakshminarayanan, A. Pritzel, C. Blundell, Simple and scalable pre- dictive uncertainty estimation using deep ensembles, Advances in neural information processing systems 30 (2017).
- V. Nath, D. Yang, B. A. Landman, D. Xu, H. R. Roth, Diminishing uncertainty within the training pool: Active learning for medical im- age segmentation, IEEE Transactions on Medical Imaging 40 (10) (2020) 2534-2547.
Cited by
Sorted by citations
Lost in the Folds: When Cross-Validation Is Not a Deep Ensemble for Uncertainty Estimation
Kirscher, T., Bujotzek, M., Kirchhoff, Y., Rokuss, M., Isensee, F., Kahl, K.-C., Kovacs, B., & Maier-Hein, K. (2026) · arXiv · doi:10.48550/arxiv.2605.18329
- Authors
- Khalili, N., Spronck, J., Ciompi, F., van der Laak, J., & Litjens, G.
- Affiliations
- Radboud University Medical Center (NL)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2404.07208
- OpenAlex
- W4394780582
- Cited by
- 1
- References
- 0
- Citations by year
- 2025: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › AI in cancer detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- human-in-the-loop, uncertainty quantification, medical image segmentation, automated quality control, out-of-domain generalization, continual learning, clinical trust in AI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
