Evaluating Segmentation Error without Ground Truth
Open accessAbstract
The automatic delineation of the boundaries of organs and other anatomical structures is a key component of many medical image processing systems. In this paper we present a generic learning approach based on a novel space of segmentation features, which can be trained to predict the overlap error and Dice coefficient of an arbitrary organ segmentation without knowing the ground truth delineation. We show the regressor to be much stronger a predictor of these error metrics than the responses of probabilistic boosting classifiers trained on the segmentation boundary. The presented approach not only allows us to build reliable confidence measures and fidelity checks, but also to rank several segmentation hypotheses against each other during online usage of the segmentation algorithm in clinical practice.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Approaches [31], [33] can only be applied when a training database with good and bad segmentations is available.
“However, the approaches [31], [33] can only be applied when a training database with good and bad segmentations is available from which a mapping from features to segmentation accuracy can be learned.”
From Reverse Classification Accuracy: Predicting Segmentation Performance in the Absence of Ground Truth (2017) · cites this paper as [31]
Unlike the supervised methods in [33] and [31], RCA requires no training data that captures examples of good and bad segmentations.
“An appealing property of the proposed framework is that unlike the supervised methods in [33] and [31] no training data is required that captures examples of good and bad segmentations.”
From Reverse Classification Accuracy: Predicting Segmentation Performance in the Absence of Ground Truth (2017) · cites this paper as [31]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Tu, Z. (2005) · Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1 · doi:10.1109/iccv.2005.194 · Cited by 397 · see citing works
Flagged first, then by citations
Research works printed but not linked: 17
- Blake, A., Zisserman, A.: Visual Reconstruction. MIT Press (1987)
- Delp, S.L., DiGoia, A.M., Jaramaz, B. (eds.) MICCAI 2000. LNCS, vol. 1935, pp. 276-286. Springer, Heidelberg (2000)
- Caselles, V., Kimmel, R., Sapiro, G., Sbert, C.: Minimal surfaces based object segmentation. IEEE Trans. on Pat. Anal. and Mach. Int. 19(4), 394-398 (1997)
- Deng, X., Zhu, L., Sun, Y., Xu, C., Song, L., Chen, J., Merges, R.D., Jolly, M.- P., Suehling, M., Xu, X.-D.: On Simulating Subjective Evaluation Using Com- bined Objective Metrics for Validation of 3D Tumor Segmentation. In: Ayache, N., Ourselin, S., Maeder, A. (eds.) MICCAI 2007, Part I. LNCS, vol. 4791, pp. 977-984. Springer, Heidelberg (2007)
- El-Zehiry, N., Grady, L.: Fast global optimization of curvature. In: CVPR 2010, pp. 3257-3264. IEEE Computer Society (2010)
- Frounchi, K., Briand, L.C., Grady, L., Labiche, Y., Subramanyan, R.: Automating image segmentation verification and validation by learning test oracles. Information and Software Technology 53(12), 1337-1348 (2011)
- Ginneken, B.V., Heimann, T., Styner, M.: 3D segmentation in the clinic: A grand challenge. In: MICCAI Workshop 2007, pp. 7-15 (2007)
- Grady, L.: Random walks for image segmentation. IEEE Trans. on Pat. Anal. and Mach. Int. 28(11), 1768-1783 (2006)
- Grady, L., Schwartz, E.L.: Isoperimetric graph partitioning for image segmentation. IEEE Trans. on Pat. Anal. and Mach. Int. 28(3), 469-475 (2006)
- Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.: The WEKA Data Mining Software: An Update. SIGKDD Explorations 1(11)
- Heimann, T., van Ginneken, B., Styner, M.: Comparison and evaluation of methods for liver segmentation from CT datasets. IEEE Trans. on Medical Imaging 28(8), 1251-1265 (2009)
- Shi, J., Malik, J.: Normalized cuts and image segmentation. IEEE Trans. on Pat. Anal. and Mach. Int. 22(8), 888-905 (2000)
- Kohlberger, T., Sofka, M., Zhang, J., Birkbeck, N., Wetzl, J., Kaftan, J., Declerck, J., Zhou, S.: Automatic Multi-organ Segmentation Using Learning-Based Segmen- tation and Level Set Optimization. In: Fichtinger, G., Martel, A., Peters, T. (eds.) MICCAI 2011, Part III. LNCS, vol. 6893, pp. 338-345. Springer, Heidelberg (2011)
- Ling, H., Zhou, S., Zheng, Y., Georgescu, B., Suehling, M., Comaniciu, D.: Hierar- chical, learning-based automatic liver segmentation. In: Proc. of CVPR 2008, pp. 1-8. IEEE Computer Society (2008)
- Sethian, J.A.: Level Set Methods and Fast Marching Methods. Cambridge Univer- sity Press (1999)
- Shevade, S., Keerthi, S., Bhattacharyya, C., Murthy, K.: Improvements to the SMO algorithm for SVM regression. IEEE Trans. on Neural Networks 11(5), 1188-1193 (2000)
- Zheng, Y., Barbu, A., Georgescu, B., Scheuering, M., Comaniciu, D.: Four-chamber heart modeling and automatic segmentation for 3-D cardiac CT volumes using marginal space learning and steerable features. IEEE Trans. on Medical Imag- ing 27(11), 1668-1681 (2008)
Cited by
Sorted by citations
Reverse Classification Accuracy: Predicting Segmentation Performance in the Absence of Ground Truth
Valindria, V. V., Lavdas, I., Bai, W., Kamnitsas, K., Aboagye, E. O., Rockall, A. G., Rueckert, D., & Glocker, B. (2017) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2017.2665165 · Cited by 152 · see citing works
Roy, A. G., Conjeti, S., Navab, N., & Wachinger, C. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.03.042 · Cited by 137 · see citing works
Automatic Quality Control of Cardiac MRI Segmentation in Large-Scale Population Imaging
Robinson, R., Valindria, V. V., Bai, W., Suzuki, H., Matthews, P. M., Page, C., Rueckert, D., & Glocker, B. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-66182-7_82 · Cited by 37 · see citing works
Efficient Model Monitoring for Quality Control in Cardiac Image Segmentation
Galati, F., & Zuluaga, M. A. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-78710-3_11 · Cited by 15 · see citing works
O.K., S., Riera-Marín, M., Galdran, A., López, J. G., Rodríguez-Comas, J., Piella, G., & Ballester, M. A. G. (2025) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2025.102547 · Cited by 11 · see citing works
Yazdan-Panah, A., Stankoff, B., & Colliot, O. (2024) · Medical Imaging 2024: Image Processing · doi:10.1117/12.3006580 · Cited by 1 · see citing works
ConfIC-RCA: Statistically Grounded Efficient Estimation of Segmentation Quality
Cosarinsky, M., Billot, R., Mansilla, L., Jimenez, G., Gaggión, N., Fu, G., Tirer, T., & Ferrante, E. (2025) · arXiv
“Such approaches have been applied to prostate segmentation [12], cardiac segmentation [13], and a more generic framework has been used to assess the quality for automated segmentations for lung and liver in [14].”
- Authors
- Kohlberger, T., Singh, V., Alvino, C., Bahlmann, C., & Grady, L.
- Affiliations
- Siemens (United States) (US)
- HeartFlow (United States) (US)
- Venue
- Lecture Notes in Computer Science
- Publisher
- Springer Berlin Heidelberg
- Published
- 2012
- Volume
- 15
- Issue
- Pt 1
- Pages
- 528-536
- ISSN
- 0302-9743 (ISSN-L)
- 1611-3349
- OpenAlex
- W130387664
- Cited by
- 102
- References
- 1
- Citations by year
- 2026: 2 · 2025: 5 · 2024: 9 · 2023: 11 · 2022: 10 · 2021: 8 · 2020: 10 · 2019: 9 · 2018: 10 · 2017: 7 · 2016: 6 · 2015: 7 · 2014: 3 · 2013: 5
- FWCI
- 3.41
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Lecture notes in computer science (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
- Physical Sciences › Engineering › Biomedical Engineering › Medical Imaging and Analysis
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- organ segmentation, ground truth-free evaluation, Dice coefficient, confidence measures, medical image segmentation
- MeSH terms
- Algorithms, Computer Simulation, False Positive Reactions, Humans, Image Processing, Computer-Assisted, Lung, Pattern Recognition, Automated, Probability, Reproducibility of Results, Models, Statistical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
