Conformal Performance Range Prediction for Segmentation Output Quality Control
Open accessAbstract
Recent works have introduced methods to estimate segmentation performance without ground truth, relying solely on neural network softmax outputs. These techniques hold potential for intuitive output quality control. However, such performance estimates rely on calibrated softmax outputs, which is often not the case in modern neural networks. Moreover, the estimates do not take into account inherent uncertainty in segmentation tasks. These limitations may render precise performance predictions unattainable, restricting the practical applicability of performance estimation methods. To address these challenges, we develop a novel approach for predicting performance ranges with statistical guarantees of containing the ground truth with a user specified probability. Our method leverages sampling-based segmentation uncertainty estimation to derive heuristic performance ranges, and applies split conformal prediction to transform these estimates into rigorous prediction ranges that meet the desired guarantees. We demonstrate our approach on the FIVES retinal vessel segmentation dataset and compare five commonly used sampling-based uncertainty estimation techniques. Our results show that it is possible to achieve the desired coverage with small prediction ranges, highlighting the potential of performance range prediction as a valuable tool for output quality control.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
PHiSeg is the top performer in the work of [35].
“In order to enable direct comparison with [35], we adopt PHiSeg [37]-the top performer in that work-as our backbone segmentation model.”
From ConfIC-RCA: Statistically Grounded Efficient Estimation of Segmentation Quality (2025) · cites this paper as [35]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 29
- Angelopoulos, A.N., Bates, S.: A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511 (2021) · arXiv
- Angelopoulos, A.N., Kohli, A.P., Bates, S., Jordan, M., Malik, J., Alshaabi, T., Upadhyayula, S., Romano, Y.: Image-to-image regression with distribution-free uncertainty quantification and applications in imaging. In: International Confer- ence on Machine Learning. pp. 717-730. PMLR (2022)
- Azad, R., Aghdam, E.K., Rauland, A., Jia, Y., Avval, A.H., Bozorgpour, A., Karim- ijafarbigloo, S., Cohen, J.P., Adeli, E., Merhof, D.: Medical image segmentation review: The success of u-net. arXiv preprint arXiv:2211.14830 (2022) · arXiv
- Barber, R.F., Candes, E.J., Ramdas, A., Tibshirani, R.J.: Conformal prediction beyond exchangeability. The Annals of Statistics 51(2), 816-845 (2023)
- Baumgartner, C.F., Tezcan, K.C., Chaitanya, K., Hötker, A.M., Muehlematter, U.J., Schawkat, K., Becker, A.S., Donati, O., Konukoglu, E.: Phiseg: Capturing uncertainty in medical image segmentation. In: Medical Image Computing and Computer Assisted Intervention -MICCAI 2019. pp. 119-127. Springer Interna- tional Publishing, Cham (2019)
- Bernard, O., Lalande, A., Zotti, C., Cervenansky, F., Yang, X., Heng, P.A., Cetin, I., Lekadir, K., Camara, O., Ballester, M.A.G., et al.: Deep learning techniques for automatic mri cardiac multi-structures segmentation and diagnosis: is the problem solved? IEEE transactions on medical imaging 37(11), 2514-2525 (2018)
- Cho, W., Park, J., Choo, J.: Training auxiliary prototypical classifiers for explain- able anomaly detection in medical image segmentation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 2624- 2633 (2023)
- Fournel, J., Bartoli, A., Bendahan, D., Guye, M., Bernard, M., Rauseo, E., Khanji, M.Y., Petersen, S.E., Jacquier, A., Ghattas, B.: Medical image segmentation auto- matic quality control: A multi-dimensional approach. Medical Image Analysis 74, 102213 (2021)
- Gal, Y., Ghahramani, Z.: Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In: International Conference on Machine Learn- ing. pp. 1050-1059. PMLR (2016)
- Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neural networks. In: International conference on Machine Learning. pp. 1321-1330. PMLR (2017)
- Hann, E., Gonzales, R.A., Popescu, I.A., Zhang, Q., Ferreira, V.M., Piechnik, S.K.: Ensemble of deep convolutional neural networks with monte carlo dropout sam- pling for automated image segmentation quality control and robust deep learning using small datasets. In: Medical Image Understanding and Analysis. pp. 280-293. Springer International Publishing, Cham (2021)
- Herrera, W.G., Pereira, M., Bento, M., Lapa, A.T., Appenzeller, S., Rittner, L.: A framework for quality control of corpus callosum segmentation in large-scale studies. Journal of Neuroscience Methods 334, 108593 (2020)
- Isensee, F., Jaeger, P.F., Kohl, S.A., Petersen, J., Maier-Hein, K.H.: nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods 18(2), 203-211 (2021)
- Jin, K., Huang, X., Zhou, J., Li, Y., Yan, Y., Sun, Y., Zhang, Q., Wang, Y., Ye, J.: Fives: A fundus image dataset for artificial intelligence based vessel segmentation. Scientific Data 9(1), 475 (2022)
- Kahl, K.C., Lüth, C.T., Zenk, M., Maier-Hein, K., Jaeger, P.F.: Values: A frame- work for systematic validation of uncertainty estimation in semantic segmentation. arXiv preprint arXiv:2401.08501 (2024) · arXiv
- Kohl, S., Romera-Paredes, B., Meyer, C., De Fauw, J., Ledsam, J.R., Maier-Hein, K., Eslami, S.M.A., Jimenez Rezende, D., Ronneberger, O.: A probabilistic u-net for segmentation of ambiguous images. In: International Conference on Neural Information Processing Systems. vol. 31. Curran Associates, Inc. (2018)
- Kohlberger, T., Singh, V., Alvino, C., Bahlmann, C., Grady, L.: Evaluating segmen- tation error without ground truth. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 528-536. Springer (2012)
- Li, Z., Kamnitsas, K., Islam, M., Chen, C., Glocker, B.: Estimating model perfor- mance under domain shifts with class-specific confidence scores. In: Medical Image Computing and Computer Assisted Intervention -MICCAI 2022. pp. 693-703. Springer Nature Switzerland, Cham (2022)
- Lin, Q., Chen, X., Chen, C., Garibaldi, J.M.: A novel quality control algorithm for medical image segmentation based on fuzzy uncertainty. IEEE Transactions on Fuzzy Systems pp. 2532 -2544 (2022)
- Liu, Y., Ding, C., Tian, Y., Pang, G., Belagiannis, V., Reid, I., Carneiro, G.: Residual pattern learning for pixel-wise out-of-distribution detection in seman- tic segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1151-1161 (2023)
- Ng, M., Guo, F., Biswas, L., Wright, G.A.: Estimating uncertainty in neural net- works for segmentation quality control. In: 32nd International Conference on Neu- ral Information Processing Systems (NIPS 2018), Montréal, Canada, no. NIPS. pp. 3-6 (2018)
- Puyol-Antón, E., Ruijsink, B., Baumgartner, C.F., Masci, P.G., Sinclair, M., Konukoglu, E., Razavi, R., King, A.P.: Automated quantification of myocardial tis- sue characteristics from native t1 mapping using neural networks with uncertainty- based quality-control. Journal of Cardiovascular Magnetic Resonance 22(1), 60 (2020)
- Robinson, R., Oktay, O., Bai, W., Valindria, V.V., Sanghvi, M.M., Aung, N., Paiva, J.M., Zemrak, F., Fung, K., Lukaschuk, E., Lee, A.M., Carapella, V., Kim, Y.J., Kainz, B., Piechnik, S.K., Neubauer, S., Petersen, S.E., Page, C., Rueckert, D., Glocker, B.: Real-time prediction of segmentation quality. In: Medical Image Com- puting and Computer Assisted Intervention -MICCAI 2018. pp. 578-585. Springer International Publishing, Cham (2018)
- Sohn, K., Lee, H., Yan, X.: Learning structured output representation using deep conditional generative models. International Conference on Neural Information Processing Systems 28 (2015)
- Sunoqrot, M.R., Selnæs, K.M., Sandsmark, E., Nketiah, G.A., Zavala-Romero, O., Stoyanova, R., Bathen, T.F., Elschot, M.: A quality control system for automated prostate segmentation on t2-weighted mri. Diagnostics 10(9), 714 (2020)
- Valindria, V.V., Lavdas, I., Bai, W., Kamnitsas, K., Aboagye, E.O., Rockall, A.G., Rueckert, D., Glocker, B.: Reverse classification accuracy: predicting segmentation performance in the absence of ground truth. IEEE transactions on medical imaging 36(8), 1597-1606 (2017)
- Vovk, V., Gammerman, A., Shafer, G.: Algorithmic learning in a random world, vol. 29. Springer (2005)
- Williams, E., Niehaus, S., Reinelt, J., Merola, A., Mihai, P.G., Villringer, K., Thier- bach, K., Medawar, E., Lichterfeld, D., Roeder, I., et al.: Automatic quality control framework for more reliable integration of machine learning-based image segmen- tation into medical workflows. arXiv preprint arXiv:2112.03277 (2021) · arXiv
- Zhang, L., Gooya, A., Dong, B., Hua, R., Petersen, S.E., Medrano-Gracia, P., Frangi, A.F.: Automated quality assessment of cardiac mr images using convo- lutional neural networks. In: Simulation and Synthesis in Medical Imaging: First International Workshop, SASHIMI 2016, Held in Conjunction with MICCAI 2016, Athens, Greece, October 21, 2016, Proceedings 1. pp. 138-145. Springer (2016)
Cited by
Sorted by citations
ConfIC-RCA: Statistically Grounded Efficient Estimation of Segmentation Quality
Cosarinsky, M., Billot, R., Mansilla, L., Jimenez, G., Gaggión, N., Fu, G., Tirer, T., & Ferrante, E. (2025) · arXiv
“Unlike [35], we base our conformal prediction framework on Confor-malized Quantile Regression (CQR), leveraging its ability to produce tighter prediction intervals than alternative conformal approaches commonly used for regression [8], and introducing novel modifications tailored to the RCA…”
Influential
- Authors
- Wundram, A. M., Fischer, P., Muehlebach, M., Koch, L. M., & Baumgartner, C. F.
- Affiliations
- University of Tübingen (DE)
- University of Lucerne (CH)
- Max Planck Institute for Intelligent Systems (DE)
- University of Bern (CH)
- Corresponding author
- Wundram, Anna M.
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- Pages
- 81-91
- ISSN
- 0302-9743 (ISSN-L)
- arXiv
- 2407.13307
- OpenAlex
- W4403054739
- Cited by
- 3
- References
- 0
- Citations by year
- 2026: 2 · 2025: 1
- FWCI
- 0.62
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- Lecture notes in computer science (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- International Max Planck Research School for Environmental, Cellular and Molecular Microbiology
- International Max Planck Research School for Advanced Methods in Process and Systems Engineering
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Retinal Imaging and Analysis
- Keywords
- split conformal prediction, retinal vessel segmentation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
