Lost in the Folds: When Cross-Validation Is Not a Deep Ensemble for Uncertainty Estimation

In medical imaging, rare structures, underrepresented pathologies, or uneven fold composition can cause elevated disagreement that reflects missing training examples rather than genuine uncertainty under the full data distribution.

“This distinction is particularly relevant in medical imaging, where rare structures, underrepresented pathologies, or uneven fold composition can cause elevated disagreement that reflects missing training examples rather than genuine uncertainty under the full data distribution [21].”

• cites [21] 21. Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J., Lakshminarayanan, B., Snoek, J.: Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in Neural Information Processing Systems 32 (2019) sourcePartly supported

  The Introduction (p. 2) says epistemic uncertainty can be reduced by collecting additional data, and the Background (p. 3) discusses uncertainty under dataset shift. The paper does not establish the claimed distinction between model disagreement caused by missing training examples and uncertainty under the full data distribution, or examine rare medical structures, pathologies, or uneven folds. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

In high-capacity neural networks, deep ensembles consistently outperform alternatives such as Monte Carlo dropout in terms of calibration, robustness under dataset shift, and failure detection.

“In high-capacity neural networks, deep ensembles consistently outperform alternatives such as Monte Carlo dropout in terms of calibration, robustness under dataset shift, and failure detection [21].”

• cites [21] 21. Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J., Lakshminarayanan, B., Snoek, J.: Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in Neural Information Processing Systems 32 (2019) sourcePartly supported

  Section 4.2 (p. 7) reports that ensembles outperform dropout across metrics on CIFAR-10 and ImageNet, and Appendix F (p. 24) finds a better confidence–accuracy trade-off for ensembles in an out-of-distribution detection test. But the claim's unqualified “consistently” overstates the results: Section 4.2 says SVI performs best on MNIST, while Appendix C (p. 20) reports that dropout performs best on calibration error for Criteo. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

All tasks in the study provide independent rater annotations, enabling ambiguity analysis.

“All tasks provide independent rater annotations, enabling ambiguity analysis.”

• cites [20] 20. Nyholm, T., Svensson, S., Andersson, S., Jonsson, J., Sohlin, M., Gustafsson, C., Kjellén, E., Söderström, K., Albertsson, P., Blomqvist, L., Zackrisson, B., Olsson, L.E., Gunnlaugsson, A.: MR and CT data with multiobserver delineations of organs in the pelvic area-part of the gold atlas project. Medical Physics 45 (3), 12951300 (2018). https://doi.org/10.1002/mp.12748 , https://zenodo.org/record/ 583096 , dataset also available at Zenodo; multi-rater pelvic organ delineations sourceSupported
• cites [22] 22. Riera-Marín, M., Kleiß, J.M., Aubanell, A., Antolín, A.: CURVAS dataset: Calibration and uncertainty for multirater volume assessment in multi-organ segmentation (2024). https://doi.org/10.5281/zenodo.11147559 , https://zenodo.org/ records/11147559 , 90 CT scans with multi-rater segmentations of pancreas, liver, and kidneys sourceSupported
• cites [2] 2. Almazroa, A., Alodhayb, S., Osman, E., Ramadan, E., Hummadi, M., Dlaim, M., Alkatee, M., Raahemifar, K., Lakshminarayanan, V.: Retinal fundus images for glaucoma analysis: the RIGA dataset (2018). https://doi.org/10.7302/Z23R0R29 , retinal fundus images with optic disc and optic cup annotations by six ophthalmologists sourceNot checked — no full text of the source

Disagreement among CV models is increasingly reused as a proxy for epistemic uncertainty in downstream applications such as quality control and human-in-the-loop review.

“Increasingly, disagreement among these CV models is reused as a proxy for epistemic uncertainty in downstream applications such as quality control and human-in-the-loop review [11,18,15].”

• cites [11] 11. Jungo, A., Balsiger, F., Reyes, M.: Analyzing the quality and challenges of uncertainty estimations for brain tumor segmentation. Frontiers in neuroscience 14 , 282 (2020) sourcePartly supported

  The Introduction describes uncertainty estimates as useful for visual feedback, guided corrections, and patient-level segmentation failure detection. Section 2.3.4 uses an ensemble of differently trained models to estimate uncertainty, but measures entropy of their averaged probabilities rather than explicitly measuring model disagreement. The paper does not establish that this practice is increasingly reused across computer-vision applications or identify the resulting estimate specifically as epistemic uncertainty; its Discussion also cautions against using voxel-wise uncertainty for guided corrections without better calibration. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites [18] 18. Mehrtash, A., Wells, W.M., Tempany, C.M., Abolmaesumi, P., Kapur, T.: Confidence calibration and predictive uncertainty estimation for deep medical image segmentation. IEEE Transactions on Medical Imaging 40 (12), 3436-3447 (2021). https://doi.org/10.1109/TMI.2021.3122681 sourcePartly supported

  The Contributions, Results and Discussion (page numbers unavailable) describe model ensembles whose predictive uncertainty helps assess segmentation quality and detect failures; the Introduction also proposes seeking human intervention rather than allowing a silent failure. But the paper uses entropy of ensemble predictions, not disagreement among models specifically as a proxy for epistemic uncertainty. It does not establish that this practice is increasingly reused across CV applications or document human-in-the-loop review workflows. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites [15] 15. Khalili, N., Spronck, J., Ciompi, F., van der Laak, J., Litjens, G.: Uncertainty-guided annotation enhances segmentation with the human-in-the-loop. arXiv preprint arXiv:2404.07208 (2024) sourcePartly supported

  Section 3 measures uncertainty by disagreement among five cross-validated models, and the Abstract describes its use for human-in-the-loop review and automated quality control. The copy does not identify that uncertainty specifically as epistemic or establish that this use is increasing across downstream applications. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

In recent medical image segmentation uncertainty studies, terminology-implementation mismatches are common, with 'deep ensemble' frequently used to describe a K-fold CV ensemble despite members being trained on different data subsets.

“We audit recent segmentation uncertainty studies and find that terminology-implementation mismatches are common.”

• Also stated in: § 1 Introduction · p. 2 · plain

  “We document, via a survey of uncertainty in segmentation papers, that the term deep ensemble is frequently used to describe a K-fold CV ensemble, similar to the default nnU-Net behavior, despite its members being trained on different data subsets (Table 1).”

• Also stated in: § 2 Related Work · p. 4 · plain

  “However, as shown in Table 1, several recent studies implicitly or explicitly interpret disagreement among nnU-Net CV models as epistemic uncertainty, often referring to these as DE [11,3,1,15,6,24,23].”

A model may appear uncertain due to limited exposure to specific training examples, rather than uncertainty under the full-data distribution.

“A model may therefore appear uncertain due to limited exposure to specific training examples, rather than uncertainty under the full-data distribution.”

In a CV ensemble, each model is trained on a different subset of the training data, so disagreement within a CV ensemble conflates epistemic uncertainty with variability induced by incomplete or heterogeneous data exposure.

“In contrast, each model in a CV ensemble is trained on a different subset of the training data. As a result, disagreement within a CV ensemble conflates epistemic uncertainty with variability induced by incomplete or heterogeneous data exposure.”