CoChat
Sign Up
Cited by 154Medical Image Segmentation TechniquesArtificial Intelligence in Healthcare and EducationMedical Imaging and Analysis

Reverse Classification Accuracy: Predicting Segmentation Performance in the Absence of Ground Truth

Open access
Valindria, Vanya V., Lavdas, Ioannis, Bai, Wenjia, Kamnitsas, Konstantinos, Aboagye, Eric O., Rockall, Andrea G., Rueckert, Daniel, Glocker, Ben·2017·IEEE Transactions on Medical Imaging
DOI 10.1109/tmi.2017.2665165
Open this paper in CoChat — free account

Abstract

When integrating computational tools, such as automatic segmentation, into clinical practice, it is of utmost importance to be able to assess the level of accuracy on new data and, in particular, to detect when an automatic method fails. However, this is difficult to achieve due to the absence of ground truth. Segmentation accuracy on clinical data might be different from what is found through cross validation, because validation data are often used during incremental method development, which can lead to overfitting and unrealistic performance expectations. Before deployment, performance is quantified using different metrics, for which the predicted segmentation is compared with a reference segmentation, often obtained manually by an expert. But little is known about the real performance after deployment when a reference is unavailable. In this paper, we introduce the concept of reverse classification accuracy (RCA) as a framework for predicting the performance of a segmentation method on new data. In RCA, we take the predicted segmentation from a new image to train a reverse classifier, which is evaluated on a set of reference images with available ground truth. The hypothesis is that if the predicted segmentation is of good quality, then the reverse classifier will perform well on at least some of the reference images. We validate our approach on multi-organ segmentation with different classifiers and segmentation methods. Our results indicate that it is indeed possible to predict the quality of individual segmentations, in the absence of ground truth. Thus, RCA is ideal for integration into automatic processing pipelines in clinical routine and as a part of large-scale image analysis studies.

AI snapshot

Written from the paper’s full text.

TL;DR

The paper introduces reverse classification accuracy (RCA), a framework that predicts per-case segmentation quality without ground truth by training a classifier on a predicted segmentation as pseudo-GT and evaluating it on reference images with known GT. Across multi-organ MRI experiments, RCA with Single-Atlas label propagation best predicted Dice scores, while distance-based metrics and constrained CNNs were less reliable.

Key findings

  • RCA can predict individual segmentation accuracy without ground truth. Best results used Single-Atlas as the RCA classifier: correlations of 0.955, 0.973, and 0.962 for Random Forests, CNNs, and Multi-Atlas segmentation methods, with MAEs of 0.051, 0.052, and 0.067 and 3-category accuracies of 0.888, 0.811, and 0.822.
  • Atlas Forests also performed well: correlations of 0.881, 0.828, and 0.863, MAEs of 0.120, 0.166, and 0.168, and 3-category accuracies of 0.783, 0.623, and 0.749 for the same three segmentation methods. Constrained CNNs were least suitable, with correlations below 0.78 and MAEs above 0.2.
  • In a failure-detection experiment using degraded Random Forests, Atlas Forests achieved correlation 0.853, MAE 0.096, and accuracy 0.884; Single-Atlas achieved correlation 0.875, MAE 0.097, and accuracy 0.928; constrained CNNs had correlation 0.237, MAE 0.139, and accuracy 0.924. The bladder was an exception where failed segmentations were not reliably detected.
  • RCA predicted overlap-based metrics better than distance-based metrics: DSC correlation 0.881, MAE 0.120, accuracy 0.783; Jaccard index correlation 0.899, MAE 0.110, accuracy 0.749; Hausdorff distance correlation 0.189, MAE 41.38, accuracy 0.387; average surface distance correlation 0.022, MAE 4.120, accuracy 0.344; relative volume difference correlation 0.474, MAE 0.051, accuracy 0.678.

Methods

  • Whole-body multi-sequence MRI (T1w Dixon and T2w) from 35 healthy volunteers in the MALIBO study, with manual segmentations of 15 anatomical structures, yielding 525 structures (15 organs × 35 subjects). Evaluation used 3-fold cross-validation.
  • Three segmentation methods were tested: Random Forests, 3D CNNs (DeepMedic), and probabilistic multi-atlas label propagation. Three RCA classifiers were tested: Atlas Forests, constrained CNNs, and Single-Atlas label propagation, giving nine combinations.
  • Prediction accuracy was quantified by correlation between predicted and real Dice similarity coefficient (DSC), mean absolute error (MAE), and 3-category classification accuracy using DSC ranges [0.0, 0.6) for bad, [0.6, 0.8) for medium, and [0.8, 1.0] for good. The RCA proxy was the maximum DSC across reference images. A leave-one-subject-out Random Forest regression calibration experiment was also run.

Limitations

  • The paper states that RCA assumes a linear relationship between predicted and real scores close to identity, which was only found with Single-Atlas; off-diagonal correlation for Atlas Forests may require calibration, but calibration risks overfitting. It also states that distance-based metrics such as Hausdorff distance and average surface distance cannot be accurately predicted with the current setting, and that constrained CNNs trained on single images only work for major organs and struggle with small structures, producing many zero predictions.
  • Evident from the text: validation used 35 healthy volunteers rather than clinical or pathological data, and RCA requires a reference database with available ground truth. The 3-category DSC thresholds are described as somewhat arbitrary and would need adjustment for each application.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/tmi.2017.2665165

Cites

46 of 46 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

7 of 22 cited sources checked · 3 supported · 0 flagged · accuracy not yet measured

  • Premisenone given
    § A. Related Work · p. 2

    Existing methods for evaluating segmentation without ground truth cannot be applied to a general task with many different class labels.

    “However, these methods cannot be applied to a more general task, such as an image with many different class labels to be segmented.”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites [18] [18] P. L. Correia and F. Pereira, 'Stand-alone objective segmentation quality evaluation,' EURASIP Journal on Applied Signal Processing , vol. 2002, no. 1, pp. 389-400, 2002. - source

      The Introduction and Section 3.2 say that stand-alone evaluation works best in constrained scenarios and that a single general-purpose composite metric cannot be established. They do not identify a large number of class labels as the limitation; Section 3.2 also says metrics can be adapted for content outside its two proposed classes. The categorical claim that these methods cannot be applied to such an image therefore goes beyond the paper. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites [19] [19] F. Ge, S. Wang, and T. Liu, 'New benchmark for image segmentation evaluation,' Journal of Electronic Imaging , vol. 16, no. 3, pp. 033 011033 011, 2007. - source
    • cites [20] [20] L. Goldmann, T. Adamek, P. Vajda, M. Karaman, R. M¨ orzinger, E. Galmar, T. Sikora, N. E. O?Connor, T. Ha-Minh, T. Ebrahimi et al. , 'Towards fully automatic image segmentation evaluation,' in International Conference on Advanced Concepts for Intelligent Vision Systems . Springer, 2008, pp. 566-577. - source
  • Premisereasoning
    § A. Related Work · p. 2

    Approaches [31], [33] can only be applied when a training database with good and bad segmentations is available.

    “However, the approaches [31], [33] can only be applied when a training database with good and bad segmentations is available from which a mapping from features to segmentation accuracy can be learned.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [31] [31] T. Kohlberger, V. Singh, C. Alvino, C. Bahlmann, and L. Grady, 'Evaluating segmentation error without ground truth,' in International Conference on Medical Image Computing and Computer-Assisted Intervention . Springer, 2012, pp. 528-536. - source
    • cites [33] [33] K. Frounchi, L. C. Briand, L. Grady, Y. Labiche, and R. Subramanyan, 'Automating image segmentation verification and validation by learning test oracles,' Information and Software Technology , vol. 53, no. 12, pp. 1337-1348, 2011. - source
  • Interpretationreasoning
    § IV. DISCUSSION AND CONCLUSION · p. 7

    Unlike the supervised methods in [33] and [31], RCA requires no training data that captures examples of good and bad segmentations.

    “An appealing property of the proposed framework is that unlike the supervised methods in [33] and [31] no training data is required that captures examples of good and bad segmentations.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [33] [33] K. Frounchi, L. C. Briand, L. Grady, Y. Labiche, and R. Subramanyan, 'Automating image segmentation verification and validation by learning test oracles,' Information and Software Technology , vol. 53, no. 12, pp. 1337-1348, 2011. - source
    • cites [31] [31] T. Kohlberger, V. Singh, C. Alvino, C. Bahlmann, and L. Grady, 'Evaluating segmentation error without ground truth,' in International Conference on Medical Image Computing and Computer-Assisted Intervention . Springer, 2012, pp. 528-536. - source
  • Premisecitation
    § B. Contribution · p. 2

    RCA differs from reverse validation and reverse testing in that it trains a reverse classifier on every single instance, whereas those approaches train single classifiers over the whole test set jointly.

    “While the idea of RCA is similar to reverse validation [34] and reverse testing [35], the important difference is that in our approach we train a reverse classifier on every single instance while the approaches in [34], [35] train single classifiers over the whole test set and its predictions jointly to find out what the best original predictor is.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [34] [34] E. Zhong, W. Fan, Q. Yang, O. Verscheure, and J. Ren, 'Cross validation framework to choose amongst models and datasets for transfer learning,' in Joint European Conference on Machine Learning and Knowledge Discovery in Databases . Springer, 2010, pp. 547-562. - source
    • cites [35] [35] W. Fan and I. Davidson, 'Reverse testing: an efficient framework to select amongst classifiers under sample selection bias,' in Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining . ACM, 2006, pp. 147-156. - source
  • Findingown data
    § Reverse Classification Accuracy: Predicting Segmentation Performance in the Absence of Ground Truth · p. 1

    It is possible to predict the quality of individual segmentations in the absence of ground truth.

    “Our results indicate that it is indeed possible to predict the quality of individual segmentations, in the absence of ground truth.”
    • Also stated in: § B. Contribution · p. 2 · hedged
      “Our results indicate that, at least to some extent, it is indeed possible to predict the performance level of a segmentation method on each individual case, in the absence of ground truth.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: A suitable reference database containing multiple segmented images that capture expected variability is available.
    • The authors state this as an assumption: The RCA classifier works best on images that are somewhat similar to the image I.
    • The authors state this as an assumption: There is a correlation between the segmentation metric values computed on reference images and the values one would get for the new image if ground truth were available.
    • The authors state this as an assumption: The maximum DSC score over all reference images correlates well with the real DSC on the new image if ground truth were available.
    • The authors state this as an assumption: The RCA classifier performs well on at least some reference images if and only if the predicted segmentation S_I is of good quality, and performs poorly on all reference images if S_I is of bad quality.
  • Classified as probably not a substantial claim. 19
  • May combine two claims. 21

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Premisecitation
    § A. Related Work · p. 2

    The application of unsupervised methods in medical settings is unclear.

    “While unsupervised methods can be applied to scenarios where the main purpose of segmentation is to yield visually consistent results that are meaningful to a human observer, the application in medical settings is unclear.”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites [24] [24] S. Chabrier, B. Emile, C. Rosenberger, and H. Laurent, 'Unsupervised performance evaluation of image segmentation,' EURASIP Journal on Applied Signal Processing , vol. 2006, pp. 217-217, 2006. - source

      The Introduction says supervised evaluation criteria are widely used in medical applications, while Section 2 describes unsupervised criteria in relation to human perception. The paper’s real-image examples are aerial and radar images (Section 4). None of these passages addresses whether unsupervised methods can be applied in medical settings or says that their application there is unclear. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
    • cites [25] [25] H. Zhang, J. E. Fritts, and S. A. Goldman, 'Image segmentation evaluation: A survey of unsupervised methods,' Computer Vision and Image Understanding , vol. 110, no. 2, pp. 260-280, 2008. - source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § B. Quantifying Prediction Accuracy · p. 5

    Dice's similarity coefficient is the most widely used measure for evaluating segmentation performance.

    “The Dice's similarity coefficient is the most widely used measure for evaluating segmentation performance 5 , and in our main results we focus on evaluating how well DSC can be predicted using our RCA framework.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [11] [11] E. Konukoglu, B. Glocker, D. H. Ye, A. Criminisi, and K. M. Pohl, 'Discriminative segmentation-based evaluation through shape dissimilarity,' IEEE Transactions on Medical Imaging , vol. 31, no. 12, pp. 2278-2289, 2012. - source

      The Introduction (page unavailable) describes Dice’s similarity coefficient as an example of a popular, commonly used segmentation-based score. Section II (page unavailable) lists it among several popular measures, but does not establish that it is the most widely used. (LLM, conf=0.85)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § A. Related Work · p. 2

    Reverse validation [34] is based on reverse testing [35], where a new classifier is trained on predictions on test data and evaluated on training data.

    “The basic idea of reverse validation [34] is based on reverse testing [35], where a new classifier is trained on predictions on the test data and evaluated again on the training data.”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites [34] [34] E. Zhong, W. Fan, Q. Yang, O. Verscheure, and J. Ren, 'Cross validation framework to choose amongst models and datasets for transfer learning,' in Joint European Conference on Machine Learning and Knowledge Discovery in Databases . Springer, 2010, pp. 547-562. - source

      Section 3.1 (page unavailable) describes training a new model on predicted labels for target-domain data, then using it to classify held-out source-domain data. But the paper calls this reverse validation. Section 6 (page unavailable) identifies reverse testing as a different technique from Fan and Davidson (2006); it does not establish that reverse validation is based on it. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites [35] [35] W. Fan and I. Davidson, 'Reverse testing: an efficient framework to select amongst classifiers under sample selection bias,' in Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining . ACM, 2006, pp. 147-156. - source

  • May combine two claims.

    Findingown data
    § C. Results for Predicting Dice's Similarity Coefficients · p. 5

    Constrained CNNs work for predicting segmentation accuracy for major organs such as liver, lungs, and spine but struggle with smaller structures, leading to many zero predictions even when real DSC is rather high.

    “The constrained CNNs seem to only work for predicting segmentation accuracy in case of major organs such as liver, lungs, and the spine but clearly struggle with smaller structures leading to many zero predictions even when the real DSC is rather high.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § IV. DISCUSSION AND CONCLUSION · p. 7

    Both MAEs and classification accuracies improve significantly for the case of Atlas Forests and constrained CNNs after calibration.

    “Both the MAEs and classification accuracies improve significantly for the case of Atlas Forests and constrained CNNs.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § IV. DISCUSSION AND CONCLUSION · p. 9

    Atlas Forests and single-atlas label propagation make highly accurate predictions in the low DSC ranges and thus are able to correctly detect failed segmentations, with the exception of the bladder.

    “Atlas Forests (left) and single-atlas label propagation (right) make highly accurate predictions in the low DSC ranges and thus, are able to correctly detect such failed segmentations, with the exception of the bladder.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § E. Results for Predicting Different Segmentation Metrics · p. 6

    Prediction accuracy for precision is lower than for recall.

    “Prediction accuracy for precision is lower than for recall.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § D. Detecting Segmentation Failure · p. 6

    Atlas Forests and Single-Atlas result in high correlations, low MAEs and very good classification accuracies for detecting segmentation failure.

    “Atlas Forests and Single-Atlas, however, result in high correlations, low MAEs and very good classification accuracies.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § C. Results for Predicting Dice's Similarity Coefficients · p. 5

    For constrained CNNs as RCA classifier, prediction quality is lowest, with correlations below 0.78 and MAEs above 0.2.

    “For the case of the constrained CNNs, we observe that the prediction quality is lowest confirmed by the scatter plots and all quantitative measures, with correlations below 0.78 and MAEs above 0.2.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § D. Detecting Segmentation Failure · p. 6

    Low real DSC scores are correctly predicted and failed segmentations are identified.

    “Low real DSC scores are correctly predicted and failed segmentations are identified.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § D. Detecting Segmentation Failure · p. 6

    All three segmentation methods perform reasonably well on most major organs with no failure cases among structures such as liver, heart, and lungs.

    “From the scatter plots in Fig. 2 we can see that all three segmentation methods perform reasonably well on most major organs with no failure cases among structures such as liver, heart, and lungs.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given
    § IV. DISCUSSION AND CONCLUSION · p. 7

    A constrained CNN trained on single images only works well for major organs such as liver, lungs and spine.

    “We have also experimented with a constrained CNN trained on single images, which only works well for major organs such as liver, lungs and spine.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § E. Results for Predicting Different Segmentation Metrics · p. 7

    Both Atlas Forests and constrained CNNs work best for larger organs such as liver, lungs, and pelvis while leading to many zero predictions for smaller structures such as adrenal gland and clavicles.

    “Both, Atlas Forests and constrained CNNs work best for larger organs such as liver, lungs, and pelvis while leading to many zero predictions for smaller structures such as adrenal gland and clavicles.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § D. Detecting Segmentation Failure · p. 6

    The constrained CNNs are again suffering from many zero predictions and are less suitable for making accurate predictions.

    “The constrained CNNs are again suffering from many zero predictions and less suitable for making accurate predictions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § E. Results for Predicting Different Segmentation Metrics · p. 6

    Good correlation is obtained between predicted and real overlap based scores, with low MAEs, and high accuracies.

    “Good correlation is obtained between predicted and real overlap based scores, with low MAEs, and high accuracies.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § C. Results for Predicting Dice's Similarity Coefficients · p. 6

    For Atlas Forests and Single-Atlas, the effect of zero predictions on accuracies is very little, confirming that both are well suited within the RCA framework, independent of the original segmentation method.

    “For Atlas Forests and Single-Atlas the effect on the accuracies is very little, confirming that those both are well suited within the RCA framework, independent of the original segmentation method.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § E. Results for Predicting Different Segmentation Metrics · p. 7

    The predictions for RVD are good, with high classification accuracy of 0.68%, similar to the overlap based scores.

    “The predictions for RVD are good, with high classification accuracy of 0.68%, similar to the overlap based scores.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given
    § I. INTRODUCTION · p. 1

    Clinical data may differ from annotated data due to varying imaging protocols or artefacts caused by pathology.

    “Second, the clinical data might be different due to varying imaging protocols or artefacts caused by pathology.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationnone given
    § Reverse Classification Accuracy: Predicting Segmentation Performance in the Absence of Ground Truth · p. 1

    Reverse classification accuracy (RCA) is a framework for predicting the performance of a segmentation method on new data.

    “we introduce the concept of reverse classification accuracy (RCA) as a framework for predicting the performance of a segmentation method on new data.”
    • Also stated in: § B. Contribution · p. 2 · plain
      “The main contribution of this paper is the introduction of the concept of reverse classification accuracy (RCA) to assess the segmentation quality of an individual image in the absence of GT.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: A suitable reference database containing multiple segmented images that capture expected variability is available.
    • The authors state this as an assumption: The RCA classifier works best on images that are somewhat similar to the image I.
    • The authors state this as an assumption: There is a correlation between the segmentation metric values computed on reference images and the values one would get for the new image if ground truth were available.
    • The authors state this as an assumption: The maximum DSC score over all reference images correlates well with the real DSC on the new image if ground truth were available.
    • The authors state this as an assumption: The RCA classifier performs well on at least some reference images if and only if the predicted segmentation S_I is of good quality, and performs poorly on all reference images if S_I is of bad quality.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § A. Related Work · p. 2

    The region-correlation matrix method [30] cannot evaluate segmentation performance of a particular method on a particular image independently.

    “Another work by [30] has quantitatively evaluated the performance of several segmentation algorithms by regioncorrelation matrix. The limitation of this work is that it cannot evaluate the segmentation performance of a particular method on a particular image independently.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [30] [30] K. Sikka and T. M. Deserno, 'Comparison of algorithms for ultrasound image segmentation without ground truth,' in SPIE Medical Imaging . International Society for Optics and Photonics, 2010, pp. 76 271C76 271C. - source

  • Classified as probably not a substantial claim.

    Study design and conductown data
    § III. EXPERIMENTAL VALIDATION · p. 5

    Manual segmentations generated by clinical experts serve as ground truth in the quantitative evaluation.

    “Detailed manual segmentations of 15 anatomical structures, including abdominal organs (heart, left/right lung, liver, adrenal gland, gall bladder, left/right kidney, spleen, pancreas, bladder) and bones (spine, left/right clavicle, pelvis) have been generated by clinical experts as part of the study. These manual segmentations will serve as GT in the quantitative evaluation.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given
    § C. Results for Predicting Dice's Similarity Coefficients · p. 5

    The constrained CNNs' struggle with smaller structures is most likely caused by the difficulty of training the CNNs with single images and small structures, which does not provide sufficient amounts of training data.

    “This is most likely caused by the difficulty of training the CNNs with single images and small structures which does not provide sufficient amounts of training data.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § B. Predicting Segmentation Accuracy · p. 4

    Mean or median scores are not useful measures because the RCA classifier is not expected to work well on the majority of reference images.

    “Note, that the mean or median scores are not very useful measures as we do not expect the RCA classifier to work well on the majority of the reference images.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The RCA classifier works best on images that are somewhat similar to the image I.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given
    § B. Predicting Segmentation Accuracy · p. 4

    The RCA classifier overfits to the single image and will not generalize to perform well on dissimilar images.

    “Afterall, the RCA classifier does overfit to the single image and will not generalize to perform well on dissimilar images.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: The RCA classifier works best on images that are somewhat similar to the image I.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning
    § A. Learning Reverse Classifiers · p. 3

    Reducing the number of filters substantially decreases the number of parameters, leading to faster computations and reduced overfitting when trained on a single image.

    “With less number of filters, the number of parameters is substantially decreased, which leads to faster computations, but more importantly, reduces overfitting when trained on a single image.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § E. Results for Predicting Different Segmentation Metrics · p. 7

    RCA works very well for overlap based measures and for RVD to some extent, while distance based metrics cannot be accurately predicted with the current setting and would require further investigation.

    “In conclusion, it seems RCA works very well for overlap based measures and for RVD to some extent, while distance based metrics cannot be accurately predicted with the current setting and would require further investigation.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisereasoning
    § A. Learning Reverse Classifiers · p. 3

    Random Forests operate as voxel-wise classifiers and do not necessarily require preregistration of images at training or testing time.

    “Since they operate as voxel-wise classifiers, they do not (necessarily) require preregistration of the images neither at training nor testing time.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [36] [36] D. Zikic, B. Glocker, and A. Criminisi, 'Encoding atlases by randomized classification forests for efficient multi-atlas label propagation,' Medical Image Analysis , vol. 18, no. 8, pp. 1262-1273, 2014. - source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § IV. DISCUSSION AND CONCLUSION · p. 7

    Calibration comes with a risk of overfitting as the method will learn the relationship on the available training data but might not generalize to new data.

    “Calibration, however, comes with a risk of overfitting as the method will learn the relationship on the available training data but might not generalize to new data.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § A. Learning Reverse Classifiers · p. 3

    For RCA with single atlas label propagation, no label fusion is required.

    “For the purpose of RCA, however, there is only a single atlas and thus no label fusion is required.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § C. Results for Predicting Dice's Similarity Coefficients · p. 6

    Because the real DSC for these structures is zero with no voxels labelled in the segmentation map, the RCA predictions are always correct as there are no labels for the RCA classifier for this structure.

    “Because the real DSC for these is zero with no voxels labelled in the segmentation map, the RCA predictions are always correct as there are no labels for the RCA classifier for this structure.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § A. Learning Reverse Classifiers · p. 3

    Reducing feature maps without changing the number of layers preserves the network's capability to see large image context.

    “By reducing the feature maps without changing the architecture in terms of number of layers the network preserves its capability to see large image context as the size of the receptive field remains unchanged.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning
    § B. Quantifying Prediction Accuracy · p. 5

    The three-category DSC ranges are somewhat arbitrary and would need to be adjusted specifically to the application at hand.

    “Note, that those ranges are somewhat arbitrary, in particular, as the quality of absolute DSC values is highly depending on the structure of interest. So in practice, those ranges would need to be adjusted specifically to the application at hand.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § E. Results for Predicting Different Segmentation Metrics · p. 6

    Precision and recall capture different parts of segmentation error; under-segmentation is not reflected in precision, while over-segmentation is not captured in recall.

    “The two metrics capture different parts of segmentation error; under-segmentation is not reflected in precision, while over-segmentation is not captured in recall”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [7] [7] W. R. Crum, O. Camara, and D. L. Hill, 'Generalized overlap measures for evaluation and validation in medical image analysis,' IEEE Transactions on Medical Imaging , vol. 25, no. 11, pp. 1451-1461, 2006. - source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § A. Related Work · p. 2

    The generative model approach [32] cannot directly predict segmentation scores such as DSC.

    “Such an approach cannot not be used to directly predict segmentation scores such as DSC, but can be useful to inform automatic quality control or to automatically select the best segmentation from a set of candidates.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [32] [32] L. Grady, V. Singh, T. Kohlberger, C. Alvino, and C. Bahlmann, 'Automatic segmentation of unknown objects, with application to baggage security,' in European Conference on Computer Vision (ECCV) . Springer, 2012, pp. 430-444. - source

  • May combine two claims.

    Interpretationreasoning
    § B. Contribution · p. 2

    RCA allows predicting accuracy for each individual case and aggregating predictions to draw conclusions for overall performance of a segmentation method.

    “RCA has the advantage of allowing to predict the accuracy for each individual case, while at the same time aggregating over such accuracy predictions allows drawing conclusions for the overall performance of a particular segmentation method.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: A suitable reference database containing multiple segmented images that capture expected variability is available.
    • The authors state this as an assumption: The RCA classifier works best on images that are somewhat similar to the image I.
    • The authors state this as an assumption: There is a correlation between the segmentation metric values computed on reference images and the values one would get for the new image if ground truth were available.
    • The authors state this as an assumption: The maximum DSC score over all reference images correlates well with the real DSC on the new image if ground truth were available.
    • The authors state this as an assumption: The RCA classifier performs well on at least some reference images if and only if the predicted segmentation S_I is of good quality, and performs poorly on all reference images if S_I is of bad quality.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Interpretationnone given
    § IV. DISCUSSION AND CONCLUSION · p. 7

    The RCA framework is generic, so other methods can be considered and it might be necessary to select the most appropriate one for the application at hand.

    “As the RCA framework is generic, other methods can be considered and it might be necessary to select the most appropriate one for the application at hand.”
    Quote in the paperNot a substantial claim
36 of 36

  • The authors state this as an assumption.

    A suitable reference database containing multiple segmented images that capture expected variability is available.

    “we further assume that a suitable reference database is available that contains multiple segmented images (or atlases) T = { ( J k , S GT J k ) } m k =1 that capture the expected variability.”
    “The assumption is that in machine learning approaches, such a database is usually already available, but it could also be specifically constructed for the purpose of RCA.”

  • The authors state this as an assumption.

    The RCA classifier works best on images that are somewhat similar to the image I.

    “In fact, the RCA classifier f I,S I is assumed to work best on images that are somewhat similar to I .”

  • The authors state this as an assumption.

    There is a correlation between the segmentation metric values computed on reference images and the values one would get for the new image if ground truth were available.

    “The underlying hypothesis in our RCA framework is that there is a correlation between the values computed on ( S J , S GT J ) and the values one would get for the pair ( S I , S GT I ) , where S GT I is the reference GT of image I which in practice, however, is unavailable.”

  • The authors state this as an assumption.

    The maximum DSC score over all reference images correlates well with the real DSC on the new image if ground truth were available.

    “It is expected that the maximum DSC score over all reference images correlates well with the real DSC that one would get on the new image if GT were available.”

  • The authors state this as an assumption.

    The RCA classifier performs well on at least some reference images if and only if the predicted segmentation S_I is of good quality, and performs poorly on all reference images if S_I is of bad quality.

    “When testing the RCA classifier on all of the available m reference images, we expect that the RCA classifier performs well on at least some of these, if and only if the predicted segmentation S I is of good quality. If S I is of bad quality, we expect the RCA classifier to perform poorly on all reference images.”

  • The authors state this as an assumption.

    There is a linear relationship between predicted and real scores which should be close to an identity mapping.

    “The drawback, however, is that we assume a linear relationship between predicted and real scores which should be close to an identity mapping”

  • The paper relies on this without support.

    DSC is the most widely considered segmentation evaluation measure.

    “Still, DSC is the most widely considered measure”

  • The paper relies on this without support.

    Manual quality control by visual inspection is not feasible when thousands of images are automatically processed for the purpose of deriving population statistics.

    “In settings where thousands of images are automatically processed for the purpose of deriving population statistics, it is not feasible to employ manual quality control with visually inspection of the segmentation results.”
8 of 8

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The distribution of RCA scores over the reference set serves as a proxy for the quality of the original segmentation.

    “The distribution of scores over the reference set serves as a proxy for the quality of the original segmentation.”

    From ConfIC-RCA: Statistically Grounded Efficient Estimation of Segmentation Quality (2025) · cites this paper as [5]

    The Results (p. 5) show distributions of scores across the reference set, but say that selecting the maximum score is the proxy for segmentation quality. Predicting Segmentation Accuracy (p. 4) likewise specifies the maximum, noting that mean and median scores are not useful. The claim captures the use of reference-set scores but is imprecise about which aspect serves as the proxy. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • RCA tends to underestimate segmentation quality, making higher scores more informative.

    “This asymmetry captures the observation that RCA tends to underestimate segmentation quality, making higher scores more informative [5].”

    From ConfIC-RCA: Statistically Grounded Efficient Estimation of Segmentation Quality (2025) · cites this paper as [5]

    In “Quantifying Prediction Accuracy” (p. 5), the paper says that a predicted score consistently lower than the real score could still be useful, but presents this as a hypothetical bias, not an observed tendency. Its results (p. 5) describe one RCA classifier’s predictions as close to the real scores, while the Discussion (p. 8) says RCA can identify high-quality segmentations. The paper therefore supports the usefulness of high scores, but not the claimed general tendency to underestimate quality or the conclusion that high scores are more informative than low ones. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Reverse Classification Accuracy (RCA) is a framework designed to predict the performance of an image segmentation method when ground-truth masks are unavailable.

    “Reverse Classification Accuracy (RCA) [5] is a framework originally designed to predict the performance of an image segmentation method in scenarios where GT masks are unavailable.”

    From ConfIC-RCA: Statistically Grounded Efficient Estimation of Segmentation Quality (2025) · cites this paper as [5]

  • RCA works by using the predicted segmentation of a new image as pseudo-ground truth to train a reverse classifier, applying it to reference images with known ground truth, and evaluating performance with metrics like DSC.

    “It works by using the predicted segmentation of a new image as a pseudo-ground truth to train a reverse classifier (a segmentation network in our case) 1 . This model is then applied to a set of reference images with known GT, and its performance is evaluated using standard metrics like the DSC.”

    From ConfIC-RCA: Statistically Grounded Efficient Estimation of Segmentation Quality (2025) · cites this paper as [5]

  • Unlike previous studies that explored RCA focusing on cardiac MRI or multi-organ segmentation on whole-body MRI only, this work broadens the analysis to a wider range of medical imaging challenges.

    “Unlike previous studies that explored RCA focusing on cardiac MRI [28], [29] or multi-organ segmentation on whole-body MRI [5] only, we broaden our analysis to a wider range of medical imaging challenges.”

    From ConfIC-RCA: Statistically Grounded Efficient Estimation of Segmentation Quality (2025) · cites this paper as [5]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • RVD classification accuracy is misstated as 0.68%

    Minor

    Table III gives the RVD classification accuracy as 0.678, or 67.8%, rather than approximately 0.68%.

    Reported 0.68 · recomputed 67.8

  • Atlas Forests classification accuracies are reported as fractions of a percent

    Minor

    The cited table reports accuracies of 0.623, 0.749, and 0.783, corresponding to approximately 62%, 75%, and 78%, not 0.62%, 0.75%, and 0.78%.

    Reported 0.62 · recomputed 62.3

  • Calibration does not improve every Atlas Forests classification accuracy

    Minor

    The text says classification accuracies improve with calibration for Atlas Forests, but Table IV shows the Multi-Atlas/Atlas Forests accuracy falling from 0.749 to 0.745.

3 of 3

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Valindria, V. V., Lavdas, I., Bai, W., Kamnitsas, K., Aboagye, E. O., Rockall, A. G., Rueckert, D., & Glocker, B.
Affiliations
  • Imperial College London (GB)
  • Hammersmith Hospital (GB)
  • Royal Marsden NHS Foundation Trust (GB)
ORCID
Venue
IEEE Transactions on Medical Imaging
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2017-08
Volume
36
Issue
8
Pages
1597-1606
ISSN
  • 0278-0062 (ISSN-L)
  • 1558-254X
DOI
10.1109/tmi.2017.2665165
OpenAlex
W2594608309
Semantic Scholar
d11e2caf177686536879226b14e089c9382dbb30
Cited by
152
References
41
Citations by year
2026: 12 · 2025: 21 · 2024: 14 · 2023: 21 · 2022: 20 · 2021: 15 · 2020: 15 · 2019: 21 · 2018: 12 · 2017: 1
FWCI
5.45
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Transactions on Medical Imaging (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Spiral (Imperial College London) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Institute for Health Research under EME Project (13/122/01)
  • EPSRC First Grant Scheme (EP/N023668/1)
  • Indonesia Endowment for Education (LPDP)–Indonesian Presidential Ph.D. Scholarship Programme
  • Imperial College President’s Ph.D. Scholarship Programme
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Engineering › Biomedical Engineering › Medical Imaging and Analysis
Keywords
automated image segmentation, multi-organ segmentation, overfitting, cross-validation
MeSH terms
Algorithms, Reproducibility of Results

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-22.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-22.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review