Automatic quality control of segmentation results using early epochs as data augmentation: application to choroid plexuses
Open accessAbstract
The establishment of automated image segmentation methods in medical imaging allows the analysis of very large datasets. However, visual quality control (QC) of segmentation results is impractical in large datasets, hence the need for automatic QC. In this paper, we introduce a novel automatic approach for QC of segmentation results. We developed a QC deep learning model (referred to as QC model) that, for a given patient, predicts the accuracy of the corresponding automatic segmentation (in our work the Dice score) provided by a deep learning segmentation model (referred to as segmentation model) in the absence of a ground truth annotation. To train the QC model, we introduce data augmentation by using the early epochs of the segmentation model. These early epochs allow us to feed the training of the QC model with examples of poor segmentation. We applied our approach to the QC of automatic segmentation of the choroid plexuses of the brain from MRI in controls and patients with multiple sclerosis. However, the method is generic and could be used with any segmentation model. The experiments showed that the proposed approach is very effective for predicting the segmentation accuracy with a correlation coefficient of 0.92, an R2 of 0.763, a mean absolute error (MAE) of 0.078, and a mean squared error (MSE) of 0.009. Overall, this work shall provide a valuable tool for the automatic QC of segmentation results.
AI snapshot
Written from the paper’s full text.
TL;DR
The paper proposes an automatic quality-control model that predicts the Dice accuracy of a deep-learning segmentation without ground truth, using early-epoch segmentations from the segmentation model as data augmentation. Applied to choroid plexus MRI segmentation in multiple sclerosis, radiologically isolated syndrome, and controls, it achieved test correlation 0.92, R² 0.763, MAE 0.078, and MSE 0.009.
Key findings
- On the validation set, the best model (conv7fc3 + L1 loss + data augmentation) reached MAE 0.052±0.001, MSE 0.005±0.000, ρ 0.941, and R² 0.885; MSE loss with augmentation was similar (MAE 0.059±0.001, MSE 0.005±0.000, ρ 0.94, R² 0.873).
- On the independent test set, the proposed approach (conv7fc3 + L1 loss + data augmentation) achieved MAE 0.078±0.001, MSE 0.009±0.000, ρ 0.920, and R² 0.763.
- Models trained without data augmentation performed poorly, even worse than the trivial model: for example, L1 loss without augmentation had MAE 0.190±0.005, MSE 0.063±0.003, ρ 0.056, and R² -0.594, while the trivial model had MAE 0.165±0.004, MSE 0.040±0.002, and R² 0.
- MRAE loss with augmentation performed worse than L1 or MSE loss with augmentation on validation (MAE 0.106±0.002, MSE 0.015±0.000, ρ 0.9, R² 0.625).
Methods
- The QC model is a convolutional neural network with 7 convolutional and 3 fully connected layers (conv7fc3), dropout 0.5, trained to predict the Dice score between automatic and manual segmentation from the image and automatic segmentation.
- Data came from a previously published choroid plexus segmentation model: 168 scans from MS (N=97), RIS (N=27), and healthy controls (N=44), split at subject level into training (72 subjects, 3960 image-segmentation pairs), validation (19 subjects, 1045 pairs), and testing (77 subjects, 4235 pairs).
- Training used 5-fold cross-validation, PyTorch, weighted Adam with initial learning rate 10⁻³, batch size 17, and two phases: 10 epochs with data augmentation and 100 epochs without. Data augmentation used early-epoch segmentations from the segmentation model; performance was assessed with MAE, MSE, R², and Pearson correlation, comparing with/without augmentation, different losses, and a trivial mean-prediction model.
Limitations
- The paper states that test performance was slightly lower than validation performance and that this requires further investigation, as it may indicate slight overfitting of the validation set by trying different approaches and losses.
- Evident from the text: Table 1 reports validation and testing results on one fold despite the stated use of 5-fold cross-validation, so fold-to-fold variability is not shown in the reported table.
- Evident from the text: evaluation is limited to Dice score and choroid plexus segmentation, although the authors state the method is generic and could be applied to other segmentation models.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
2 of 2 cited sources checked · 1 supported · 0 flagged · accuracy not yet measured
- Findingown data
On the test set, the performances of the proposed approach were lower than on the validation set.
“On the test set, the performances of the proposed approach remained very good (correlation of 0.92, MAE of 0.078) but were lower than on the validation set.”
- Also stated in: § 4. DISCUSSION · plain
“However, the performance was slightly lower than those obtained on the validation set.”
Quote in the paperNot a substantial claim- Rests on an assumption: Manual segmentations used as ground truth are accurate.
- Also stated in: § 4. DISCUSSION · plain
- Findingown data
All models trained without data augmentation performed poorly, even worse than the trivial model.
“All models trained without DA performed poorly, even worse than the trivial model.”
Quote in the paperSubstantial claim- Rests on an assumption: Manual segmentations used as ground truth are accurate.
- Findingown data
On the test set, the proposed approach achieved an R2 of 0.763.
“an R 2 of 0.763”
- Also stated in: § 4. DISCUSSION · plain
“coefficient of determination (R 2 = 0.763)”
Quote in the paperSubstantial claim- Rests on an assumption: Manual segmentations used as ground truth are accurate.
- Also stated in: § 4. DISCUSSION · plain
- Findingown data
On the test set, the proposed approach achieved a correlation coefficient of 0.92.
“with a correlation coefficient of 0.92”
- Also stated in: § 3. EXPERIMENTS AND RESULTS · plain
“On the test set, the performances of the proposed approach remained very good (correlation of 0.92, MAE of 0.078) but were lower than on the validation set.”
Quote in the paperSubstantial claim- Rests on an assumption: Manual segmentations used as ground truth are accurate.
- Also stated in: § 3. EXPERIMENTS AND RESULTS · plain
- Findingown data
On the test set, the proposed approach achieved a mean absolute error (MAE) of 0.078.
“a mean absolute error (MAE) of 0.078”
- Also stated in: § 3. EXPERIMENTS AND RESULTS · plain
“MAE of 0.078”
Quote in the paperSubstantial claim- Rests on an assumption: Manual segmentations used as ground truth are accurate.
- Also stated in: § 3. EXPERIMENTS AND RESULTS · plain
- Interpretationown data
There is a strong relationship between predicted and true quality scores.
“Our best-performing model, trained with DA, with the L1loss, reached a high correlation coefficient (ρ = 0.92) and coefficient of determination (R 2 = 0.763) indicating a strong relationship between predicted and true quality scores.”
Quote in the paperSubstantial claim- Rests on an assumption: Manual segmentations used as ground truth are accurate.
- Interpretationown data
The proposed approach is very effective for predicting segmentation accuracy.
“The experiments showed that the proposed approach is very effective for predicting the segmentation accuracy with a correlation coefficient of 0.92, an R 2 of 0.763, a mean absolute error (MAE) of 0.078, and a mean squared error (MSE) of 0.009.”
Quote in the paperSubstantial claim- Rests on an assumption: Manual segmentations used as ground truth are accurate.
- Interpretationreasoning
Using early epochs of the segmentation model as data augmentation provides examples of poor segmentation for training the QC model.
“These early epochs allow us to feed the training of the QC model with examples of poor segmentation.”
- Also stated in: § 4. DISCUSSION · plain
“we used results from early epochs of the segmentation model for data augmentation, thereby providing a variety of possible segmentation quality and therefore ground truth annotations.”
Quote in the paperSubstantial claim - Also stated in: § 4. DISCUSSION · plain
- Interpretationown data
The proposed method can potentially be useful for automatic QC of large datasets.
“Nevertheless, the performance remained high and indicates that the proposed method can potentially be useful for automatic QC of large datasets.”
Quote in the paperNot a substantial claim- Rests on an assumption: Manual segmentations used as ground truth are accurate.
- Classified as probably not a substantial claim. 9
- May combine two claims. 4
- The quote only partly states this claim. 1
Classified as probably not a substantial claim.
PremisecitationAuto-encoder-like QC methods may not be best-suited to objects of interest that present highly variable spatial conformation between subjects and even between visits for the same subject.
“These types of methods may however not be best-suited to objects of interest that present highly variable spatial conformation between subjects and even between visits for the same subject.”
Quote in the paperSubstantial claim1 of 1 sources checked- cites [21] Galati, 2021, Efficient model monitoring for quality control in cardiac image segmentation source
The Introduction (p. 2) says the autoencoder models variability in cardiac segmentation masks, while its warning about major morphological deformations concerns an atlas-registration method, not autoencoder-based QC. The Discussion (p. 8) identifies poor sensitivity to small segmentation errors, but does not say these methods are ill-suited to variation between subjects or visits. Searches for spatial conformation and inter- or intra-subject variation found no passage making that claim. (LLM, conf=0.75)
Experimental — accuracy not yet measured
Classified as probably not a substantial claim. May combine two claims.
PremisecitationThe segmentation model used in this study was previously published and segments the choroid plexuses of the brain from T1-weighted MRI.
“For this study, we utilized a previously published segmentation model that segments the choroid plexuses of the brain T1-weighted (T1w) from magnetic resonance imaging (MRI) scans 22”
Quote in the paperSubstantial claim1 of 1 sources checked- cites [22] source
The quote only partly states this claim.
Findingown dataOn the test set, the proposed approach achieved a mean squared error (MSE) of 0.009.
“a mean squared error (MSE) of 0.009”
Quote in the paperNot a substantial claim- Rests on an assumption: Manual segmentations used as ground truth are accurate.
May combine two claims.
Findingown dataOn the validation set, the best performances are obtained with models trained with data augmentation, notably with L1loss and MSEloss, while MRAEloss showed poorer performances.
“On the validation set, the best performances are obtained with the models trained with DA, notably with L1loss and MSEloss, while MRAEloss showed poorer performances.”
Quote in the paperSubstantial claim- Rests on an assumption: Manual segmentations used as ground truth are accurate.
Probably not a substantial claim: may not pass S2 (checkable).
PremisereasoningIt is necessary to perform quality control of segmentation results to identify poor segmentations that can bias subsequent analyses.
“Even though the performances of the automatic segmentation models are remarkable, it remains necessary to perform quality control (QC) of segmentation results in order to identify poor segmentations which can bias subsequent analyses.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim. May combine two claims.
Premisenone givenManual annotation of segmentation quality requires manual labeling and suffers from possible subjectivity of the annotator.
“This method requires manual labeling and suffers from a possible subjectivity of the annotator.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductreasoningDropout with a 0.5 probability was implemented to avoid overfitting.
“To avoid overfitting, we implemented dropout with a 0.5 probability.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based). May combine two claims.
Premisenone givenVisual quality control of segmentation results is time-consuming and may be impractical for very large datasets.
“However, visual QC is time-consuming and may thus be impractical for very large datasets.”
- Also stated in: § Abstract · plain
“visual quality control (QC) of segmentation results is impractical in large datasets”
Quote in the paperNot a substantial claim- Also stated in: § Abstract · plain
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenAll training procedures were carried out using 5-fold cross-validation.
“All training procedures were carried out using 5-fold cross-validation.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone givenThe proposed QC method is generic and could be used with any segmentation model.
“However, the method is generic and could be used with any segmentation model.”
- Also stated in: § 1. INTRODUCTION · hedged
“but this concept could be easily applied to other tasks.”
Quote in the paperNot a substantial claim- Also stated in: § 1. INTRODUCTION · hedged
Classified as probably not a substantial claim.
Premisenone givenTraining a QC model to predict a segmentation accuracy metric requires a large and diverse set of segmentation outputs together with manual segmentation ground truth.
“However, the training requires a large and diverse set of segmentation outputs together with manual segmentation ground truth.”
Quote in the paperSubstantial claim
The paper relies on this without support.
Manual segmentations used as ground truth are accurate.
“S m the manual segmentation (ground-truth)”
“manual segmentation ground truth”
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473 · see citing works
Image processing and Quality Control for the first 10,000 brain imaging datasets from UK Biobank
Alfaro‐Almagro, F., Jenkinson, M., Bangerter, N. K., Andersson, J., Griffanti, L., Douaud, G., Sotiropoulos, S. N., Jbabdi, S., Hernández-Fernández, M., Vallée, E., et al. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2017.10.034 · Cited by 1,563 · see citing works
Medical Image Computing and Computer Assisted Intervention – MICCAI 2018
Frangi, A. F., Schnabel, J. A., Davatzikos, C. A., Alberola-López, C., & Fichtinger, G. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00937-3 · Cited by 1,049 · see citing works
MRIQC: Advancing the automatic prediction of image quality in MRI from unseen sites
Esteban, O., Birman, D., Schaer, M., Koyejo, O. O., Poldrack, R. A., & Gorgolewski, K. J. (2017) · PLOS ONE · doi:10.1371/journal.pone.0184661 · Cited by 920 · see citing works
IEEE Transactions on Medical Imaging
(2026) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.42 · Cited by 753 · see citing works
Qoala-T: A supervised-learning tool for quality control of FreeSurfer segmented MRI data
Klapwijk, E. T., van de Kamp, F., van der Meulen, M., Peters, S., & Wierenga, L. M. (2019) · NeuroImage · doi:10.1016/j.neuroimage.2019.01.014 · Cited by 192 · see citing works
Reverse Classification Accuracy: Predicting Segmentation Performance in the Absence of Ground Truth
Valindria, V. V., Lavdas, I., Bai, W., Kamnitsas, K., Aboagye, E. O., Rockall, A. G., Rueckert, D., & Glocker, B. (2017) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2017.2665165 · Cited by 152 · see citing works
Synthesize Then Compare: Detecting Failures and Anomalies for Semantic Segmentation
Xia, Y., Zhang, Y., Liu, F., Shen, W., & Yuille, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58452-8_9 · Cited by 143 · see citing works
Robinson, R., Valindria, V. V., Bai, W., Oktay, O., Kainz, B., Suzuki, H., Sanghvi, M. M., Aung, N., Paiva, J. M., Zemrak, F., et al. (2019) · Journal of Cardiovascular Magnetic Resonance · doi:10.1186/s12968-019-0523-x · Cited by 120 · see citing works
Leveraging Uncertainty Estimates for Predicting Segmentation QualityPreprint
DeVries, T., & Taylor, G. W. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.00502 · Cited by 117 · see citing works
MRIQC: Advancing the Automatic Prediction of Image Quality in MRI from Unseen SitesPreprint
Estéban, O., Birman, D., Schaer, M., et al. (2017) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/111294 · Cited by 114 · see citing works
Evaluating Segmentation Error without Ground Truth
Kohlberger, T., Singh, V., Alvino, C., Bahlmann, C., & Grady, L. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33415-3_65 · Cited by 102 · see citing works
Sujit, S. J., Coronado, I., Kamali, A., Narayana, P. A., & Gabr, R. E. (2019) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.26693 · Cited by 68 · see citing works
Real-Time Prediction of Segmentation Quality
Robinson, R., Oktay, O., Bai, W., Valindria, V. V., Sanghvi, M. M., Aung, N., Paiva, J. M., Zemrak, F., Fung, K., Lukaschuk, E. I., et al. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00937-3_66 · Cited by 61 · see citing works
Yazdan-Panah, A., Schmidt-Mengin, M., Ricigliano, V. A. G., Soulier, T., Stankoff, B., & Colliot, O. (2023) · NeuroImage: Clinical · doi:10.1016/j.nicl.2023.103368 · Cited by 58 · see citing works
Self-citationBottani, S., Burgos, N., Maire, A., Wild, A., Ströer, S., Dormont, D., & Colliot, O. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102219 · Cited by 48 · see citing works
Self-citationMedical image segmentation automatic quality control: A multi-dimensional approach
Fournel, J., Bartoli, A., Bendahan, D., Guye, M., Bernard, M., Rauseo, E., Khanji, M. Y., Petersen, S. E., Jacquier, A., & Ghattas, B. (2021) · Medical Image Analysis · doi:10.1016/j.media.2021.102213 · Cited by 45 · see citing works
Image Processing and Quality Control for the first 10,000 Brain Imaging Datasets from UK BiobankPreprint
Alfaro‐Almagro, F., Jenkinson, M., Bangerter, N. K., et al. (2017) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/130385 · Cited by 40 · see citing works
Kang Li, B., Yu, L., & Heng, P. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102426 · Cited by 30 · see citing works
Deep Generative Model-Based Quality Control for Cardiac MRI Segmentation
Wang, S., Tarroni, G., Qin, C., Mo, Y., Dai, C., Chen, C., Glocker, B., Guo, Y., Rueckert, D., & Bai, W. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-59719-1_9 · Cited by 27 · see citing works
An Alarm System for Segmentation Algorithm Based on Shape Model
Liu, F., Xia, Y., Yang, D., Yuille, A., & Xu, D. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.01075 · Cited by 25 · see citing works
Efficient Model Monitoring for Quality Control in Cardiac Image Segmentation
Galati, F., & Zuluaga, M. A. (2021) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-78710-3_11 · Cited by 15 · see citing works
Robust Image Segmentation Quality AssessmentPreprint
Zhou, L., Deng, W., & Wu, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.08773 · Cited by 12 · see citing works
Li, K., Ye, C., Yang, Z., et al. (2016) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2217309 · Cited by 10 · see citing works
SEG: Segmentation Evaluation in absence of Ground truth labelsPreprint
Sims, Z., Strgar, L., Thirumalaisamy, D., Heussner, R. T., Thibault, G., & Chang, Y. H. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.02.23.529809 · Cited by 5 · see citing works
Qoala-T: A supervised-learning tool for quality control of automatic segmented MRI dataPreprint
Klapwijk, E. T., van de Kamp, F., van der Meulen, M., Peters, S., & Wierenga, L. M. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/278358 · Cited by 5 · see citing works
Region Specific Automatic Quality Assurance For MRI-Derived Cortical Segmentations
Gadewar, S. P., Zhu, A. H., Thomopoulos, S. I., Li, Z., Gari, I. B., Maiti, P., Thompson, P. M., & Jahanshad, N. (2021) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi48211.2021.9433755 · Cited by 4 · see citing works
QANet -- Quality Assurance Network for Image SegmentationPreprint
Arbelle, A., Elul, E., & Raviv, T. R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.08503 · Cited by 2 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- 10.1002/jmri.v50.4 · DOI
- Biewald, 2020, Experiment tracking with weights and biases, Software available from wandb. com, 2, 233
Cited by
Sorted by citations
Norajitra, T., Schlett, C. L., von Krüchten, R., Agarwal, P., Ravindran, A., Do, T. D., Kausch, L., Karrasch, S., Kauczor, H.-U., Maier-Hein, K., et al. (2025) · Tomography · doi:10.3390/tomography11120135
“…have reported on algorithmic approaches investigating anatomic plausibility of resulting segmentations to detect segmentation errors [37] and point to an approach for segmentation quality control independent of underlying raw data, similar to the visual approach chosen in our study [38,39].”
- Authors
- Yazdan-Panah, A., Stankoff, B., & Colliot, O.
- Affiliations
- Centre National de la Recherche Scientifique (FR)
- Venue
- Medical Imaging 2024: Image Processing
- Publisher
- SPIE
- Published
- 2024-04-02
- Pages
- 44
- OpenAlex
- W4393408098
- Semantic Scholar
- 2c31332857a19c2037a82a64370ea5a8ed55c155
- Cited by
- 1
- References
- 25
- Citations by year
- 2025: 1
- FWCI
- 0.21
- Type
- conference paper
- Language
- en
- Open access
- green
- inria.hal.science
- Licence
- cc-by
- Locations
- SPIE Medical Imaging: Image Processing (publishedVersion)
- HAL (Le Centre pour la Communication Scientifique Directe) (submittedVersion, cc-by)
- Funders
- Agence Nationale de la Recherche (ANR-10-IAIHU06, ANR‐19, 10-IAIHU-06, ANR-19-P3IA-0001, ANR-10, ANR-10-IAHU-0006)
- Grand Équipement National De Calcul Intensif
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
- Life Sciences › Neuroscience › Neurology › Brain Tumor Detection and Classification
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- automated quality control, segmentation quality assessment, multiple sclerosis, brain MRI, ground truth-free evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-22.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-22.
Cited sources are checked by gpt-6-sol.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
