CoChat
Sign Up
PreprintCited by 3Domain Adaptation and Few-Shot LearningArtificial Intelligence in Healthcare and EducationAdversarial Robustness in Machine Learning

MedMNIST-C: Comprehensive benchmark and improved classifier robustness by simulating realistic image corruptions

Open access
Di Salvo, Francesco, Doerrich, Sebastian, Ledig, Christian·2024·arXiv
DOI 10.48550/arxiv.2406.17536arXiv 2406.17536
Open this paper in CoChat — free accountOpen PDF

Abstract

The integration of neural-network-based systems into clinical practice is limited by challenges related to domain generalization and robustness. The computer vision community established benchmarks such as ImageNet-C as a fundamental prerequisite to measure progress towards those challenges. Similar datasets are largely absent in the medical imaging community which lacks a comprehensive benchmark that spans across imaging modalities and applications. To address this gap, we create and open-source MedMNIST-C, a benchmark dataset based on the MedMNIST+ collection covering 12 datasets and 9 imaging modalities. We simulate task and modality-specific image corruptions of varying severity to comprehensively evaluate the robustness of established algorithms against real-world artifacts and distribution shifts. We further provide quantitative evidence that our simple-to-use artificial corruptions allow for highly performant, lightweight data augmentation to enhance model robustness. Unlike traditional, generic augmentation strategies, our approach leverages domain knowledge, exhibiting significantly higher robustness when compared to widely adopted methods. By introducing MedMNIST-C and open-sourcing the corresponding library allowing for targeted data augmentations, we contribute to the development of increasingly robust methods tailored to the challenges of medical imaging. The code is available at https://github.com/francescodisalvo05/medmnistc-api .

AI snapshot

Written from the paper’s full text.

TL;DR

MedMNIST-C is a new corruption benchmark built on MedMNIST+ covering 12 datasets and 9 imaging modalities, with task- and modality-specific corruptions at five severity levels. The authors also show that using these same corruptions as targeted data augmentation improves classifier robustness more than generic augmentations.

Key findings

  • MedMNIST-C spans 12 datasets and 9 imaging modalities, with corruptions grouped into digital, noise, blur, color, and task-specific categories, each evaluated at 5 severity levels.
  • In robustness experiments averaged across all 12 datasets, ViT-B/16 was the most robust model (clean bACC 78.9, corrupted bACC 72.0, rBE 59.9, BE 76.3), while ResNet50 was least robust (clean bACC 75.4, corrupted bACC 56.2, rBE 166.1, BE 131.5), followed by DenseNet121 (rBE 148.4, BE 114.8).
  • Targeted corruption augmentation with ResNet18 achieved the top average AUC gain across all 12 datasets: 9.2, compared with RandAugment 3.9, MixUp 2.7, and CutMix 1.0; under k-fold evaluation by corruption type, the method still gained 4.0 on average.
  • The augmentation gains were especially large on smaller datasets with fewer than 10,000 samples, and the method was overall best performing (p < 0.05, Wilcoxon signed-rank test).

Methods

  • The authors created MedMNIST-C from the MedMNIST+ collection at 224×224 resolution, simulating dataset-specific corruptions across 12 datasets and 9 imaging modalities, with 5 severity levels per corruption.
  • Robustness was measured using balanced error (BE) and relative balanced error (rBE), normalized by AlexNet errors, averaged across severities and corruptions; multi-label tasks were averaged across labels. Models evaluated were AlexNet, ResNet50, DenseNet121, ViT-B/16, and VGG16, initialized with ImageNet weights, trained for 100 epochs with early stopping, AdamW at 1e-4, cosine annealing, and three seed runs.
  • For augmentation, ResNet18 was trained with the corruption API, which randomly selects a dataset-specific corruption or identity and samples severity/hyperparameters; comparisons used MixUp (α = 0.2), CutMix (α = 1.0), and RandAugment (k = 1), with AUC averaged over three seed runs.

Limitations

  • The paper states that future work could extend the method to manage multiple corruptions simultaneously, mirroring RandAugment, or implement them across multiple chains as in AugMix; AugMix was excluded from comparison for this reason.
  • The paper notes that CutMix and MixUp do not natively support multilabel problems, so they do not contribute to the overall mean, and that the non-k-fold augmentation results “might seem as an overestimate of performance gains.”
  • Evident from the text: the benchmark is limited to MedMNIST+ at 224×224 and uses simulated corruptions rather than real clinical distribution shifts, and the robustness evaluation covers a limited set of standard architectures.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2406.17536

Cites

37 of 37 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 9 cited sources checked · 0 supported · 0 flagged

  • Findingreasoning

    The benchmark bridges MedMNIST+ with the ImageNet-C robustness evaluation framework.

    “We introduce a novel robustness benchmark, bridging the widely utilized MedMNIST+ dataset with the well-established robustness evaluation framework of ImageNet-C.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Clinical integration of neural-network-based systems is limited by challenges related to domain generalization and robustness.
  • Findingown data

    The targeted augmentation approach achieves the top average AUC gain across all 12 datasets.

    “Table 3 shows that, even under a k-fold cross-validation method, our augmentation approach achieves the top average AUC gain across all 12 datasets.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The simulated corruptions faithfully represent real-world artifacts and distribution shifts in medical imaging.
    • Rests on an assumption: The AUC gains on corrupted test sets indicate improved robustness to real-world distribution shifts.
  • Findingown data

    The targeted augmentation excels particularly in smaller datasets (fewer than 10,000 samples).

    “Notably, our augmentation excels particularly in smaller datasets, those with fewer than 10,000 samples, which also tend to have lower initial AUC values.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The simulated corruptions faithfully represent real-world artifacts and distribution shifts in medical imaging.
    • Rests on an assumption: The AUC gains on corrupted test sets indicate improved robustness to real-world distribution shifts.
  • Findingcitation

    VGG16 follows ViT-B/16 in normalized performance (BE).

    “As expected [23], while ViT-B/16 is the most robust one, VGG follows in terms of normalized performance (BE).”
    Quote in the paperNot a substantial claim
  • Findingown data

    The corrupted datasets and APIs are publicly shared.

    “We publicly share our corrupted datasets and APIs to facilitate the online, and dataset-specific augmentation of medical image analysis problems.”
    • Also stated in: § 4 Conclusions · plain
      “By providing reproducible corrupted test sets and APIs, we aim to encourage the community to both assess and enhance the robustness of medical image analysis models.”
    Quote in the paperSubstantial claim
  • May combine two claims. 6
  • Classified as probably not a substantial claim. 5

  • May combine two claims.

    Findingown data

    ViT exhibits higher robustness to all corruptions, with the gap even more pronounced on noise and digital artifacts.

    “Also, while ViT exhibits higher robustness to all corruptions, the gap is even more pronounced on noise and digital artifacts.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Findingnone given

    The designed corruptions are dataset-specific and reflect possible artifacts encountered during image acquisition and processing, simulating real-world artifacts or possible distribution shifts.

    “The designed corruptions are dataset-specific and reflect the possible artifacts encountered during image acquisition and processing, simulating real-world artifacts or possible distribution shifts.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Clinical integration of neural-network-based systems is limited by challenges related to domain generalization and robustness.
    • Rests on an assumption: The simulated corruptions faithfully represent real-world artifacts and distribution shifts in medical imaging.
    • Rests on an assumption: The MedMNIST+ datasets are representative of their respective imaging modalities and clinical tasks.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The code is available at github.com/francescodisalvo05/medmnistc-api.

    “The code is available at github.com/francescodisalvo05/medmnistc-api.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    VGG16 exhibits a severe performance drop (rBE) despite achieving the best clean performance.

    “However, despite achieving the best clean performance, probably due to the highest number of parameters, it exhibits a severe performance drop (rBE).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    ResNet50 is the least robust model, followed by DenseNet121.

    “Notably, ResNet50 is the least robust one, followed by DenseNet121.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The targeted augmentation approach leverages domain knowledge and exhibits significantly higher robustness compared to widely adopted methods.

    “Unlike traditional, generic augmentation strategies, our approach leverages domain knowledge, exhibiting significantly higher robustness when compared to widely adopted methods.”
    • Also stated in: § 4 Conclusions · boosted
      “Our approach, straightforward yet impactful, consistently enhances robustness and outperforms standard methods, even under k-fold validation, ensuring an unbiased evaluation against our test corruptions.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Clinical integration of neural-network-based systems is limited by challenges related to domain generalization and robustness.
    • Rests on an assumption: The simulated corruptions faithfully represent real-world artifacts and distribution shifts in medical imaging.
    • Rests on an assumption: The AUC gains on corrupted test sets indicate improved robustness to real-world distribution shifts.

  • May combine two claims.

    Findingnone given

    The proposed augmentations can be readily applied via open-source APIs during training and meaningfully enhance the robustness of traditional domain-agnostic approaches.

    “The proposed augmentations can be readily applied via our open-source APIs during training and meaningfully enhance the robustness of traditional domain-agnostic approaches.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The simulated corruptions faithfully represent real-world artifacts and distribution shifts in medical imaging.
    • Rests on an assumption: The AUC gains on corrupted test sets indicate improved robustness to real-world distribution shifts.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    Training set size might play an important role in robustness.

    “Thus, the training set size might play an important role.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given

    The targeted augmentations are simpler and more interpretable than traditional ones.

    “We demonstrate the effectiveness of our targeted augmentations, increasing the robustness through a simpler and more interpretable method, compared to traditional ones.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The MedMNIST+ datasets span four distinct tasks: binary classification, multiclass classification, multi-label (binary) classification, and ordinal regression.

    “The MedMNIST+ datasets span four distinct tasks: binary classification, multiclass classification, multi-label (binary) classification, and ordinal regression.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [29]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • Rests on an assumption: The MedMNIST+ datasets are representative of their respective imaging modalities and clinical tasks.
10 of 10

  • The paper relies on this without support.

    Clinical integration of neural-network-based systems is limited by challenges related to domain generalization and robustness.

    “The integration of neural-network-based systems into clinical practice is limited by challenges related to domain generalization and robustness.”

  • The paper relies on this without support.

    The simulated corruptions faithfully represent real-world artifacts and distribution shifts in medical imaging.

    “We simulate task and modality-specific image corruptions of varying severity to comprehensively evaluate the robustness of established algorithms against real-world artifacts and distribution shifts.”
    “The designed corruptions are dataset-specific and reflect the possible artifacts encountered during image acquisition and processing, simulating real-world artifacts or possible distribution shifts.”

  • The paper relies on this without support.

    The MedMNIST+ datasets are representative of their respective imaging modalities and clinical tasks.

    “leveraging the publicly available MedMNIST+ [29] dataset collection (resolution: 224×224).”
    “The MedMNIST+ datasets span four distinct tasks: binary classification, multiclass classification, multi-label (binary) classification, and ordinal regression.”

  • The paper relies on this without support.

    The AUC gains on corrupted test sets indicate improved robustness to real-world distribution shifts.

    “We further provide quantitative evidence that our simple-to-use artificial corruptions allow for highly performant, lightweight data augmentation to enhance model robustness.”
    “Our approach, straightforward yet impactful, consistently enhances robustness and outperforms standard methods, even under k-fold validation, ensuring an unbiased evaluation against our test corruptions.”
4 of 4

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • RetinaMNIST blur types differ between text and table

    Minor

    The text identifies Gaussian and defocus blur for RetinaMNIST, but Table 1 lists defocus and motion under its blur category. Gaussian appears there under noise, not blur.

  • ViT's task-specific corruptions are not its least impactful category

    Minor

    The table reports a lower normalized error for ViT-B on noise (50) than on task-specific corruptions (71). Since lower BE indicates less impact, task-specific corruptions are not the least impactful for ViT.

2 of 2

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Authors
Di Salvo, F., Doerrich, S., & Ledig, C.
Affiliations
University of Bamberg (DE)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2406.17536
arXiv
2406.17536
OpenAlex
W4400104233
Cited by
3
References
34
Citations by year
2026: 1 · 2025: 1 · 2024: 1
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
robustness benchmark, medical image classification, data augmentation, domain generalization, distribution shift, image corruptions, medical imaging modalities, classifier robustness

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-23.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-23.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review