MedMNIST v2 - A large-scale lightweight benchmark for 2D and 3D biomedical image classification

May combine two claims.

MedMNIST v2 is extended from MedMNIST v1, with 10 2D datasets for medical image classification.

“MedMNIST v2 is extended from our preliminary version, MedMNIST v19, with 10 2D datasets for medical image classification.”

• cites [9] Yang, J., Shi, R. & Ni, B. Medmnist classification decathlon: A lightweight automl benchmark for medical image analysis. In International Symposium on Biomedical Imaging, 191–195 (2021). sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Each MedMNIST v2 subset is saved in NumPy npz format, named as <data>mnist.npz for MedMNIST2D and <data>mnist3d.npz for MedMNIST3D, and is comprised of 6 keys.

“Each subset is saved in NumPy39 npz format, named as <data> mnist.npz for MedMNIST2D and <data> mnist3d.npz for MedMNIST3D, and is comprised of 6 keys (“train_images”, “train_labels”, “val_images”, “val_labels”, “test_images”, “test_labels”).”

• cites [39] Harris, C. R. et al. Array programming with numpy. Nature 585, 357–362 (2020). sourceSupported

May combine two claims.

VesselMNIST3D is a 3D dataset based on the IntrA dataset, containing 103 3D models of entire brain vessels, with 1,694 healthy vessel segments and 215 aneurysm segments.

“The VesselMNIST3D is based on an open-access 3D intracranial aneurysm dataset, IntrA34, containing 103 3D models (meshes) of entire brain vessels collected by reconstructing MRA images. 1,694 healthy vessel segments and 215 aneurysm segments are generated automatically from the complete models.”

• cites [34] Yang, X., Xia, D., Kin, T. & Igarashi, T. Intra: 3d intracranial aneurysm dataset for deep learning. In Conference on Computer Vision and Pattern Recognition (2020). sourceSupported

May combine two claims.

SynapseMNIST3D is a new 3D volume dataset to classify whether a synapse is excitatory or inhibitory, using a 3D image volume of an adult rat acquired by a multi-beam scanning electron microscope.

“The SynapseMNIST3D is a new 3D volume dataset to classify whether a synapse is excitatory or inhibitory. It uses a 3D image volume of an adult rat acquired by a multi-beam scanning electron microscope.”

• cites [37] Wei, D. et al. Mitoem dataset: Large-scale 3d mitochondria instance segmentation from em images. In Conference on Medical Image Computing and Computer Assisted Intervention, 66–76 (Springer, 2020). sourcePartly supported

  The Dataset Acquisition section (page unavailable) says the adult-rat tissue used for MitoEM-R was imaged with a multi-beam scanning electron microscope. But the Introduction identifies MitoEM as a 3D mitochondria instance-segmentation dataset; our copy does not identify SynapseMNIST3D or describe excitatory-versus-inhibitory synapse classification. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

May combine two claims.

TissueMNIST is a 2D dataset based on BBBC051, containing 236,386 human kidney cortex cells organized into 8 categories.

“We use the BBBC05128, available from the Broad Bioimage Benchmark Collection29. The dataset contains 236,386 human kidney cortex cells, segmented from 3 reference tissue specimens and organized into 8 categories.”

• cites [28] Woloshuk, A. et al. In situ classification of cell types in human kidney tissue using 3d nuclear staining. Cytometry Part A (2020). sourcePartly supported

  The Data organization section describes approximately 230,000 nuclei from three reference tissue specimens, and the Results describe eight classification categories and derived 2D datasets. The copy does not give the exact count of 236,386 or identify the dataset as TissueMNIST or BBBC051; it says the dataset would be made available through an online repository such as the Broad Bioimage Benchmark Collection. (LLM, conf=0.86)

  Experimental — accuracy not yet measured
• cites [29] Ljosa, V., Sokolnicki, K. L. & Carpenter, A. E. Annotated high-throughput microscopy image sets for validation. Nature methods 9, 637–637 (2012). sourceSupported

May combine two claims.

DermaMNIST is a 2D dataset based on HAM10000, comprising 10,015 dermatoscopic images categorized as 7 different diseases for multi-class classification.

“The DermaMNIST is based on the HAM1000019–21, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. The dataset consists of 10,015 dermatoscopic images categorized as 7 different diseases, formulized as a multi-class classification task.”

• cites [19] Tschandl, P., Rosendahl, C. & Kittler, H. The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data 5, 180161 (2018). sourcePartly supported

  The Abstract reports 10,015 dermatoscopic images, and Methods (“Unifying pathologic diagnoses”) says HAM10000 has seven diagnostic classes intended as a benchmark for classification. The paper does not identify DermaMNIST or establish that DermaMNIST is based on HAM10000, so it supports the underlying dataset description but not that relationship. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites [20] Tschandl, P. The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Harvard Dataverse https://doi.org/10.7910/DVN/DBW86T (2018). sourceNot checked — no full text of the source
• cites [21] Codella, N. et al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). Preprint at https://arxiv.org/abs/1902.03368v2 (2019). sourcePartly supported

  Section 2.3 (p. 4) reports 10,015 dermoscopic training images with seven classification labels and cites the HAM10000 paper, identified in the References (p. 8). It does not identify DermaMNIST, establish that DermaMNIST is based on HAM10000, or describe it as a 2D dataset; the 10,015 figure is stated for the challenge’s training set. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

May combine two claims.

The data files of MedMNIST v2 can be accessed at Zenodo, containing 12 pre-processed 2D datasets and 6 pre-processed 3D datasets.

“The data files of MedMNIST v2 dataset can be accessed at Zenodo38. It contains 12 pre-processed 2D datasets (MedMNIST2D) and 6 pre-processed 3D datasets (MedMNIST3D).”

• cites [38] Yang, J. et al. Medmnist v2: A large-scale lightweight benchmark for 2d and 3d biomedical image classification. Zenodo https://doi.org/10.5281/zenodo.5208230 (2021). sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

BreastMNIST is a 2D dataset based on a dataset of 780 breast ultrasound images, simplified into binary classification by combining normal and benign as positive and classifying them against malignant as negative.

“The BreastMNIST is based on a dataset25 of 780 breast ultrasound images. It is categorized into 3 classes: normal, benign, and malignant. As we use low-resolution images, we simplify the task into binary classification by combining normal and benign as positive and classifying them against malignant as negative.”

• cites [25] Al-Dhabyani, W., Gomaa, M., Khaled, H. & Fahmy, A. Dataset of breast ultrasound images. Data in Brief 28, 104863, https://doi.org/10.1016/j.dib.2019.104863 (2020). sourceSupported

May combine two claims.

OrganMNIST3D is a 3D dataset based on the same source as Organ{A,C,S}MNIST, using 3D bounding boxes processed into 28 × 28 × 28 for multi-class classification of 11 body organs.

“The source of the OrganMNIST3D is the same as that of the Organ{A,C,S}MNIST. Instead of 2D images, we directly use the 3D bounding boxes and process the images into 28 × 28 × 28 to perform multi-class classification of 11 body organs.”

• cites [30] Bilic, P. et al. The liver tumor segmentation benchmark (lits). Medical Image Analysis 84,102680 (2023). sourceCould not verify

  The Methods > Dataset passage describes LiTS as a collection of abdominal CT scans, but does not identify it as the source of OrganMNIST3D or Organ{A,C,S}MNIST. Searches for those dataset names and for 3D bounding boxes, 28 × 28 × 28 images, and classification of 11 organs found no supporting passage in our copy. (LLM, conf=0.75)

  Experimental — accuracy not yet measured
• cites [31] Xu, X. et al. Efficient multiple organ localization in ct image using 3d region proposal network. IEEE Transactions on Medical Imaging 38, 1885–1898 (2019). sourceNot checked — no full text of the source

May combine two claims.

NoduleMNIST3D is a 3D dataset based on LIDC-IDRI, designed for binary classification of lung nodules by categorizing malignancy level 1/2 as negative and 4/5 as positive, ignoring level 3.

“The NoduleMNIST3D is based on the LIDC-IDRI32, a large public lung nodule dataset, containing images from thoracic CT scans. The dataset is designed for both lung nodule segmentation and 5-level malignancy classification task. To perform binary classification, we categorize cases with malignancy level 1/2 into negative class and 4/5 into positive class, ignoring the cases with malignancy level 3.”

• cites [32] Armato, S. G. III et al. The lung image database consortium (lidc) and image database resource initiative (idri): A completed reference database of lung nodules on ct scans. Medical Physics 38, 915–931, https://doi.org/10.1118/1.3528204 (2011). sourceSupported

May combine two claims.

The MedMNIST v2 dataset consists of 12 2D and 6 3D standardized datasets from carefully selected sources covering primary data modalities, diverse classification tasks, and dataset scales.

“The MedMNIST v2 dataset consists of 12 2D and 6 3D standardized datasets from carefully selected sources covering primary data modalities (e.g., X-ray, OCT, ultrasound, CT, electron microscope), diverse classification tasks (binary/multi-class, ordinal regression, and multi-label) and dataset scales (from 100 to 100,000).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

MedMNIST v2 uses the classification labels from the source datasets directly in most cases, but the labels could be simplified (merged or deleted classes) if the classification tasks on the small images are too difficult.

“The MedMNIST uses the classification labels from the source datasets directly in most cases, but the labels could be simplified (merged or deleted classes) if the classification tasks on the small images are too difficult.”

• AssumptionRests on an assumption: The classification labels from the source datasets are accurate and suitable for the benchmark tasks.

May combine two claims.

MedMNIST v2 is a large-scale MNIST-like dataset collection of standardized biomedical images, including 12 datasets for 2D and 6 datasets for 3D.

“We introduce MedMNIST v2, a large-scale MNIST-like dataset collection of standardized biomedical images, including 12 datasets for 2D and 6 datasets for 3D.”

• Also stated in: § Background & Summary · plain

  “MedMNIST v2 is a large-scale benchmark for 2D and 3D biomedical image classification, covering 12 2D datasets with 708,069 images and 6 3D datasets with 9,998 images.”

May combine two claims.

All source datasets for MedMNIST v2 are either associated with Creative Commons (CC) Licenses or developed by the authors, which allows them to develop derivative datasets based on them.

“All source datasets are either associated with the Creative Commons (CC) Licenses or developed by us, which allows us to develop derivative datasets based on them.”

May combine two claims.

MedMNIST v2 is designed to perform classification on lightweight 2D and 3D images with various dataset scales (from 100 to 100,000) and diverse tasks (binary/multi-class, ordinal regression, and multi-label).

“Covering primary data modalities in biomedical images, MedMNIST v2 is designed to perform classification on lightweight 2D and 3D images with various dataset scales (from 100 to 100,000) and diverse tasks (binary/multi-class, ordinal regression, and multi-label).”

May combine two claims.

MedMNIST v2 is designed to be diverse, covering diverse data modalities, dataset scales (from 100 to 100,000), and tasks (binary/multi-class, multi-label, and ordinal regression).

“It is designed to be:Diverse: It covers diverse data modalities, dataset scales (from 100 to 100,000), and tasks (binary/multi-class, multi-label, and ordinal regression).”

May combine two claims.

Each sub-dataset in MedMNIST v2 is pre-processed into the same format, which requires no background knowledge for users.

“Standardized: Each sub-dataset is pre-processed into the same format (see details in Methods), which requires no background knowledge for users.”

May combine two claims.

Some MedMNIST v2 datasets are under CC-BY-NC license; the authors have contacted the authors and obtained permission to re-distribute the datasets.

“Some datasets are under CC-BY-NC license; we have contacted the authors and obtained the permission to re-distribute the datasets.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The label arrays in MedMNIST v2 have shapes N × 1 for multi-class/binary-class/ordinal regression datasets and N × L for multi-label binary-class datasets, where L is 14 for ChestMNIST.

““{train,val,test}_labels”: an array containing ground-truth labels, with a shape of N × 1 for multi-class/binary-class/ordinal regression datasets, of N × L for multi-lable binary-class datasets. N denotes the number of samples in training/validation/test set and L denotes the number of task labels in the multi-label dataset (i.e., 14 for the ChestMNIST).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The image arrays in MedMNIST v2 have shapes N × 28 × 28 for 2D gray-scale datasets, N × 28 × 28 × 3 for 2D RGB datasets, and N × 28 × 28 × 28 for 3D datasets.

““{train,val,test}_images”: an array containing images, with a shape of N × 28 × 28 for 2D gray-scale datasets, of N × 28 × 28 × 3 for 2D RGB datasets, of N × 28 × 28 × 28 for 3D datasets.”

May combine two claims.

RetinaMNIST is a 2D dataset based on the DeepDRiD challenge, comprising 1,600 retina fundus images for ordinal regression of 5-level grading of diabetic retinopathy severity.

“The RetinaMNIST is based on the DeepDRiD24 challenge, which provides a dataset of 1,600 retina fundus images. The task is ordinal regression for 5-level grading of diabetic retinopathy severity.”

• cites [24] DeepDRiD. The 2nd diabetic retinopathy–grading and image quality estimation challenge. https://isbi.deepdr.org/data.html (2020).Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

The data type of the MedMNIST v2 dataset is uint8.

“The data type of the dataset is uint8.”

May combine two claims.

For MedMNIST2D, Google AutoML Vision is well-performing in general, but it could not always win, even compared with the baseline ResNet-18 and ResNet-50.

“For 2D datasets, Google AutoML Vision is well-performing in general, however it could not always win, even compared with the baseline ResNet-18 and ResNet-50.”

May combine two claims.

OCTMNIST is a 2D dataset based on a prior dataset of 109,309 optical coherence tomography images for retinal diseases, comprising 4 diagnosis categories for multi-class classification.

“The OCTMNIST is based on a prior dataset22,23 of 109,309 valid optical coherence tomography (OCT) images for retinal diseases. The dataset is comprised of 4 diagnosis categories, leading to a multi-class classification task.”

• cites [22] Kermany, D. S. et al. Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell 172, 1122–1131.e9, https://doi.org/10.1016/j.cell.2018.02.010 (2018). sourceNot checked — no full text of the source
• cites [23] Kermany, D. S., Zhang, K. & Goldbaum, M. Large dataset of labeled optical coherence tomography (oct) and chest x-ray images https://doi.org/10.17632/rscbjbr9sj.3 (2018). sourceNot checked — no full text of the source

May combine two claims.

For MedMNIST2D, Google AutoML Vision outperforms all the other methods in average AUC, but it is very close to the performance of baseline ResNets.

“For MedMNIST2D, Google AutoML Vision outperforms all the other methods in average AUC, however, it is very close to the performance of baseline ResNets.”

May combine two claims.

AutoKeras performs well on MedMNIST2D datasets with large scales, but relatively worse on datasets with small scale.

“AutoKeras performs well on datasets with large scales, however relatively worse on datasets with small scale.”

May combine two claims.

PneumoniaMNIST is a 2D dataset based on a prior dataset of 5,856 pediatric chest X-Ray images, for binary-class classification of pneumonia against normal.

“The PneumoniaMNIST is based on a prior dataset22,23 of 5,856 pediatric chest X-Ray images. The task is binary-class classification of pneumonia against normal.”

• cites [22] Kermany, D. S. et al. Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell 172, 1122–1131.e9, https://doi.org/10.1016/j.cell.2018.02.010 (2018). sourceNot checked — no full text of the source
• cites [23] Kermany, D. S., Zhang, K. & Goldbaum, M. Large dataset of labeled optical coherence tomography (oct) and chest x-ray images https://doi.org/10.17632/rscbjbr9sj.3 (2018). sourceNot checked — no full text of the source

May combine two claims.

ChestMNIST is a 2D dataset based on the NIH-ChestXray14 dataset, comprising 112,120 frontal-view X-Ray images of 30,805 unique patients with 14 disease labels, formulated as multi-label binary-class classification.

“The ChestMNIST is based on the NIH-ChestXray14 dataset18, a dataset comprising 112,120 frontal-view X-Ray images of 30,805 unique patients with the text-mined 14 disease labels, which could be formulized as a multi-label binary-class classification task.”

• cites [18] Wang, X. et al. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Conference on Computer Vision and Pattern Recognition, 3462–3471 (2017). sourceNot checked — no full text of the source

May combine two claims.

Under the same ResNet backbone, 2.5D models are inferior to 3D and ACS models and perform worse than auto-sklearn and AutoKeras on MedMNIST3D.

“Under the same ResNet backbone, 2.5D models are inferior to 3D and ACS models and perform worse than auto-sklearn and AutoKeras.”

Classified as probably not a substantial claim.

Organ{A,C,S}MNIST are sliced before being resized into 1 × 28 × 28.

“They are sliced before being resized into 1 × 28 × 28.”

Probably not a substantial claim: may not pass S4 (based).

There is no severe class imbalance on the MedMNIST datasets.

“Since there is no severe class imbalance on our datasets”

May combine two claims.

Organ{A,C,S}MNIST and OrganMNIST3D are generated from the same source dataset, and share the same task and the same data split.

“Organ{A,C,S}MNIST and OrganMNIST3D are generated from the same source dataset, and share the same task and the same data split.”

Classified as probably not a substantial claim.

OrganMNIST3D is resized into 28 × 28 × 28 directly.

“On the other hand, OrganMNIST3D is resized into 28 × 28 × 28 directly.”

Classified as probably not a substantial claim.

Organ{A,C,S}MNIST are sampled slices of 3D bounding boxes of 3D CT images in axial/coronal/sagittal views (planes), respectively.

“Organ{A,C,S}MNIST are sampled slices of 3D bounding boxes of 3D CT images in axial/coronal/sagittal views (planes), respectively.”

May combine two claims.

The ResNets surpass auto-sklearn and AutoKeras, and outperform Google AutoML Vision in average ACC on MedMNIST2D.

“The ResNets surpass auto-sklearn and AutoKeras, and outperform Google AutoML Vision in average ACC.”

Probably not a substantial claim: may not pass S3 (load-bearing).

All benchmark experiments on MedMNIST v2 are repeated at least 3 times for more stable results than in MedMNIST v1.

“All benchmark experiments are repeated at least 3 times for more stable results than in MedMNIST v1.”

May combine two claims.

MedMNIST v2 is as diverse as the VDD and MSD to fairly evaluate the generalizable performance of machine learning algorithms in different settings, but both 2D and 3D biomedical images are provided.

“It is as diverse as the VDD4 and MSD5 to fairly evaluate the generalizable performance of machine learning algorithms in different settings, but both 2D and 3D biomedical images are provided.”

• cites VDD4Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites MSD5Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable).

MedMNIST v2 could support numerous research/educational purposes in biomedical image analysis, computer vision, and machine learning.

“The resulting dataset, consisting of 708,069 2D images and 9,998 3D images in total, could support numerous research/educational purposes in biomedical image analysis, computer vision, and machine learning.”

• AssumptionRests on an assumption: Resizing biomedical images to 28 × 28 (2D) or 28 × 28 × 28 (3D) preserves sufficient information for meaningful classification.

Classified as probably not a substantial claim.

For datasets without an official split, MedMNIST v2 splits the dataset randomly at the patient level with a ratio of 7:1:2 into training-validation-test.

“For the dataset without an official split, we split the dataset randomly at the patient level with a ratio of 7:1:2 into training-validation-test.”

• AssumptionRests on an assumption: Patient-level random splitting avoids data leakage for datasets without official splits.

Probably not a substantial claim: may not pass S2 (checkable).

The diverse dataset design of MedMNIST v2 could lead to diverse task difficulty, which is desirable as a biomedical image classification benchmark.

“The diverse dataset design could lead to diverse task difficulty, which is desirable as a biomedical image classification benchmark.”

Classified as probably not a substantial claim.

MedMNIST v2 uses the official data split from source dataset (if provided) to avoid data leakage.

“We use the official data split from source dataset (if provided) to avoid data leakage.”

• AssumptionRests on an assumption: The official data splits from source datasets avoid data leakage.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

ACC could also serve as a good metric for the MedMNIST datasets.

“ACC could also serve as a good metric”

Classified as probably not a substantial claim.

The reported performance for each method on each dataset is the mean value of at least 3 trials.

“We calculate the mean value of at least 3 trials for each method on each dataset.”