MedMNIST v2 - A large-scale lightweight benchmark for 2D and 3D biomedical image classification

In the paper

“MedMNIST v2 is extended from our preliminary version, MedMNIST v19, with 10 2D datasets for medical image classification.”

What the sources say

• Supported

  The Introduction (p. 1) presents MedMNIST as a collection of 10 datasets for classification on 28 × 28 images, and the Dataset section (p. 2) describes those images as 2D. The citation identifies the preliminary MedMNIST collection; this earlier paper need not describe the later v2 extension. (LLM, conf=0.88)

  cites [9] Yang, J., Shi, R. & Ni, B. Medmnist classification decathlon: A lightweight automl benchmark for medical image analysis. In International Symposium on Biomedical Imaging, 191–195 (2021). source

In the paper

“Each subset is saved in NumPy39 npz format, named as <data> mnist.npz for MedMNIST2D and <data> mnist3d.npz for MedMNIST3D, and is comprised of 6 keys (“train_images”, “train_labels”, “val_images”, “val_labels”, “test_images”, “test_labels”).”

What the sources say

• Supported

  The citation is attached to “NumPy,” identifying the software rather than sourcing the MedMNIST file names or six keys. The Main section describes NumPy as an open-source library providing a multidimensional Python array object, which supports that identification. (LLM, conf=0.91)

  cites [39] Harris, C. R. et al. Array programming with numpy. Nature 585, 357–362 (2020). source

In the paper

“The VesselMNIST3D is based on an open-access 3D intracranial aneurysm dataset, IntrA34, containing 103 3D models (meshes) of entire brain vessels collected by reconstructing MRA images. 1,694 healthy vessel segments and 215 aneurysm segments are generated automatically from the complete models.”

What the sources say

• Supported

  Section 3.1 (Data) identifies IntrA as a dataset of 103 complete 3D brain-vessel models reconstructed from MRA images. It also reports 1,694 healthy vessel segments and 215 aneurysm segments generated automatically from those models. (LLM, conf=0.94)

  cites [34] Yang, X., Xia, D., Kin, T. & Igarashi, T. Intra: 3d intracranial aneurysm dataset for deep learning. In Conference on Computer Vision and Pattern Recognition (2020). source

In the paper

“The SynapseMNIST3D is a new 3D volume dataset to classify whether a synapse is excitatory or inhibitory. It uses a 3D image volume of an adult rat acquired by a multi-beam scanning electron microscope.”

What the sources say

• Partly supported

  The Dataset Acquisition section (page unavailable) says the adult-rat tissue used for MitoEM-R was imaged with a multi-beam scanning electron microscope. But the Introduction identifies MitoEM as a 3D mitochondria instance-segmentation dataset; our copy does not identify SynapseMNIST3D or describe excitatory-versus-inhibitory synapse classification. (LLM, conf=0.88)

  cites [37] Wei, D. et al. Mitoem dataset: Large-scale 3d mitochondria instance segmentation from em images. In Conference on Medical Image Computing and Computer Assisted Intervention, 66–76 (Springer, 2020). sourceExperimental — accuracy not yet measured

In the paper

“We use the BBBC05128, available from the Broad Bioimage Benchmark Collection29. The dataset contains 236,386 human kidney cortex cells, segmented from 3 reference tissue specimens and organized into 8 categories.”

What the sources say

• Partly supported

  The Data organization section describes approximately 230,000 nuclei from three reference tissue specimens, and the Results describe eight classification categories and derived 2D datasets. The copy does not give the exact count of 236,386 or identify the dataset as TissueMNIST or BBBC051; it says the dataset would be made available through an online repository such as the Broad Bioimage Benchmark Collection. (LLM, conf=0.86)

  cites [28] Woloshuk, A. et al. In situ classification of cell types in human kidney tissue using 3d nuclear staining. Cytometry Part A (2020). sourceExperimental — accuracy not yet measured
• Supported

  Citation 29 is attached to the Broad Bioimage Benchmark Collection, not to the BBBC051 dataset details. The paper describes the BBBC as a publicly available collection of microscopy image sets and gives its access URL. The dataset-specific figures in the citing sentence are not attributed to this citation. (LLM, conf=0.93)

  cites [29] Ljosa, V., Sokolnicki, K. L. & Carpenter, A. E. Annotated high-throughput microscopy image sets for validation. Nature methods 9, 637–637 (2012). source

In the paper

“The DermaMNIST is based on the HAM1000019–21, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. The dataset consists of 10,015 dermatoscopic images categorized as 7 different diseases, formulized as a multi-class classification task.”

What the sources say

• Partly supported

  The Abstract reports 10,015 dermatoscopic images, and Methods (“Unifying pathologic diagnoses”) says HAM10000 has seven diagnostic classes intended as a benchmark for classification. The paper does not identify DermaMNIST or establish that DermaMNIST is based on HAM10000, so it supports the underlying dataset description but not that relationship. (LLM, conf=0.84)

  cites [19] Tschandl, P., Rosendahl, C. & Kittler, H. The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data 5, 180161 (2018). sourceExperimental — accuracy not yet measured
• ?Not checked — no full text of the sourcecites [20] Tschandl, P. The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Harvard Dataverse https://doi.org/10.7910/DVN/DBW86T (2018). source
• Partly supported

  Section 2.3 (p. 4) reports 10,015 dermoscopic training images with seven classification labels and cites the HAM10000 paper, identified in the References (p. 8). It does not identify DermaMNIST, establish that DermaMNIST is based on HAM10000, or describe it as a 2D dataset; the 10,015 figure is stated for the challenge’s training set. (LLM, conf=0.88)

  cites [21] Codella, N. et al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). Preprint at https://arxiv.org/abs/1902.03368v2 (2019). sourceExperimental — accuracy not yet measured

In the paper

“The data files of MedMNIST v2 dataset can be accessed at Zenodo38. It contains 12 pre-processed 2D datasets (MedMNIST2D) and 6 pre-processed 3D datasets (MedMNIST3D).”

What the sources say

• Supported

  The Abstract states that MedMNIST v2 includes 12 2D and 6 3D datasets, with all images pre-processed. The Zenodo record’s Files section lists the downloadable dataset files, supporting the access claim. (LLM, conf=0.95)

  cites [38] Yang, J. et al. Medmnist v2: A large-scale lightweight benchmark for 2d and 3d biomedical image classification. Zenodo https://doi.org/10.5281/zenodo.5208230 (2021). source

In the paper

“The BreastMNIST is based on a dataset25 of 780 breast ultrasound images. It is categorized into 3 classes: normal, benign, and malignant. As we use low-resolution images, we simplify the task into binary classification by combining normal and benign as positive and classifying them against malignant as negative.”

What the sources say

• Supported

  The Data section reports 780 breast ultrasound images categorized as normal, benign, and malignant, supporting the dataset description for which this source is cited. The binary grouping is described as the citing authors’ own simplification, not as a feature attributed to this source. (LLM, conf=0.91)

  cites [25] Al-Dhabyani, W., Gomaa, M., Khaled, H. & Fahmy, A. Dataset of breast ultrasound images. Data in Brief 28, 104863, https://doi.org/10.1016/j.dib.2019.104863 (2020). source

In the paper

“The source of the OrganMNIST3D is the same as that of the Organ{A,C,S}MNIST. Instead of 2D images, we directly use the 3D bounding boxes and process the images into 28 × 28 × 28 to perform multi-class classification of 11 body organs.”

What the sources say

• ?Could not verify

  The Methods > Dataset passage describes LiTS as a collection of abdominal CT scans, but does not identify it as the source of OrganMNIST3D or Organ{A,C,S}MNIST. Searches for those dataset names and for 3D bounding boxes, 28 × 28 × 28 images, and classification of 11 organs found no supporting passage in our copy. (LLM, conf=0.75)

  cites [30] Bilic, P. et al. The liver tumor segmentation benchmark (lits). Medical Image Analysis 84,102680 (2023). sourceExperimental — accuracy not yet measured
• ?Not checked — no full text of the sourcecites [31] Xu, X. et al. Efficient multiple organ localization in ct image using 3d region proposal network. IEEE Transactions on Medical Imaging 38, 1885–1898 (2019). source

In the paper

“The NoduleMNIST3D is based on the LIDC-IDRI32, a large public lung nodule dataset, containing images from thoracic CT scans. The dataset is designed for both lung nodule segmentation and 5-level malignancy classification task. To perform binary classification, we categorize cases with malignancy level 1/2 into negative class and 4/5 into positive class, ignoring the cases with malignancy level 3.”

What the sources say

• Supported

  The citation is attached to the identification of LIDC-IDRI as a large, public dataset of lung nodules from thoracic CT scans. The paper reports 1018 thoracic CT scans (II.A), describes its nodule annotations (Results), and says the database is publicly available (II.E). The later binary-classification rule is presented as the citing authors’ own procedure, not as a finding attributed to this source. (LLM, conf=0.91)

  cites [32] Armato, S. G. III et al. The lung image database consortium (lidc) and image database resource initiative (idri): A completed reference database of lung nodules on ct scans. Medical Physics 38, 915–931, https://doi.org/10.1118/1.3528204 (2011). source

In the paper

“The MedMNIST v2 dataset consists of 12 2D and 6 3D standardized datasets from carefully selected sources covering primary data modalities (e.g., X-ray, OCT, ultrasound, CT, electron microscope), diverse classification tasks (binary/multi-class, ordinal regression, and multi-label) and dataset scales (from 100 to 100,000).”

In the paper

“The MedMNIST uses the classification labels from the source datasets directly in most cases, but the labels could be simplified (merged or deleted classes) if the classification tasks on the small images are too difficult.”

In the paper

“We introduce MedMNIST v2, a large-scale MNIST-like dataset collection of standardized biomedical images, including 12 datasets for 2D and 6 datasets for 3D.”

• Also stated in: § Background & Summary · plain

  “MedMNIST v2 is a large-scale benchmark for 2D and 3D biomedical image classification, covering 12 2D datasets with 708,069 images and 6 3D datasets with 9,998 images.”

In the paper

“All source datasets are either associated with the Creative Commons (CC) Licenses or developed by us, which allows us to develop derivative datasets based on them.”

In the paper

“Covering primary data modalities in biomedical images, MedMNIST v2 is designed to perform classification on lightweight 2D and 3D images with various dataset scales (from 100 to 100,000) and diverse tasks (binary/multi-class, ordinal regression, and multi-label).”

In the paper

“It is designed to be:Diverse: It covers diverse data modalities, dataset scales (from 100 to 100,000), and tasks (binary/multi-class, multi-label, and ordinal regression).”

In the paper

“Standardized: Each sub-dataset is pre-processed into the same format (see details in Methods), which requires no background knowledge for users.”

In the paper

“Some datasets are under CC-BY-NC license; we have contacted the authors and obtained the permission to re-distribute the datasets.”

In the paper

““{train,val,test}_labels”: an array containing ground-truth labels, with a shape of N × 1 for multi-class/binary-class/ordinal regression datasets, of N × L for multi-lable binary-class datasets. N denotes the number of samples in training/validation/test set and L denotes the number of task labels in the multi-label dataset (i.e., 14 for the ChestMNIST).”

In the paper

““{train,val,test}_images”: an array containing images, with a shape of N × 28 × 28 for 2D gray-scale datasets, of N × 28 × 28 × 3 for 2D RGB datasets, of N × 28 × 28 × 28 for 3D datasets.”

In the paper

“The RetinaMNIST is based on the DeepDRiD24 challenge, which provides a dataset of 1,600 retina fundus images. The task is ordinal regression for 5-level grading of diabetic retinopathy severity.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites [24] DeepDRiD. The 2nd diabetic retinopathy–grading and image quality estimation challenge. https://isbi.deepdr.org/data.html (2020).

In the paper

“The data type of the dataset is uint8.”

In the paper

“For 2D datasets, Google AutoML Vision is well-performing in general, however it could not always win, even compared with the baseline ResNet-18 and ResNet-50.”

In the paper

“The OCTMNIST is based on a prior dataset22,23 of 109,309 valid optical coherence tomography (OCT) images for retinal diseases. The dataset is comprised of 4 diagnosis categories, leading to a multi-class classification task.”

What the sources say

• ?Not checked — no full text of the sourcecites [22] Kermany, D. S. et al. Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell 172, 1122–1131.e9, https://doi.org/10.1016/j.cell.2018.02.010 (2018). source
• ?Not checked — no full text of the sourcecites [23] Kermany, D. S., Zhang, K. & Goldbaum, M. Large dataset of labeled optical coherence tomography (oct) and chest x-ray images https://doi.org/10.17632/rscbjbr9sj.3 (2018). source

In the paper

“For MedMNIST2D, Google AutoML Vision outperforms all the other methods in average AUC, however, it is very close to the performance of baseline ResNets.”

In the paper

“AutoKeras performs well on datasets with large scales, however relatively worse on datasets with small scale.”

In the paper

“The PneumoniaMNIST is based on a prior dataset22,23 of 5,856 pediatric chest X-Ray images. The task is binary-class classification of pneumonia against normal.”

What the sources say

• ?Not checked — no full text of the sourcecites [22] Kermany, D. S. et al. Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell 172, 1122–1131.e9, https://doi.org/10.1016/j.cell.2018.02.010 (2018). source
• ?Not checked — no full text of the sourcecites [23] Kermany, D. S., Zhang, K. & Goldbaum, M. Large dataset of labeled optical coherence tomography (oct) and chest x-ray images https://doi.org/10.17632/rscbjbr9sj.3 (2018). source

In the paper

“The ChestMNIST is based on the NIH-ChestXray14 dataset18, a dataset comprising 112,120 frontal-view X-Ray images of 30,805 unique patients with the text-mined 14 disease labels, which could be formulized as a multi-label binary-class classification task.”

What the sources say

• ?Not checked — no full text of the sourcecites [18] Wang, X. et al. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Conference on Computer Vision and Pattern Recognition, 3462–3471 (2017). source

In the paper

“Under the same ResNet backbone, 2.5D models are inferior to 3D and ACS models and perform worse than auto-sklearn and AutoKeras.”

In the paper

“They are sliced before being resized into 1 × 28 × 28.”

In the paper

“Since there is no severe class imbalance on our datasets”

In the paper

“Organ{A,C,S}MNIST and OrganMNIST3D are generated from the same source dataset, and share the same task and the same data split.”

In the paper

“On the other hand, OrganMNIST3D is resized into 28 × 28 × 28 directly.”

In the paper

“Organ{A,C,S}MNIST are sampled slices of 3D bounding boxes of 3D CT images in axial/coronal/sagittal views (planes), respectively.”

In the paper

“The ResNets surpass auto-sklearn and AutoKeras, and outperform Google AutoML Vision in average ACC.”

In the paper

“All benchmark experiments are repeated at least 3 times for more stable results than in MedMNIST v1.”

In the paper

“It is as diverse as the VDD4 and MSD5 to fairly evaluate the generalizable performance of machine learning algorithms in different settings, but both 2D and 3D biomedical images are provided.”

What the sources say

• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites VDD4
• ?Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  cites MSD5

In the paper

“The resulting dataset, consisting of 708,069 2D images and 9,998 3D images in total, could support numerous research/educational purposes in biomedical image analysis, computer vision, and machine learning.”

In the paper

“For the dataset without an official split, we split the dataset randomly at the patient level with a ratio of 7:1:2 into training-validation-test.”

In the paper

“The diverse dataset design could lead to diverse task difficulty, which is desirable as a biomedical image classification benchmark.”

In the paper

“We use the official data split from source dataset (if provided) to avoid data leakage.”

In the paper

“ACC could also serve as a good metric”

In the paper

“We calculate the mean value of at least 3 trials for each method on each dataset.”