The DOI matches its Crossref record; 132 of the 132 references it cites were found in a registry or an index.
The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS)
Open accessAbstract
In this paper we report the set-up and results of the Multimodal Brain Tumor Image Segmentation Benchmark (BRATS) organized in conjunction with the MICCAI 2012 and 2013 conferences. Twenty state-of-the-art tumor segmentation algorithms were applied to a set of 65 multi-contrast MR scans of low- and high-grade glioma patients-manually annotated by up to four raters-and to 65 comparable scans generated using tumor image simulation software. Quantitative evaluations revealed considerable disagreement between the human raters in segmenting various tumor sub-regions (Dice scores in the range 74%-85%), illustrating the difficulty of this task. We found that different algorithms worked best for different sub-regions (reaching performance comparable to human inter-rater variability), but that no single algorithm ranked in the top for all sub-regions simultaneously. Fusing several good algorithms using a hierarchical majority vote yielded segmentations that consistently ranked above all individual algorithms, indicating remaining opportunities for further methodological improvements. The BRATS image data and manual annotations continue to be publicly available through an online evaluation system as an ongoing benchmarking resource.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“different lesion appearances and annotation targets have been reported across tumor benchmarks [3,8,23]”
From Correction-aware interactive 3D tumor segmentation with sparse and revisable prompts (2026) · cites this paper as [23]
The cited source says what the claim attributes to it.
In the citing paper
“Tumor segmentation is less stable because lesions are not tied to a fixed anatomical shape or location and may be small, heterogeneous, or weakly contrasted [3,8,23].”
From Correction-aware interactive 3D tumor segmentation with sparse and revisable prompts (2026) · cites this paper as [23]
The cited source says what the claim attributes to it.
In the citing paper
“Among expert annotators there is often considerable variability206,207.”
From The METRIC-framework for assessing data quality for trustworthy AI in medicine: A systematic review (2024) · cites this paper as [207]
In the citing paper
“This year, BraTS 2018 training dataset included 285 cases (210 HGG and 75 LGG), each with four 3D MRI modalities (T1, T1c, T2 and FLAIR) rigidly aligned, resampled to 1x1x1 mm isotropic resolution and skull-stripped.”
From 3D MRI Brain Tumor Segmentation Using Autoencoder Regularization (2019) · cites this paper as [18]
In the citing paper
“The testing dataset (191 cases) allowed only a single submission, and was used to calculate the final challenge ranking.”
From 3D MRI Brain Tumor Segmentation Using Autoencoder Regularization (2019) · cites this paper as [18]
In the citing paper
“The validation dataset (66 cases) allowed multiple submissions and was designed for intermediate evaluations.”
From 3D MRI Brain Tumor Segmentation Using Autoencoder Regularization (2019) · cites this paper as [18]
In the citing paper
“Further noting that the preliminary federation model was already within the intra- and inter-rater variability range for this use-case (20% and 28%, respectively)47, any further improvements for the full federation consensus model would be expected to be minimal35–38.”
From Federated learning enables big data for rare cancer boundary detection (2022) · cites this paper as [35]
In the citing paper
“The complexity of boundary detection drops when moving from smaller to larger sub-compartments, i.e., from ET to TC, and then to WT35–38.”
From Federated learning enables big data for rare cancer boundary detection (2022) · cites this paper as [35]
In the citing paper
“Moreover, varying characteristics of the mpMRI data due to scanner hardware and acquisition protocol differences33,34 were handled at each collaborating site via established harmonized preprocessing pipelines35–39.”
From Federated learning enables big data for rare cancer boundary detection (2022) · cites this paper as [35]
In the citing paper
“Additionally, we used a single off-the-shelf neural network architecture for training, but it has been shown that model ensembles perform better for the task at hand35–38, and it remains to be explored how such a strategy could be explored in a federated study.”
From Federated learning enables big data for rare cancer boundary detection (2022) · cites this paper as [35]
In the citing paper
“The BraTS challenge35–38, seeking methodological advancements in the domain of neuro-oncology, has been providing the community with (i) the largest publicly available and manually-curated mpMRI dataset of diffuse glioma patients (an example of which is illustrated in Fig. 1b), and (ii) a harmonized preprocessing pipeline51,70,71 to handle differences in inter-site acquisition protocols.”
From Federated learning enables big data for rare cancer boundary detection (2022) · cites this paper as [35]
In the citing paper
“This functionality follows the harmonized data preprocessing protocol defined by the BraTS challenge35–38, as described below. This would allow accounting for inter-site acquisition protocol variations, e.g., 3D vs. 2D axial plane acquisitions.”
From Federated learning enables big data for rare cancer boundary detection (2022) · cites this paper as [35]
In the citing paper
“which is typically done in academic research projects35–38”
From Federated learning enables big data for rare cancer boundary detection (2022) · cites this paper as [35]
In the citing paper
“Following the precise definition of our problem statement35–38, ensuring strict compliance with the preprocessing and annotation protocol for the generation of reference standards was vital for the model to learn correct information during training.”
From Federated learning enables big data for rare cancer boundary detection (2022) · cites this paper as [35]
In the citing paper
“Eaton-Rosene et al. [75] proposed a generalizable technique for quantifying uncertainty with Bayesian Neural Networks for semantic segmentation on the BraTS 2017 dataset [76], including 285 subjects with high-or low-grade gliomas.”
From Explainable AI in medical imaging: An overview for clinical practitioners – Beyond saliency-based XAI approaches (2023) · cites this paper as [76]
In the citing paper
“For tumor segmentation in the BraTS dataset Menze et al. (2014) (120 training and 50 test MR scans), the authors report six and seven points increase in Dice over supervised model when 20 and 50 labeled MR images are used for training.”
From Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation (2020) · cites this paper as Menze et al. (2014)
In the citing paper
“The suggested method outperforms other modifications such as CRF post-processing as well as sophisticated test-time data augmentation, winning the first place in the 2018 BraTS challenge”
From Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation (2020) · cites this paper as Menze et al. (2014)
In the citing paper
“their 3D generalization on BraTS dataset Menze et al. (2014) (274 training and 110 testing) leads to marginal performance gains in Dice, i.e. 3.7% over Random Forests, 0.3% over an ensemble method, and merely 0.7% over their proposed architecture”
From Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation (2020) · cites this paper as Menze et al. (2014)
In the citing paper
“And on BraTS 2015 Menze et al. (2014), consisting of 220 multi-modal MR images of brain tumors, 3D RNN-CRF slightly improves the Dice for tumor segmentation from 73.5% ± 10.5% to 73.8% ± 10.5% based upon a split of 85%/15% (187 training and 33 testing).”
From Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation (2020) · cites this paper as Menze et al. (2014)
In the citing paper
“Brain Tumor Image Segmentation Benchmark (BraTS) [104] is a benchmark dataset for brain tumor segmentation.”
From Federated learning for medical image analysis: A survey (2024) · cites this paper as [104]
In the citing paper
“It contains brain MRIs acquired by various scanners from around 19 independent institutions.”
From Federated learning for medical image analysis: A survey (2024) · cites this paper as [104]
In the citing paper
“The existing automatic and semi-automatic brain tumor segmentation methods can be broadly categorized as either generative model based or discriminative model based methods (Menze et al., 2015).”
From A deep learning model integrating FCNNs and CRFs for brain tumor segmentation (2018) · cites this paper as Menze et al., 2015
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421
Maximum Likelihood from Incomplete Data Via the EM Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1977.tb01600.x · Cited by 50,444
Cortes, C., & Vapnik, V. N. (1995) · Machine Learning · doi:10.1023/a:1022627411411 · Cited by 34,565
New response evaluation criteria in solid tumours: revised RECIST guideline (version 1.1)
Eisenhauer, E. A., Therasse, P., Bogaerts, J., Schwartz, L. H., Sargent, D., Ford, R., Dancey, J., Arbuck, S., Gwyther, S., Mooney, M., et al. (2009) · European Journal of Cancer · doi:10.1016/j.ejca.2008.10.026 · Cited by 29,534
Self-citationThe Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
The 2007 WHO Classification of Tumours of the Central Nervous System
Louis, D. N., Ohgaki, H., Wiestler, O. D., Cavenee, W. K., Burger, P. C., Jouvet, A., Scheithauer, B. W., & Kleihues, P. (2007) · Acta Neuropathologica · doi:10.1007/s00401-007-0243-4 · Cited by 15,656
Robust Real-Time Face Detection
Viola, P., & Jones, M. J. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000013087.49260.fb · Cited by 14,097
Flagged first, then by citations
Research works printed but not linked: 10
- 10.1007/s00401-007-0278-6 · DOI
- Menze, Segmenting glioma in multi-modal images using a generative model for brain lesion segmentation, 1
- Ibanez, 2003, The ITK Software Guide
- Cocosco, 1997, Brainweb: Online interface to a 3D MRI simulated brain database, NeuroImage, 301
- Bauer, Segmentation of brain tumor images based on integrated hierarchical classification and regularization, 32
- 10.1007/978-3-642-33454-2_46 · DOI · Paper page
- 10.2307/2984875 · DOI
- 10.1007/978-3-642-23626-6_44 · DOI
- 10.29172/7c2a6982-6d72-4cd8-bba6-2fccb06a7011 · DOI
- 10.1109/34.969114 · DOI · Paper page
Cited by
Sorted by citations
Items 101–115 of 115
MMedAgent: Learning to Use Medical Tools with Multi-modal Agent
Li, B., Yan, T., Pan, Y., Luo, J., Ji, R., Ding, J., Xu, Z., Liu, S., Dong, H., Lin, Z., et al. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.510 · Cited by 54
Pati, S., Baid, U., Edwards, B., Sheller, M. J., Foley, P., Anthony Reina, G., Thakur, S., Sako, C., Bilello, M., Davatzikos, C., et al. (2022) · Physics in Medicine & Biology · doi:10.1088/1361-6560/ac9449 · Cited by 51
Self-citationGrøvik, E., Yi, D., Iv, M., Tong, E., Nilsen, L. B., Latysheva, A., Saxhaug, C., Jacobsen, K. D., Helland, Å., Emblem, K. E., et al. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00398-4 · Cited by 51
Bakas, S., Vollmuth, P., Galldiks, N., Booth, T. C., Aerts, H. J. W. L., Bi, W. L., Wiestler, B., Tiwari, P., Pati, S., Baid, U., et al. (2024) · The Lancet Oncology · doi:10.1016/s1470-2045(24)00315-2 · Cited by 36
Lee, W.-K., Wu, C.-C., Lee, C.-C., Lu, C.-F., Yang, H.-C., Huang, T.-H., Lin, C.-Y., Chung, W.-Y., Wang, P.-S., Wu, H.-M., et al. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2020.101911 · Cited by 35
Ahmadi, S.-A., Bötzel, K., Levin, J., Maiostre, J., Klein, T., Wein, W., Rozanski, V., Dietrich, O., Ertl-Wagner, B., Navab, N., et al. (2020) · NeuroImage: Clinical · doi:10.1016/j.nicl.2020.102185 · Cited by 31
Sharing Data Is Essential for the Future of AI in Medical Imaging
Bell, L. C., & Shimron, E. (2024) · Radiology: Artificial Intelligence · doi:10.1148/ryai.230337 · Cited by 30
Wang, T.-W., Shiao, Y.-C., Hong, J.-S., Lee, W.-K., Hsu, M.-S., Cheng, H.-M., Yang, H.-C., Lee, C.-C., Pan, H.-C., You, W. C., et al. (2024) · Mayo Clinic Proceedings: Digital Health · doi:10.1016/j.mcpdig.2024.01.002 · Cited by 23
Bouget, D., Eijgelaar, R. S., Pedersen, A., Kommers, I., Ardon, H., Barkhof, F., Bello, L., Berger, M. S., Nibali, M. C., Furtner, J., et al. (2021) · Cancers · doi:10.3390/cancers13184674 · Cited by 19
Mitchell, J. R., Kamnitsas, K., Singleton, K. W., Whitmire, S. A., Clark-Swanson, K. R., Ranjbar, S., Rickertsen, C. R., Johnston, S. K., Egan, K. M., Rollison, D. E., et al. (2020) · Journal of Medical Imaging · doi:10.1117/1.jmi.7.5.055501 · Cited by 15
Self-citationModel Ensemble for Brain Tumor Segmentation in Magnetic Resonance Imaging
Capellán-Martín, D., Jiang, Z., Parida, A., Liu, X., Lam, V., Nisar, H., Tapp, A., Elsharkawi, S., Ledesma-Carbayo, M. J., Anwar, S. M., et al. (2024) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-76163-8_20 · Cited by 15
“The brain tumor segmentation (BraTS) challenge held in conjunction with the International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI) conference, established in 2012, has generated a benchmark dataset for the segmentation of adult brain gliomas [4,5,6,7,21].”
MethodBeyond Medical Imaging - A Review of Multimodal Deep Learning in Radiology
Heiliger, L., Sekuboyina, A., Menze, B., Egger, J., & Kleesiek, J. (2022) · doi:10.36227/techrxiv.19103432 · Cited by 13
Self-citationPrabhudesai, S., Wang, N. C., Ahluwalia, V., Huan, X., Bapuraj, J. R., Banovic, N., & Rao, A. (2021) · Frontiers in Neuroscience · doi:10.3389/fnins.2021.740353 · Cited by 13
Qian, C., Yang, H., & Li, J. (2023) · 2023 IEEE International Conference on Systems, Man, and Cybernetics (SMC) · doi:10.1109/smc53992.2023.10393973 · Cited by 1
Correction-aware interactive 3D tumor segmentation with sparse and revisable prompts
Li, H., & Li, H. (2026) · The Visual Computer · doi:10.1007/s00371-026-04560-5
Metadata
- Authors
- Menze, B. H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., Burren, Y., Porz, N., Slotboom, J., Wiest, R., Lanczi, L., Gerstner, E., Weber, M.-A., Arbel, T., Avants, B. B., Ayache, N., Buendia, P., Collins, D. L., Cordier, N., Corso, J. J., Criminisi, A., Das, T., Delingette, H., Demiralp, C., Durst, C. R., Dojat, M., Doyle, S., Festa, J., Forbes, F., Geremia, E., Glocker, B., Golland, P., Guo, X., Hamamci, A., Iftekharuddin, K. M., Jena, R., John, N. M., Konukoglu, E., Lashkari, D., Mariz, J. A., Meier, R., Pereira, S., Precup, D., Price, S. J., Raviv, T. R., Reza, S. M. S., Ryan, M., Sarikaya, D., Schwartz, L., Shin, H.-C., Shotton, J., Silva, C. A., Sousa, N., Subbanna, N. K., Szekely, G., Taylor, T. J., Thomas, O. M., Tustison, N. J., Unal, G., Vasseur, F., Wintermark, M., Ye, D. H., Zhao, L., Zhao, B., Zikic, D., Prastawa, M., Reyes, M., & Van Leemput, K.
- Affiliations
- Institut national de recherche en sciences et technologies du numérique (FR)
- University of Bern (CH)
- National Institutes of Health (US)
- University Hospital of Bern (CH)
- Harvard University (US)
- McGill University (CA)
- University of Pennsylvania (US)
- Infotech Soft (United States) (US)
- University at Buffalo, State University of New York (US)
- Microsoft Research (United Kingdom) (GB)
- Cambridge University Hospitals NHS Foundation Trust (GB)
- University of Virginia (US)
- University of Minho (PT)
- Imperial College London (GB)
- MIT Computer Science and Artificial Intelligence Laboratory (US)
- Columbia University (US)
- Old Dominion University (US)
- University of Miami (US)
- Ben-Gurion University of the Negev (IL)
- University Hospital Heidelberg (DE)
- Purdue University West Lafayette (US)
- GE Global Research (United States) (US)
- Corresponding author
- Menze, Bjoern H.
- ORCID
- Menze, Bjoern H.
- Farahani, Keyvan
- Kirby, Justin
- Porz, Nicole
- Slotboom, Johannes
- Wiest, Roland
- Gerstner, Elizabeth
- Arbel, Tal
- Avants, Brian B.
- Ayache, Nicholas
- Buendia, Patricia
- Collins, D. Louis
- Cordier, Nicolas
- Corso, Jason J.
- Criminisi, Antonio
- Das, Tilak
- Durst, Christopher R.
- Dojat, Michel
- Doyle, Senan
- Festa, Joana
- Forbes, Florence
- Glocker, Ben
- Golland, Polina
- Meier, Raphael
- Precup, Doina
- Price, Stephen J.
- Raviv, Tammy Riklin
- Ryan, Michael
- Silva, Carlos A.
- Sousa, Nuno
- Thomas, Owen M.
- Tustison, Nicholas J.
- Wintermark, Max
- Ye, Dong Hye
- Zhao, Liang
- Zhao, Binsheng
- Prastawa, Marcel
- Reyes, Mauricio
- Venue
- IEEE Transactions on Medical Imaging
- Published
- 2015-10
- Volume
- 34
- Issue
- 10
- Pages
- 1993-2024
- ISSN
- 0278-0062 (ISSN-L)
- 1558-254X
- PMC
- PMC4833122
- OpenAlex
- W1641498739
- Cited by
- 6,781
- References
- 132
- Citations by year
- 2026: 863 · 2025: 937 · 2024: 899 · 2023: 845 · 2022: 791 · 2021: 709 · 2020: 612 · 2019: 468 · 2018: 310 · 2017: 161 · 2016: 120 · 2015: 49
- FWCI
- 9.37
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- IEEE Transactions on Medical Imaging (publishedVersion, other-oa)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- ODU Digital Commons (Old Dominion University) (submittedVersion)
- HAL (submittedVersion, other-oa)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc-sa)
- Funders
- National Science Foundation
- National Institute for Health and Care Research (NIHR/CS/009/011)
- European Commission (600841)
- Academy of Finland
- Tekes
- Lundbeckfonden (R141-2013-13117)
- Krebsliga Schweiz
- Technische Universität München
- National Institutes of Health
- Fundação para a Ciência e a Tecnologia
- National Cancer Institute (U01CA154601, R15CA115464)
- National Institute of Biomedical Imaging and Bioengineering (P41EB015902, U54EB005149, R01EB013565)
- National Center for Research Resources (P41-RR-14075, P41RR014075, P41RR013218, P41 RR13218)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Brain Tumor Detection and Classification
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
- Health Sciences › Medicine › Oncology › Glioma Diagnosis and Treatment
- Keywords
- brain tumor segmentation, glioma, multimodal MRI, benchmark, interobserver variability, algorithm fusion, majority voting, manual annotation
- MeSH terms
- Algorithms, Glioma, Humans, Magnetic Resonance Imaging, Benchmarking, Neuroimaging
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
