Cited by 6,781Brain Tumor Detection and ClassificationMedical Image Segmentation TechniquesGlioma Diagnosis and Treatment

The multimodal brain tumor image segmentation benchmark (BRATS)

Open access
Menze, Bjoern H., Jakab, Andras, Bauer, Stefan, Kalpathy-Cramer, Jayashree, Farahani, Keyvan, Kirby, Justin, Burren, Yuliya, Porz, Nicole +60 more·2015·IEEE Transactions on Medical Imaging

Abstract

In this paper we report the set-up and results of the Multimodal Brain Tumor Image Segmentation Benchmark (BRATS) organized in conjunction with the MICCAI 2012 and 2013 conferences. Twenty state-of-the-art tumor segmentation algorithms were applied to a set of 65 multi-contrast MR scans of low- and high-grade glioma patients-manually annotated by up to four raters-and to 65 comparable scans generated using tumor image simulation software. Quantitative evaluations revealed considerable disagreement between the human raters in segmenting various tumor sub-regions (Dice scores in the range 74%-85%), illustrating the difficulty of this task. We found that different algorithms worked best for different sub-regions (reaching performance comparable to human inter-rater variability), but that no single algorithm ranked in the top for all sub-regions simultaneously. Fusing several good algorithms using a hierarchical majority vote yielded segmentations that consistently ranked above all individual algorithms, indicating remaining opportunities for further methodological improvements. The BRATS image data and manual annotations continue to be publicly available through an online evaluation system as an ongoing benchmarking resource.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tmi.2014.2377694

Cites

132 of 132 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Different lesion appearances and annotation targets have been reported across tumor benchmarks.

    “different lesion appearances and annotation targets have been reported across tumor benchmarks [3,8,23]”
  • Tumor segmentation is less stable because lesions are not tied to a fixed anatomical shape or location and may be small, heterogeneous, or weakly contrasted.

    “Tumor segmentation is less stable because lesions are not tied to a fixed anatomical shape or location and may be small, heterogeneous, or weakly contrasted [3,8,23].”
  • Among expert annotators there is often considerable variability.

    “Among expert annotators there is often considerable variability206,207.”
  • The BraTS 2018 training dataset included 285 cases (210 HGG and 75 LGG), each with four 3D MRI modalities (T1, T1c, T2 and FLAIR) rigidly aligned, resampled to 1x1x1 mm isotropic resolution and skull-stripped.

    “This year, BraTS 2018 training dataset included 285 cases (210 HGG and 75 LGG), each with four 3D MRI modalities (T1, T1c, T2 and FLAIR) rigidly aligned, resampled to 1x1x1 mm isotropic resolution and skull-stripped.”
  • The BraTS 2018 testing dataset contained 191 cases, allowed only a single submission, and was used to calculate the final challenge ranking.

    “The testing dataset (191 cases) allowed only a single submission, and was used to calculate the final challenge ranking.”

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 6

  1. D. Zikic, B. Glocker, E. Konukoglu, A. Criminisi, C. Demiralp, J. Shotton, O. M. Thomas, T. Das, R. Jena, and P. S. J., "Decision forests for tissue-specific segmentation of high-grade gliomas in multi- channel MR," in Proc MICCAI, 2012.
  2. B. H. Menze, K. Van Leemput, D. Lashkari, M.-A. Weber, N. Ayache, and P. Golland, "Segmenting glioma in multi-modal images using a generative model for brain lesion segmentation," in Proc MICCAI- BRATS (Multimodal Brain Tumor Segmentation Challenge), 2012, p. 7p.
  3. S. Bauer, T. Fejes, J. Slotboom, R. Wiest, L.-P. Nolte, and M. Reyes, "Segmentation of Brain Tumor Images Based on Integrated Hierarchi- cal Classification and Regularization," in Proc MICCAI-BRATS, 2012.
  4. S. Bauer, L.-P. Nolte, and M. Reyes, "Fully Automatic Segmentation of Brain Tumor Images using Support Vector Machine Classification in Combination with Hierarchical Conditional Random Field Regular- ization," Medical image computing and computer-assisted intervention : MICCAI ... International Conference on Medical Image Computing and Computer-Assisted Intervention, vol. 14, 2011.
  5. J. Sled and G. Pike, "Correction for B(1) and B(0) variations in quantitative T2 measurements using MRI," Mag. Reson. Med., vol. 43, no. 4, pp. 589-593, 2000.
  6. Y. Boykov, O. Veksler, and R. Zabih., "Efficient approximate energy minimization via graph cuts," IEEE TPAMI, vol. 20, pp. 1222-1239, 2001.

Cited by

Sorted by citations

Authors
Menze, B. H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., Burren, Y., Porz, N., Slotboom, J., Wiest, R., Lanczi, L., Gerstner, E., Weber, M.-A., Arbel, T., Avants, B. B., Ayache, N., Buendia, P., Collins, D. L., Cordier, N., Corso, J. J., Criminisi, A., Das, T., Delingette, H., Demiralp, C., Durst, C. R., Dojat, M., Doyle, S., Festa, J., Forbes, F., Geremia, E., Glocker, B., Golland, P., Guo, X., Hamamci, A., Iftekharuddin, K. M., Jena, R., John, N. M., Konukoglu, E., Lashkari, D., Mariz, J. A., Meier, R., Pereira, S., Precup, D., Price, S. J., Raviv, T. R., Reza, S. M. S., Ryan, M., Sarikaya, D., Schwartz, L., Shin, H.-C., Shotton, J., Silva, C. A., Sousa, N., Subbanna, N. K., Szekely, G., Taylor, T. J., Thomas, O. M., Tustison, N. J., Unal, G., Vasseur, F., Wintermark, M., Ye, D. H., Zhao, L., Zhao, B., Zikic, D., Prastawa, M., Reyes, M., & Van Leemput, K.
Affiliations
  • Institut national de recherche en sciences et technologies du numérique (FR)
  • University of Bern (CH)
  • National Institutes of Health (US)
  • University Hospital of Bern (CH)
  • Harvard University (US)
  • McGill University (CA)
  • University of Pennsylvania (US)
  • Infotech Soft (United States) (US)
  • University at Buffalo, State University of New York (US)
  • Microsoft Research (United Kingdom) (GB)
  • Cambridge University Hospitals NHS Foundation Trust (GB)
  • University of Virginia (US)
  • University of Minho (PT)
  • Imperial College London (GB)
  • MIT Computer Science and Artificial Intelligence Laboratory (US)
  • Columbia University (US)
  • Old Dominion University (US)
  • University of Miami (US)
  • Ben-Gurion University of the Negev (IL)
  • University Hospital Heidelberg (DE)
  • Purdue University West Lafayette (US)
  • GE Global Research (United States) (US)
Corresponding author
Menze, Bjoern H.
Venue
IEEE Transactions on Medical Imaging
Published
2015-10
Volume
34
Issue
10
Pages
1993-2024
ISSN
  • 0278-0062 (ISSN-L)
  • 1558-254X
OpenAlex
W1641498739
Cited by
6,781
References
132
Citations by year
2026: 863 · 2025: 937 · 2024: 899 · 2023: 845 · 2022: 791 · 2021: 709 · 2020: 612 · 2019: 468 · 2018: 310 · 2017: 161 · 2016: 120 · 2015: 49 · 2014: 16
FWCI
9.37
Type
journal article
Language
en
Open access
Locations
  • IEEE Transactions on Medical Imaging (publishedVersion, other-oa)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
  • ODU Digital Commons (Old Dominion University) (submittedVersion)
  • HAL (submittedVersion, other-oa)
  • DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc-sa)
Funders
  • National Science Foundation
  • National Institute for Health and Care Research (NIHR/CS/009/011)
  • European Commission (600841)
  • Academy of Finland
  • Tekes
  • Lundbeckfonden (R141-2013-13117)
  • Krebsliga Schweiz
  • Technische Universität München
  • National Institutes of Health
  • Fundação para a Ciência e a Tecnologia
  • National Cancer Institute (U01CA154601, R15CA115464)
  • National Institute of Biomedical Imaging and Bioengineering (P41EB015902, U54EB005149, R01EB013565)
  • National Center for Research Resources (P41-RR-14075, P41RR014075, P41RR013218, P41 RR13218)
Topics
  • Life Sciences › Neuroscience › Neurology › Brain Tumor Detection and Classification
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
  • Health Sciences › Medicine › Genetics › Glioma Diagnosis and Treatment
Keywords
brain tumor segmentation, glioma, multimodal MRI, benchmark, interobserver variability, algorithm fusion, majority voting, manual annotation
MeSH terms
Algorithms, Glioma, Humans, Magnetic Resonance Imaging, Benchmarking, Neuroimaging

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review