MRBrainS Challenge: Online Evaluation Framework for Brain Image Segmentation in 3T MRI Scans
Open accessAbstract
Many methods have been proposed for tissue segmentation in brain MRI scans. The multitude of methods proposed complicates the choice of one method above others. We have therefore established the MRBrainS online evaluation framework for evaluating (semi)automatic algorithms that segment gray matter (GM), white matter (WM), and cerebrospinal fluid (CSF) on 3T brain MRI scans of elderly subjects (65–80 y). Participants apply their algorithms to the provided data, after which their results are evaluated and ranked. Full manual segmentations of GM, WM, and CSF are available for all scans and used as the reference standard. Five datasets are provided for training and fifteen for testing. The evaluated methods are ranked based on their overall performance to segment GM, WM, and CSF and evaluated using three evaluation metrics (Dice, H95, and AVD) and the results are published on the MRBrainS13 website. We present the results of eleven segmentation algorithms that participated in the MRBrainS13 challenge workshop at MICCAI, where the framework was launched, and three commonly used freeware packages: FreeSurfer, FSL, and SPM. The MRBrainS evaluation framework provides an objective and direct comparison of all evaluated algorithms and can aid in selecting the best performing method for the segmentation goal at hand.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421 · see citing works
Cortes, C., & Vapnik, V. N. (1995) · Machine Learning · doi:10.1023/a:1022627411411 · Cited by 34,565 · see citing works
Measures of the Amount of Ecologic Association Between Species
Dice, L. R. (1945) · Ecology · doi:10.2307/1932409 · Cited by 12,153 · see citing works
Jenkinson, M., Beckmann, C. F., Behrens, T. E. J., Woolrich, M. W., & Smith, S. M. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2011.09.015 · Cited by 11,314 · see citing works
Cortical Surface-Based Analysis
Dale, A. M., Fischl, B., & Sereno, M. I. (1999) · NeuroImage · doi:10.1006/nimg.1998.0395 · Cited by 10,910 · see citing works
Fast robust automated brain extraction
Smith, S. M. (2002) · Human Brain Mapping · doi:10.1002/hbm.10062 · Cited by 10,496 · see citing works
Ashburner, J., & Friston, K. J. (2005) · NeuroImage · doi:10.1016/j.neuroimage.2005.02.018 · Cited by 7,507 · see citing works
Cortical Surface-Based Analysis
Fischl, B., Sereno, M. I., & Dale, A. M. (1999) · NeuroImage · doi:10.1006/nimg.1998.0396 · Cited by 6,147 · see citing works
A nonparametric method for automatic correction of intensity nonuniformity in MRI data
Sled, J. G., Zijdenbos, A. P., & Evans, A. C. (1998) · IEEE Transactions on Medical Imaging · doi:10.1109/42.668698 · Cited by 4,731 · see citing works
Comparing images using the Hausdorff distance
Huttenlocher, D. P., Klanderman, G. A., & Rucklidge, W. J. (1993) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.232073 · Cited by 4,545 · see citing works
Elastix: a toolbox for intensity-based medical image registration
Klein, S., Staring, M., Murphy, K., Viergever, M. A., & Pluim, J. (2010) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2009.2035616 · Cited by 4,311 · see citing works
Self-citationStatistical Parametric Mapping: The Analysis of Functional Brain Images
Penny, W. D., Friston, K. J., Ashburner, J., Kiebel, S. J., & Thomas, N. (2007) · UCL Discovery (University College London) · Cited by 3,339 · see citing works
Current Methods in Medical Image Segmentation
Pham, D. L., Xu, C., & Prince, J. L. (2000) · Annual Review of Biomedical Engineering · doi:10.1146/annurev.bioeng.2.1.315 · Cited by 2,352 · see citing works
101 Labeled Brain Images and a Consistent Human Cortical Labeling Protocol
Klein, A., & Tourville, J. A. (2012) · Frontiers in Neuroscience · doi:10.3389/fnins.2012.00171 · Cited by 1,215 · see citing works
Fast free-form deformation using graphics processing units
Marc Modat, A., Ridgway, G. R., Taylor, Z. A., Lehmann, M., Barnes, J., Hawkes, D. J., Fox, N. C., & Sebastien Ourselin, A. (2009) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2009.09.002 · Cited by 1,087 · see citing works
Review of MR image segmentation techniques using pattern recognition
Bezdek, J. C., Hall, L., & Clarke, L. P. (1993) · Medical Physics · doi:10.1118/1.597000 · Cited by 956 · see citing works
MRI segmentation: Methods and applications
Clarke, L. P., Velthuizen, R. P., Camacho, M. A., Heine, J., Vaidyanathan, M., Hall, L., Thatcher, R. W., & Silbiger, M. L. (1995) · Magnetic Resonance Imaging · doi:10.1016/0730-725x(94)00124-l · Cited by 697 · see citing works
Longitudinal pattern of regional brain volume change differentiates normal aging from MCI
Driscoll, I. F., Davatzikos, C. A., An, Y., Wu, X., Shen, D. G., Kraut, M. A., & Resnick, S. M. (2009) · Neurology · doi:10.1212/wnl.0b013e3181a82634 · Cited by 527 · see citing works
Review of brain MRI image segmentation methods
Balafar, M. A., Ramli, A. R., Saripan, M. I., & Mashohor, S. (2010) · Artificial Intelligence Review · doi:10.1007/s10462-010-9155-0 · Cited by 440 · see citing works
Ritter, F., Boskamp, T., Homeyer, A., Laue, H., Schwier, M., Link, F., & Peitgen, H. (2011) · IEEE Pulse · doi:10.1109/mpul.2011.942929 · Cited by 366 · see citing works
Cuadra, M. B., Cammoun, L., Butz, T., Cuisenaire, O., & Thiran, J. (2005) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2005.857652 · Cited by 365 · see citing works
Whitwell, J. L., Crum, W. R., Watt, H., & Fox, N. C. (2001) · PubMed Central · Cited by 357 · see citing works
Vrooman, H. A., Cocosco, C. A., van der Lijn, F., Stokking, R., Ikram, M. A., Vernooij, M. W., Breteler, M. M. B., & Niessen, W. J. (2007) · NeuroImage · doi:10.1016/j.neuroimage.2007.05.018 · Cited by 313 · see citing works
Self-citationFranke, K., & Christian Gaser, B. (2012) · GeroPsych · doi:10.1024/1662-9647/a000074 · Cited by 259 · see citing works
Multispectral analysis of magnetic resonance images.
Vannier, M. W., Butterfield, R. L., Jordan, D., et al. (1985) · Radiology · doi:10.1148/radiology.154.1.3964938 · Cited by 254 · see citing works
LINKS: Learning-based multi-source IntegratioN frameworK for Segmentation of infant brain images
Wang, L., Gao, Y., Shi, F., Li, G., Gilmore, J. H., Lin, W., & Shen, D. G. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2014.12.042 · Cited by 231 · see citing works
Brain segmentation and white matter lesion detection in MR images.
Zijdenbos, A. P., & Dawant, B. M. (1994) · PubMed · Cited by 224 · see citing works
Evaluation of automated brain MR image segmentation and volumetry methods
Klauschen, F., Goldman, A. L., Barra, V., Meyer‐Lindenberg, A., & Lundervold, A. (2008) · Human Brain Mapping · doi:10.1002/hbm.20599 · Cited by 217 · see citing works
3D Segmentation in the Clinic: A Grand Challenge
van Ginneken, B., Heimann, T., & Styner, M. A. (2007) · Cited by 212 · see citing works
Brain tissue volumes in the general elderly population
Ikram, M. A., Vrooman, H. A., Vernooij, M. W., van der Lijn, F., Albert Hofman, D., van der Lugt, A., Niessen, W. J., & Breteler, M. M. B. (2007) · Neurobiology of Aging · doi:10.1016/j.neurobiolaging.2006.12.012 · Cited by 189 · see citing works
Self-citationOnline resource for validation of brain segmentation methods
Shattuck, D. W., Prasad, G., Mirza, M., Narr, K. L., & Toga, A. W. (2008) · NeuroImage · doi:10.1016/j.neuroimage.2008.10.066 · Cited by 167 · see citing works
Popescu, V., Battaglini, M., Hoogstrate, W. S., Verfaillie, S. C. J., Sluimer, I. C., van Schijndel, R. A., van Dijk, B. W., Cover, K. S., Knol, D. L., Jenkinson, M., et al. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.03.074 · Cited by 165 · see citing works
Clinical use of brain volumetry
Giorgio, A., & Stefano, N. D. (2012) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.23671 · Cited by 153 · see citing works
Reijmer, Y. D., Leemans, A., Brundel, M., Kappelle, L. J., Biessels, G. J., & on behalf of the Utrecht Vascular Cognitive Impairment (VCI) Study Group (2013) · Diabetes · doi:10.2337/db12-1644 · Cited by 143 · see citing works
Self-citationAccuracy and reproducibility study of automatic MRI brain tissue segmentation methods
de Boer, R., Vrooman, H. A., Ikram, M. A., Vernooij, M. W., Breteler, M. M. B., van der Lugt, A., & Niessen, W. J. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2010.03.012 · Cited by 134 · see citing works
Self-citationDiagnostic and Sex Effects on Limbic Volumes in Early-Onset Bipolar Disorder and Schizophrenia
Frazier, J. A., Hodge, S. M., Breeze, J. L., Giuliano, A. J., Terry, J. E., Moore, C. M., Kennedy, D., Lopez-Larson, M. P., Caviness, V. S., Seidman, L. J., et al. (2007) · Schizophrenia Bulletin · doi:10.1093/schbul/sbm120 · Cited by 120 · see citing works
Geometric strategies for neuroanatomic analysis from MRI
Duncan, J. S., Papademetris, X., Yang, J., Jackowski, M. P., Zeng, X., & Staib, L. H. (2004) · NeuroImage · doi:10.1016/j.neuroimage.2004.07.027 · Cited by 119 · see citing works
Simple paradigm for extra-cerebral tissue removal: Algorithm and analysis
Carass, A., Cuzzocreo, J. L., Wheeler, M. B., Bazin, P., Resnick, S. M., & Prince, J. L. (2011) · NeuroImage · doi:10.1016/j.neuroimage.2011.03.045 · Cited by 114 · see citing works
Self-citationZaidi, H., Ruest, T., Schoenahl, F., & Montandon, M.-L. (2006) · NeuroImage · doi:10.1016/j.neuroimage.2006.05.031 · Cited by 108 · see citing works
Evaluation of automatic neonatal brain segmentation algorithms: The NeoBrainS12 challenge
Išgum, I., Benders, M. J. N. L., Avants, B., Cardoso, M. J., Counsell, S. J., Fischi-Gómez, E., Gui, L., Hüppi, P. S., Kersbergen, K. J., Makropoulos, A., et al. (2014) · Medical Image Analysis · doi:10.1016/j.media.2014.11.001 · Cited by 98 · see citing works
Self-citationAutomatic segmentation of MR brain images of preterm infants using supervised classification
Moeskops, P., Benders, M. J. N. L., Chiţ, S. M., Kersbergen, K. J., Groenendaal, F., de Vries, L. S., Viergever, M. A., & Išgum, I. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2015.06.007 · Cited by 87 · see citing works
Self-citationMagnetic resonance image synthesis through patch regression
Jog, A., Roy, S., Carass, A., & Prince, J. L. (2013) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2013.6556484 · Cited by 84 · see citing works
Self-citationInteractive Hierarchical-Flow Segmentation of Scar Tissue From Late-Enhancement Cardiac MR Images
Rajchl, M., Yuan, J. Y., White, J. A., Ukwatta, E., Stirrat, J., Nambakhsh, C. M. S., Li, F. P., & Peters, T. M. (2013) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2013.2282932 · Cited by 72 · see citing works
Self-citationSuri, J. S., Singh, S. K., & Reden, L. (2002) · Pattern Analysis and Applications · doi:10.1007/s100440200005 · Cited by 72 · see citing works
A comparison of MR based segmentation methods for measuring brain atrophy progression
de Bresser, J., Portegies, M., Leemans, A., Biessels, G. J., Kappelle, L. J., & Viergever, M. A. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2010.09.060 · Cited by 55 · see citing works
Self-citationFast level-set based image segmentation using coherent propagation
Wang, C., Frimmel, H., & Smedby, Ö. (2014) · Medical Physics · doi:10.1118/1.4881315 · Cited by 45 · see citing works
Self-citationAnything you can do, I can do better (No you can't)…
Price, K. (1986) · Computer Vision Graphics and Image Processing · doi:10.1016/0734-189x(86)90083-6 · Cited by 37 · see citing works
Lisboa, P. J. G., Kirby, S., Vellido, A., Lee, Y. Y. B., & El‐Deredy, W. (1998) · NMR in Biomedicine · doi:10.1002/(sici)1099-1492(199806/08)11:4/5 · Cited by 30 · see citing works
An integrated geometrical and stochastic approach for accurate infant brain extraction
Alansary, A., Soliman, A., Nitzken, M., et al. (2014) · Proceedings - International Conference on Image Processing · doi:10.1109/icip.2014.7025719 · Cited by 18 · see citing works
Self-citationEvaluation of an automatic brain segmentation method developed for neonates on adult MR brain images
Moeskops, P., Viergever, M. A., Benders, M. J. N. L., & Išgum, I. (2015) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.2081833 · Cited by 16 · see citing works
Self-citationA novel Bayesian approach to adaptive mean shift segmentation of brain images
Mahmood, Q., Chodorowski, A., Mehnert, A., et al. (2012) · IEEE International Symposium on Computer-Based Medical Systems (CBMS) · doi:10.1109/cbms.2012.6266304 · Cited by 16 · see citing works
Self-citationFully automatic brain segmentation using model-guided level sets and skeleton-based models
Wang, C., Wang, C., & Smedby, Ö. (2013) · doi:10.54294/igzmed · Cited by 9 · see citing works
Self-citationFully automatic brain segmentation using model-guided level set and skeleton based models
Wang, C., & Smedby, Ö. (2013) · Cited by 3 · see citing works
Self-citationMultimodal MR Brain Segmentation Using Bayesian-based Adaptive Mean-Shift (BAMS)
Mahmood, Q., Alipoor, M., Chodorowski, A., et al. (2013) · doi:10.54294/fefo4q · Cited by 2 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 9
- [33]Alansary A. Soliman A. Khalifa F. MAP-based framework for segmentation of MR brain images based on visual appearance and prior shape Proceedings of the MICCAI Grand Challenge on MR Brain Image Segmentation (MRBrainS '13) September 2013 The Midas Journal
- [34]Jog A. Roy S. Prince J. L. Carass A. MR brain segmentation using decision trees Proceedings of the MICCAI Grand Challenge on MR Brain Image Segmentation (MRBrainS '13) September 2013 Nagoya, Japan The Midas Journal
- [35]Katyal R. Paneri S. Kuse M. Gaussian intensity model with neighborhood cues for fluid-tissue categorization of multi-sequence MR brain images Proceedings of the MICCAI Grand Challenge on MR Brain Image Segmentation (MRBrainS '13) September 2013 Nagoya, Japan The Midas Journal
- [37]van Opbroek A. van der Lijn F. de Bruijne M. Automated brain-tissue segmentation by multi-feature SVM classification Proceedings of the MICCAI Workshops—The MICCAI Grand Challenge on MR Brain Image Segmentation (MRBrainS '13) September 2013 The Midas Journal
- [38]Pereira S. Festa J. Mariz J. A. Sousa N. Silva C. A. Automatic brain tissue segmentation of multi-sequence MR images using random decision forests Proceedings of the MICCAI Grand Challenge on MR Brain Image Segmentation (MRBrainS '13) September 2013 Nagoya, Japan The Midas Journal
- [39]Rajchl M. Baxter J. Yuan J. Peters T. Khan A. Multi-atlas-based segmentation with hierarchical max-flow Proceedings of the MICCAI Grand Challenge on MR Brain Image Segmentation (MRBrainS '13) September 2013 Nagoya, Japan The Midas Journal
- [40]Sarikaya D. Zhao L. Corso J. J. Multi-atlas brain MRI segmentation with multiway cut Proceedings of the MICCAI Workshops—The MICCAI Grand Challenge on MR Brain Image Segmentation (MRBrainS '13) September 2013 The Midas Journal
- [41]Vrooman H. van der Lijn F. Niessen W. Auto-kNN: brain tissue segmentation using automatically trained k-nearest-neighbor classification Proceedings of the MICCAI Workshops—The MICCAI Grand Challenge on MR Brain Image Segmentation (MRBrainS '13) September 2013 The Midas Journal
- [42]Vyas S. Burlina P. Kleissas D. Mukherjee R. Automated walks using machine learning for segmentation Proceedings of the MICCAI Grand Challenge on MR Brain Image Segmentation (MRBrainS '13) September 2013 Nagoya, Japan The Midas Journal
Cited by
Sorted by citations
Automatic Segmentation of MR Brain Images With a Convolutional Neural Network
Moeskops, P., Viergever, M. A., Mendrik, A. M., de Vries, L. S., Benders, M. J. N. L., & Isgum, I. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2548501 · Cited by 861 · see citing works
Self-citationAI in Medical Imaging Informatics: Current Challenges and Future Directions
Panayides, A. S., Amini, A., Filipovic, N. D., Sharma, A., Tsaftaris, S. A., Young, A., Foran, D., Do, N., Golemati, S., Kurc, T., et al. (2020) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2020.2991043 · Cited by 713 · see citing works
- Authors
- Mendrik, A. M., Vincken, K. L., Kuijf, H. J., Breeuwer, M., Bouvy, W. H., de Bresser, J., Alansary, A., de Bruijne, M., Carass, A., El-Baz, A., Jog, A., Katyal, R., Khan, A. R., van der Lijn, F., Mahmood, Q., Mukherjee, R., van Opbroek, A., Paneri, S., Pereira, S., Persson, M., Rajchl, M., Sarikaya, D., Smedby, Ö., Silva, C. A., Vrooman, H. A., Vyas, S., Wang, C., Zhao, L., Biessels, G. J., & Viergever, M. A.
- Affiliations
- University Medical Center Utrecht (NL)
- Philips (Netherlands) (NL)
- Department of Radiology, University Medical Center Utrecht, 3584 CX Utrecht, Netherlands
- University of Louisville (US)
- Biomedical Imaging Group Rotterdam, Departments of Medical Informatics and Radiology, Erasmus MC, 3015 CN Rotterdam, Netherlands
- Johns Hopkins University (US)
- LNM Institute of Information Technology (IN)
- Western University (CA)
- Chalmers University of Technology (SE)
- University of Minho (PT)
- Robarts Research Institute (CA)
- Computer Science and Engineering Department, State University of New York at Buffalo, Buffalo, NY 14260-2500, USA
- Linköping University (SE)
- Erasmus MC (NL)
- University at Buffalo, State University of New York (US)
- Corresponding author
- Mendrik, Adriënne M.
- ORCID
- Venue
- Computational Intelligence and Neuroscience
- Publisher
- Wiley
- Published
- 2015
- Volume
- 2015
- Pages
- 1-16
- ISSN
- 1687-5265 (ISSN-L)
- 1687-5273
- PMC
- PMC4680055
- OpenAlex
- W1794121648
- Cited by
- 261
- References
- 54
- Citations by year
- 2026: 7 · 2025: 25 · 2024: 11 · 2023: 35 · 2022: 28 · 2021: 30 · 2020: 26 · 2019: 39 · 2018: 22 · 2017: 19 · 2016: 16 · 2015: 1 · 2014: 1
- FWCI
- 16.75
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Computational Intelligence and Neuroscience (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Research at the University of Copenhagen (University of Copenhagen) (publishedVersion, other-oa)
- Chalmers Research (Chalmers University of Technology) (submittedVersion)
- TU/e Research Portal (submittedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion)
- RePub (Erasmus University Rotterdam) (submittedVersion, other-oa)
- KTH Publication Database DiVA (KTH Royal Institute of Technology) (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion, cc-by)
- TU/e Research Portal (submittedVersion, other-oa)
- Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT) (submittedVersion, other-oa)
- Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT) (submittedVersion, cc-by-nc-nd)
- RePub (Erasmus University Rotterdam) (submittedVersion)
- Spiral (Imperial College London) (submittedVersion, other-oa)
- Funders
- ZonMw (104002002, 2010T073)
- Netherlands Heart Foundation (104002002, 2010T073)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
- Life Sciences › Neuroscience › Neurology › Brain Tumor Detection and Classification
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Advanced MRI Techniques and Applications
- Keywords
- brain MRI segmentation, tissue segmentation, gray matter segmentation, white matter segmentation, cerebrospinal fluid segmentation, 3T MRI, elderly brain, FreeSurfer, FSL, SPM, algorithm comparison
- MeSH terms
- Aged, Aged, 80 and over, Algorithms, Brain, Cerebrospinal Fluid, Female, Humans, Image Processing, Computer-Assisted, Magnetic Resonance Imaging, Male, Online Systems, Reference Standards, Software, Reproducibility of Results, Databases, Factual, White Matter, Gray Matter
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
