The DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.
Standardized Assessment of Automatic Segmentation of White Matter Hyperintensities and Results of the WMH Segmentation Challenge
Open accessAbstract
Quantification of cerebral white matter hyperintensities (WMH) of presumed vascular origin is of key importance in many neurological research studies. Currently, measurements are often still obtained from manual segmentations on brain MR images, which is a laborious procedure. The automatic WMH segmentation methods exist, but a standardized comparison of the performance of such methods is lacking. We organized a scientific challenge, in which developers could evaluate their methods on a standardized multi-center/-scanner image dataset, giving an objective comparison: the WMH Segmentation Challenge. Sixty T1 + FLAIR images from three MR scanners were released with the manual WMH segmentations for training. A test set of 110 images from five MR scanners was used for evaluation. The segmentation methods had to be containerized and submitted to the challenge organizers. Five evaluation metrics were used to rank the methods: 1) Dice similarity coefficient; 2) modified Hausdorff distance (95th percentile); 3) absolute log-transformed volume difference; 4) sensitivity for detecting individual lesions; and 5) F1-score for individual lesions. In addition, the methods were ranked on their inter-scanner robustness; 20 participants submitted their methods for evaluation. This paper provides a detailed analysis of the results. In brief, there is a cluster of four methods that rank significantly better than the other methods, with one clear winner. The inter-scanner robustness ranking shows that not all the methods generalize to unseen scanners. The challenge remains open for future submissions and provides a public platform for method evaluation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
(2014) · Cited by 33,419
A survey on deep learning in medical image analysis
Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A. W. M., van Ginneken, B., & Sánchez, C. I. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.07.005 · Cited by 15,332
Self-citationV-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation
Milletari, F., Navab, N., & Ahmadi, S.-A. (2016) · 2016 Fourth International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2016.79 · Cited by 11,931
Fully convolutional networks for semantic segmentation
Shelhamer, E., Long, J., & Darrell, T. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2572683 · Cited by 11,898
Voxel-Based Morphometry—The Methods
Ashburner, J., & Friston, K. J. (2000) · NeuroImage · doi:10.1006/nimg.2000.0582 · Cited by 8,676
Multi-scale context aggregation by dilated convolutions
Yu, F., & Koltun, V. (2015) · arXiv · doi:10.48550/arxiv.1511.07122 · Cited by 7,738 · cited 1×
“scan a densely connected convolutional network using dilated convolutions [50], [51].”
MethodRethinking Atrous Convolution for Semantic Image Segmentation
Chen, L.-C., Papandreou, G., Schroff, F., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1706.05587 · Cited by 7,517
The multimodal brain tumor image segmentation benchmark (BRATS)
Menze, B. H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., Burren, Y., Porz, N., Slotboom, J., Wiest, R., et al. (2015) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2014.2377694 · Cited by 6,781
Self-citationWardlaw, J. M., Smith, E. E., Biessels, G. J., Cordonnier, C., Fazekas, F., Frayne, R., Lindley, R. I., O'Brien, J. T., Barkhof, F., Benavente, O. R., et al. (2013) · The Lancet Neurology · doi:10.1016/s1474-4422(13)70124-8 · Cited by 5,544
Self-citationElastix: a toolbox for intensity-based medical image registration
Klein, S., Staring, M., Murphy, K., Viergever, M. A., & Pluim, J. (2010) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2009.2035616 · Cited by 4,311
Self-citationTHE GENERALIZATION OF ‘STUDENT'S’ PROBLEM WHEN SEVERAL DIFFERENT POPULATION VARLANCES ARE INVOLVED
WELCH, B. L. (1947) · Biometrika · doi:10.1093/biomet/34.1-2.28 · Cited by 4,135
Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation
Kamnitsas, K., Ledig, C., Newcombe, V. F. J., Simpson, J. P., Kane, A. D., Menon, D. K., Rueckert, D., & Glocker, B. (2017) · Medical Image Analysis · doi:10.1016/j.media.2016.10.004 · Cited by 3,498 · cited 2×
“The first eight layers were trained separately for the FLAIR and T1 images and had skip-connections [38] [39]. neuro.ml a neural network using the DeepMedic [40] architecture, having two parallel branches that process the images at two different scales.”
MethodPantoni, L. (2010) · The Lancet Neurology · doi:10.1016/s1474-4422(10)70104-6 · Cited by 3,405
Docker: lightweight Linux containers for consistent development and deployment
Merkel, D. (2014) · Linux journal · Cited by 3,191
Unbiased average age-appropriate atlases for pediatric studies
Fonov, V., Evans, A. C., Botteron, K., Almli, C. R., McKinstry, R. C., & Collins, D. L. (2011) · NeuroImage · doi:10.1016/j.neuroimage.2010.07.033 · Cited by 2,290 · cited 2×
“The MNI-152 standard brain template [30], showing different overlays.”
BackgroundMethodSelf-citationWarfield, S. K., Zou, K. H., & Wells, W. M. (2004) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2004.828354 · Cited by 2,072
The Generalization of `Student's' Problem when Several Different Population Variances are Involved
Welch, B. L. (1947) · Biometrika · doi:10.2307/2332510 · Cited by 1,924
Greenspan, H., van Ginneken, B., & Summers, R. M. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2016.2553401 · Cited by 1,700
White matter hyperintensities, cognitive impairment and dementia: an update
Prins, N. D., & Scheltens, P. (2015) · Nature Reviews Neurology · doi:10.1038/nrneurol.2015.10 · Cited by 1,284
Comparison and evaluation of methods for liver segmentation from CT datasets
Heimann, T., van Ginneken, B., Styner, M. A., Arzhaeva, Y., Aurich, V., Bauer, C., Beck, A., Becker, C., Beichel, R., Bekes, G., et al. (2009) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2009.2013851 · Cited by 1,135
Gland segmentation in colon histology images: The glas challenge contest
Sirinukunwattana, K., Pluim, J. P. W., Chen, H., Qi, X., Heng, P.-A., Guo, Y. B., Wang, L. Y., Matuszewski, B. J., Bruni, E., Sanchez, U., et al. (2017) · Medical Image Analysis · doi:10.1016/j.media.2016.08.008 · Cited by 979
Do CIFAR-10 Classifiers Generalize to CIFAR-10?Preprint
Recht, B., Roelofs, R., Schmidt, L., & Shankar, V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.00451 · Cited by 528
Evaluation of Registration Methods on Thoracic CT: The EMPIRE10 Challenge
Murphy, K., van Ginneken, B., Reinhardt, J. M., Kabus, S., Ding, K., Deng, X., Cao, K., Du, K., Christensen, G. E., Garcia, V., et al. (2011) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2011.2158349 · Cited by 440
Self-citationValverde, S., Cabezas, M., Roura, E., González-Villà, S., Pareto, D., Vilanova, J. C., Ramió-Torrentà, L., Rovira, À., Oliver, A., & Lladó, X. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2017.04.034 · Cited by 392
Self-citationLi, W., Wang, G., Fidon, L., Ourselin, S., Cardoso, M. J., & Vercauteren, T. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-59050-9_28 · Cited by 336
Self-citationDeep Retinal Image Understanding
Maninis, K.-K., Pont-Tuset, J., Arbeláez, P., & Van Gool, L. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46723-8_17 · Cited by 305
Commowick, O., Istace, A., Kain, M., Laurent, B., Leray, F., Simon, M., Pop, S. C., Girard, P., Améli, R., Ferré, J.-C., et al. (2018) · Scientific Reports · doi:10.1038/s41598-018-31911-7 · Cited by 277
Self-citationMRBrainS Challenge: Online Evaluation Framework for Brain Image Segmentation in 3T MRI Scans
Mendrik, A. M., Vincken, K. L., Kuijf, H. J., Breeuwer, M., Bouvy, W. H., de Bresser, J., Alansary, A., de Bruijne, M., Carass, A., El-Baz, A., et al. (2015) · Computational Intelligence and Neuroscience · doi:10.1155/2015/813696 · Cited by 261
Self-citationGhafoorian, M., Karssemeijer, N., Heskes, T., van Uden, I. W. M., Sanchez, C. I., Litjens, G., de Leeuw, F.-E., van Ginneken, B., Marchiori, E., & Platel, B. (2017) · Scientific Reports · doi:10.1038/s41598-017-05300-5 · Cited by 236
Self-citationGuerrero, R., Qin, C., Oktay, O., Bowles, C., Chen, L., Joules, R., Wolz, R., Valdés-Hernández, M. C., Dickie, D. A., Wardlaw, J., et al. (2018) · NeuroImage: Clinical · doi:10.1016/j.nicl.2017.12.022 · Cited by 223
Fully convolutional network ensembles for white matter hyperintensities segmentation in MR images
Li, H., Jiang, G., Zhang, J., Wang, R., Wang, Z., Zheng, W.-S., & Menze, B. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2018.07.005 · Cited by 209
Self-citationLesion location and cognitive impact of cerebral small vessel disease
Biesbroek, J. M., Weaver, N. A., & Biessels, G. J. (2017) · Clinical Science · doi:10.1042/cs20160452 · Cited by 189
Self-citation3D Segmentation in the Clinic: A Grand Challenge II: MS lesion segmentation
Styner, M., Lee, J., Chin, B., Chin, M. S., Commowick, O., Tran, H.-H., Markovic-Plese, S., Jewells, V., & Warfield, S. (2008) · The MIDAS Journal · doi:10.54294/lmkqvm · Cited by 167
Sudre, C. H., Cardoso, M. J., Bouvy, W. H., Biessels, G. J., Barnes, J., & Ourselin, S. (2015) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2015.2419072 · Cited by 162
Self-citationCaligiuri, M. E., Perrotta, P., Augimeri, A., Rocca, F., Quattrone, A., & Cherubini, A. (2015) · Neuroinformatics · doi:10.1007/s12021-015-9260-y · Cited by 161
Cortical microinfarcts on 3T MRI: Clinical correlates in memory‐clinic patients
van Veluw, S. J., Hilal, S., Kuijf, H. J., Ikram, M. K., Xin, X., Yeow, T. B., Venketasubramanian, N., Biessels, G. J., & Chen, C. (2015) · Alzheimer's & Dementia · doi:10.1016/j.jalz.2014.12.010 · Cited by 142
Self-citationMulti-dimensional Gated Recurrent Units for the Segmentation of Biomedical 3D-Data
Andermatt, S., Pezold, S., & Cattin, P. C. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46976-8_15 · Cited by 124
Self-citationDadar, M., Pascoal, T. A., Manitsirikul, S., Misquitta, K., Fonov, V., Tartaglia, M. C., Breitner, J. C. S., Rosa‐Neto, P., Carmichael, O. T., DeCarli, C. S., et al. (2017) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2017.2693978 · Cited by 111
Self-citationWolterink, J. M., Leiner, T., de Vos, B. D., Coatrieux, J.-L., Kelm, B. M., Kondo, S., Salgado, R. A., Shahzad, R., Shu, H., Snoeren, M. M., et al. (2016) · Medical Physics · doi:10.1118/1.4945696 · Cited by 108
Self-citationDadar, M., Maranzano, J., Misquitta, K., Anor, C. J., Fonov, V., Tartaglia, M. C., Carmichael, O. T., DeCarli, C. S., & Collins, D. L. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2017.06.009 · Cited by 108
Self-citationAutomated detection of white matter hyperintensities of all sizes in cerebral small vessel disease
Ghafoorian, M., Karssemeijer, N., van Uden, I. W. M., de Leeuw, F., Heskes, T., Marchiori, E., & Platel, B. (2016) · Medical Physics · doi:10.1118/1.4966029 · Cited by 81
Self-citationAutomated Segmentation of Multiple Sclerosis Lesions Using Multi-dimensional Gated Recurrent Units
Andermatt, S., Pezold, S., & Cattin, P. C. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-75238-9_3 · Cited by 71
Self-citationde Bresser, J., Kuijf, H. J., Zaanen, K., Viergever, M. A., Hendrikse, J., Biessels, G. J., Utrecht Vascular Cognitive Impairment Study Group, Algra, A., van den Berg, E., Bouvy, W. H., et al. (2018) · Scientific Reports · doi:10.1038/s41598-018-20084-y · Cited by 52
Self-citationBoomsma, J. M. F., Exalto, L. G., Barkhof, F., van den Berg, E., de Bresser, J., Heinen, R., Koek, H. L., Prins, N. D., Scheltens, P., Weinstein, H. C., et al. (2017) · JMIR Research Protocols · doi:10.2196/resprot.6864 · Cited by 37
Self-citationGlatard, T., Roura, E., Vukusic, S., Khademi, A., Mahbod, A., Dojat, M., Wang, C., Santos, W. P., Doyle, S., Malpica, N., et al. (2021) · Carolina Digital Repository (University of North Carolina at Chapel Hill) · doi:10.17615/1bq3-hv16 · Cited by 18
Self-citationXu, Y., Géraud, T., Puybareau, É., Isabelle Bloch, A., & Chazalon, J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-75238-9_42 · Cited by 14
Self-citationKnight, J., Taylor, G. W., & Khademi, A. (2018) · Magnetic Resonance Imaging · doi:10.1016/j.mri.2018.06.009 · Cited by 14
Self-citationWMH Segmentation Challenge: A Texture-Based Classification Approach
Bento, M., de Souza, R., Lotufo, R., Frayne, R., & Rittner, L. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-75238-9_41 · Cited by 5
Self-citation
Cited by
Neuroimaging standards for research into small vessel disease—advances since 2013
Duering, M., Biessels, G. J., Brodtmann, A., Chen, C., Cordonnier, C., de Leeuw, F.-E., Debette, S., Frayne, R., Jouvent, E., Rost, N. S., et al. (2023) · The Lancet Neurology · doi:10.1016/s1474-4422(23)00131-x · Cited by 1,144
Self-citationDomain adaptation for medical image analysis: A survey
Guan, H., & Liu, M. (2022) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2021.3117407 · Cited by 1,057
Confidence calibration and predictive uncertainty estimation for deep medical image segmentation
Mehrtash, A., Wells, W. M., Tempany, C. M., Abolmaesumi, P., & Kapur, T. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2020.3006437 · Cited by 428
Self-citationEelbode, T., Bertels, J., Berman, M., Vandermeulen, D., Maes, F., Bisschops, R., & Blaschko, M. B. (2020) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2020.3002417 · Cited by 420
Graph-Based Deep Learning for Medical Diagnosis and Analysis: Past, Present and Future
Ahmedt-Aristizabal, D., Armin, M. A., Denman, S., Fookes, C., & Petersson, L. (2021) · Sensors · doi:10.3390/s21144758 · Cited by 272
Metadata
- Authors
- Kuijf, H. J., Casamitjana, A., Collins, D. L., Dadar, M., Georgiou, A., Ghafoorian, M., Jin, D., Khademi, A., Knight, J., Li, H., Llado, X., Biesbroek, J. M., Luna, M., Mahmood, Q., McKinley, R., Mehrtash, A., Ourselin, S., Park, B.-Y., Park, H., Park, S. H., Pezold, S., Puybareau, E., De Bresser, J., Rittner, L., Sudre, C. H., Valverde, S., Vilaplana, V., Wiest, R., Xu, Y., Xu, Z., Zeng, G., Zhang, J., Zheng, G., Heinen, R., Chen, C., van der Flier, W., Barkhof, F., Viergever, M. A., Biessels, G. J., Andermatt, S., Bento, M., Berseth, M., Belyaev, M., & Cardoso, M. J.
- Venue
- IEEE Transactions on Medical Imaging
- Published
- 2019-11
- Volume
- 38
- Issue
- 11
- Pages
- 2556-2568
- ISSN
- 0278-0062 (ISSN-L)
- 1558-254X
- OpenAlex
- W2952735543
- Cited by
- 292
- References
- 52
- FWCI
- 14.64
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.