The DOI matches its Crossref record; 53 of the 56 references it cites were found in a registry or an index.
Evaluation of prostate segmentation algorithms for MRI: The PROMISE12 challenge
Open accessAbstract
Prostate MRI image segmentation has been an area of intense research due to the increased use of MRI as a modality for the clinical workup of prostate cancer. Segmentation is useful for various tasks, e.g. to accurately localize prostate boundaries for radiotherapy or to initialize multi-modal registration algorithms. In the past, it has been difficult for research groups to evaluate prostate segmentation algorithms on multi-center, multi-vendor and multi-protocol data. Especially because we are dealing with MR images, image appearance, resolution and the presence of artifacts are affected by differences in scanners and/or protocols, which in turn can have a large influence on algorithm accuracy. The Prostate MR Image Segmentation (PROMISE12) challenge was setup to allow a fair and meaningful comparison of segmentation methods on the basis of performance and robustness. In this work we will discuss the initial results of the online PROMISE12 challenge, and the results obtained in the live challenge workshop hosted by the MICCAI2012 conference. In the challenge, 100 prostate MR cases from 4 different centers were included, with differences in scanner manufacturer, field strength and protocol. A total of 11 teams from academic research groups and industry participated. Algorithms showed a wide variety in methods and implementation, including active appearance models, atlas registration and level sets. Evaluation was performed using boundary and volume based metrics which were combined into a single score relating the metrics to human expert performance. The winners of the challenge where the algorithms by teams Imorphics and ScrAutoProstate, with scores of 85.72 and 84.29 overall. Both algorithms where significantly better than all other algorithms in the challenge (p<0.05) and had an efficient implementation with a run time of 8min and 3s per case respectively. Overall, active appearance model based approaches seemed to outperform other approaches like multi-atlas registration, both on accuracy and computation time. Although average algorithm performance was good to excellent and the Imorphics algorithm outperformed the second observer on average, we showed that algorithm combination might lead to further improvement, indicating that optimal performance for prostate segmentation is not yet obtained. All results are available online at http://promise12.grand-challenge.org/.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“PROMISE12 as the example in the prostate dataset is the dataset made for MICCAI prostate segmentation challenge, which was used by 11 academic teams for various segmentation algorithms and showed promising results”
From A review of deep learning-based deformable medical image registration (2022) · cites this paper as [55]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Robust Real-Time Face Detection
Viola, P., & Jones, M. J. (2004) · International Journal of Computer Vision · doi:10.1023/b:visi.0000013087.49260.fb · Cited by 14,097
Marching cubes: A high resolution 3D surface construction algorithm
Lorensen, W. E., & Cline, H. E. (1987) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/37401.37422 · Cited by 10,320
Marching cubes: A high resolution 3D surface construction algorithm
Lorensen, W. E., & Cline, H. E. (1987) · ACM SIGGRAPH Computer Graphics · doi:10.1145/37402.37422 · Cited by 8,928
Cootes, T. F., Edwards, G. J., & Taylor, C. J. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.927467 · Cited by 5,409
ESUR prostate MR guidelines 2012
Barentsz, J. O., Richenberg, J., Clements, R., Choyke, P. L., Verma, S., Villeirs, G. M., Rouvière, O., Løgager, V., & Fütterer, J. J. (2012) · European Radiology · doi:10.1007/s00330-011-2377-y · Cited by 2,444
Robust real-time object detection.
Viola, P. A. (2001) · Cited by 2,340
Warfield, S. K., Zou, K. H., & Wells, W. M. (2004) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2004.828354 · Cited by 2,072
Pérez, P., Gangnet, M., & Blake, A. (2003) · ACM SIGGRAPH Conference (SIGGRAPH) · doi:10.1145/1201775.882269 · Cited by 2,041
Pérez, P., Gangnet, M., & Blake, A. (2003) · ACM Transactions on Graphics · doi:10.1145/882262.882269 · Cited by 1,769
Comparison and evaluation of methods for liver segmentation from CT datasets
Heimann, T., van Ginneken, B., Styner, M. A., Arzhaeva, Y., Aurich, V., Bauer, C., Beck, A., Becker, C., Beichel, R., Bekes, G., et al. (2009) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2009.2013851 · Cited by 1,135
New variants of a method of MRI scale standardization
Nyúl, L. G., Udupa, J. K., & Zhang, X. (2000) · IEEE Transactions on Medical Imaging · doi:10.1109/42.836373 · Cited by 958
Review of MR image segmentation techniques using pattern recognition
Bezdek, J. C., Hall, L., & Clarke, L. P. (1993) · Medical Physics · doi:10.1118/1.597000 · Cited by 956
Dickinson, L., Ahmed, H. U., ALLEN, C. M., Barentsz, J. O., Carey, B. M., Fütterer, J. J., Heijmink, S. W. T. P. J., Hoskin, P., Kirkham, A. P. S., Padhani, A. R., et al. (2010) · European Urology · doi:10.1016/j.eururo.2010.12.009 · Cited by 737
Patch-based segmentation using expert priors: Application to hippocampus and ventricle segmentation
Coupé, P., Manjón, J. V., Fonov, V., Pruessner, J. C., Robles, M., & Collins, D. L. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2010.09.018 · Cited by 704
Automated model-based bias field correction of MR images of the brain
Leemput, K. V., Maes, F., Vandermeulen, D., & Suetens, P. (1999) · IEEE Transactions on Medical Imaging · doi:10.1109/42.811268 · Cited by 581
On standardizing the MR image intensity scale
Ny�l, L. G., & Udupa, J. K. (1999) · Magnetic Resonance in Medicine · doi:10.1002/(sici)1522-2594(199912)42:6 · Cited by 575
Niemeijer, M., van Ginneken, B., Cree, M. J., Mizutani, A., Quellec, G., Sánchez, C. I., Zhang, B., Hornero, R., Lamard, M., Muramatsu, C., et al. (2009) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2009.2033909 · Cited by 519
Evaluation of Registration Methods on Thoracic CT: The EMPIRE10 Challenge
Murphy, K., van Ginneken, B., Reinhardt, J. M., Kabus, S., Ding, K., Deng, X. H., Cao, K., Du, K., Christensen, G. E., Garcia, V., et al. (2011) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2011.2158349 · Cited by 441
Klein, S., van der Heide, U. A., Lips, I. M., van Vulpen, M., Staring, M., & Pluim, J. P. W. (2008) · Medical Physics · doi:10.1118/1.2842076 · Cited by 422
DRAMMS: Deformable registration via attribute matching and mutual-saliency weighting
Ou, Y., Sotiras, A., Nikos Paragios, A., & Davatzikos, C. A. (2010) · Medical Image Analysis · doi:10.1016/j.media.2010.07.002 · Cited by 399
Self-citationSchaap, M., Metz, C., van Walsum, T., van der Giessen, A. G., Weustink, A. C., Mollet, N. R., Bauer, C., Bogunović, H., Castro‐González, C., Deng, X. B., et al. (2009) · Medical Image Analysis · doi:10.1016/j.media.2009.06.003 · Cited by 373
Tanimoto, A., Nakashima, J., Kohno, H., Shinmoto, H., & Kuribayashi, S. (2006) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.20793 · Cited by 334
Hambrock, T., Hoeks, C. M. A., de Kaa, C. H., Scheenen, T. W. J., Fütterer, J. J., Bouwense, S. A. W., van Oort, I. M., Schröder, F. H., Huisman, H. J., & Barentsz, J. O. (2011) · European Urology · doi:10.1016/j.eururo.2011.08.042 · Cited by 305
Self-citationLangerak, T., van der Heide, U. A., Kotte, A. N. T. J., Viergever, M. A., van Vulpen, M., & Pluim, J. P. W. (2010) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2010.2057442 · Cited by 267
View-based active appearance models
Cootes, T. F., Walker, K., & Taylor, C. (2002) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.2000.840639 · Cited by 250
Transition Zone Prostate Cancer: Detection and Localization with 3-T Multiparametric MR Imaging
Hoeks, C. M. A., Hambrock, T., Yakar, D., de Kaa, C. A. H., Feuth, T. B., Witjes, J. A., Fütterer, J. J., & Barentsz, J. O. (2012) · Radiology · doi:10.1148/radiol.12120281 · Cited by 232
Self-citationKitajima, K., Kaji, Y., Fukabori, Y., Yoshida, K., Suganuma, N., & Sugimura, K. (2010) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.22075 · Cited by 205
MR to ultrasound registration for image-guided prostate interventions
Hu, Y., Ahmed, H. U., Taylor, Z. A., ALLEN, C. M., Emberton, M., Hawkes, D. J., & Barratt, D. C. (2010) · Medical Image Analysis · doi:10.1016/j.media.2010.11.003 · Cited by 178
Self-citationOnline resource for validation of brain segmentation methods
Shattuck, D. W., Prasad, G., Mirza, M., Narr, K. L., & Toga, A. W. (2008) · NeuroImage · doi:10.1016/j.neuroimage.2008.10.066 · Cited by 167
MRI: Stability of three supervised segmentation techniques
Clarke, L. P., Velthuizen, R. P., Phuphanich, S., Schellenberg, J., Arrington, J. A., & Silbiger, M. L. (1993) · Magnetic Resonance Imaging · doi:10.1016/0730-725x(93)90417-c · Cited by 166
Li, C., Xu, C., Anderson, A. W., & Gore, J. C. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-02498-6_24 · Cited by 138
Pasquier, D., Lacornerie, T., Vermandel, M., Rousseau, J., Lartigau, E. F., & Betrouni, N. (2007) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2007.02.005 · Cited by 133
Multifeature Landmark-Free Active Appearance Models: Application to Prostate MRI Segmentation
Tóth, R., & Madabhushi, A. (2012) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2012.2201498 · Cited by 124
Self-citationVos, P., Barentsz, J. O., Karssemeijer, N., & Huisman, H. J. (2012) · Physics in Medicine and Biology · doi:10.1088/0031-9155/57/6/1527 · Cited by 122
Self-citationVilleirs, G. M., Meerleer, G. O. D., Visschere, P. J. D., Fonteyne, V., Verbaeys, A. C., & Oosterlinck, W. (2009) · European Journal of Radiology · doi:10.1016/j.ejrad.2009.08.007 · Cited by 116
Multi-kernel graph embedding for detection, Gleason grading of prostate cancer via MRI/MRS
Tiwari, P., Kurhanewicz, J. V., & Madabhushi, A. (2012) · Medical Image Analysis · doi:10.1016/j.media.2012.10.004 · Cited by 96
Self-citationAutomatic Segmentation of Bladder and Prostate Using Coupled 3D Deformable Models
Costa, M. J., Delingette, H., Novellas, S., & Ayache, N. (2007) · Springer eBooks · doi:10.1007/978-3-540-75757-3_31 · Cited by 88
Patient Specific Prostate Segmentation in 3-D Magnetic Resonance Images
Chandra, S. S., Dowling, J., Shen, K., Raniga, P., Pluim, J. P. W., Greer, P. B., Salvado, O., & Fripp, J. (2012) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2012.2211377 · Cited by 84
Self-citationA Pattern Recognition Approach to Zonal Segmentation of the Prostate on MRI
Litjens, G. J. S., Debats, O. A., van de Ven, W. J. M., Karssemeijer, N., & Huisman, H. J. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33418-4_51 · Cited by 77
Self-citationMulti-organ Abdominal CT Segmentation Using Hierarchically Weighted Subject-Specific Atlases
Wolz, R., Chu, C., Misawa, K., Mori, K., & Rueckert, D. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33415-3_2 · Cited by 76
Groupwise Construction of Appearance Models using Piece-wise Affine Deformations
Cootes, T. F., Twining, C., Petrovic, V. S., Schestowitz, R. S., & Taylor, C. (2005) · British Machine Vision Conference (BMVC) · doi:10.5244/c.19.88 · Cited by 73
Makni, N., Puech, P., Lopes, R., Dewalle, A. S., Colot, O., & Betrouni, N. (2008) · International Journal of Computer Assisted Radiology and Surgery · doi:10.1007/s11548-008-0281-y · Cited by 67
Transmit field inhomogeneity and T 1 estimation errors in breast DCE‐MRI at 3 tesla
Sung, K., Daniel, B. L., & Hargreaves, B. A. (2013) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.23996 · Cited by 65
Li, X., Huang, W., & Rooney, W. D. (2012) · Magnetic Resonance Imaging · doi:10.1016/j.mri.2012.05.005 · Cited by 58
Prostate segmentation by sparse representation based classification
Gao, Y., Liao, S., & Shen, D. G. (2012) · Medical Physics · doi:10.1118/1.4754304 · Cited by 55
Accurate Prostate Volume Estimation Using Multifeature Active Shape Models on T2-weighted MRI
Tóth, R., Bloch, B., Genega, E. M., Rofsky, N. M., Lenkinski, R. E., Rosen, M. A., Kalyanpur, A., Pungavkar, S., & Madabhushi, A. (2011) · Academic Radiology · doi:10.1016/j.acra.2011.01.016 · Cited by 51
Self-citationTóth, R., Tiwari, P., Rosen, M. A., Reed, G. D., Kurhanewicz, J. V., Kalyanpur, A., Pungavkar, S., & Madabhushi, A. (2010) · Medical Image Analysis · doi:10.1016/j.media.2010.09.002 · Cited by 46
Self-citationOu, Y., Doshi, J. J., Erus, G., & Davatzikos, C. A. (2012) · Cited by 35
Self-citationProstate MR image segmentation using 3D active appearance models
Maan, B., & van der Heijden, F. (2012) · University of Twente Research Information · Cited by 30
Self-citationSmart Paint : A New Interactive Segmentation Method\\ Applied to MR Prostate Segmentation
Malmberg, F., Strand, R., Kullberg, J., Nordenskjöld, R., & Bengtsson, E. W. (2012) · Medical Image Computing and Computer-Assisted Intervention · Cited by 28
Self-citationMatching Breast Masses Depicted on Different Views
Zheng, B., Tan, J., Ganott, M. A., Chough, D. M., & Gur, D. (2009) · Academic Radiology · doi:10.1016/j.acra.2009.05.005 · Cited by 24
Fütterer, J. J., & Barentsz, J. O. (2009) · Applied Radiology · doi:10.37549/ar1674 · Cited by 13
MRI based knee cartilage assessment
Kroon, D.-J., Kowalski, P., Tekieli, W., Reeuwijk, E., Saris, D. B. F., & Slump, C. H. (2012) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.906918 · Cited by 6
Research works printed but not linked: 8
- Birkbeck, N., Zhang, J., Requardt, M., Kiefer, B., Gall, P., Kevin Zhou, S., 2012. Region-specific hierarchical segmentation of MR prostate using discriminative learning. MICCAI Grand Challenge: Prostate MR Image Segmentation 2012.
- Gao, Q., Rueckert, D., Edwards, P., 2012. An automatic multi-atlas based prostate segmentation using local appearance-specific atlases and patch-based voxel weighting. MICCAI Grand Challenge: Prostate MR Image Segmentation 2012.
- Ghose, S., Mitra, J., Oliver, A., Martí, R., Lladó, X., Freixenet, J., Vilanova, J.C., Sidibé, D., Meriaudeau, F., 2012. A random forest based classification approach to prostate segmentation in MRI. MICCAI Grand Challenge: Prostate MR Image Segmentation 2012.
- Kirschner, M., Jung, F., Wesarg, S., 2012. Automatic prostate segmentation in MR images with a probabilistic active shape model. MICCAI Grand Challenge: Prostate MR Image Segmentation 2012.
- Litjens, G.J.S., Karssemeijer, N., Huisman, H.J., 2012. A multi-atlas approach for prostate segmentation in MR images. MICCAI Grand Challenge: Prostate MR Image Segmentation 2012.
- Toth, 2012, Multifeature landmark-free active appearance models: application to prostate MRI segmentation, IEEE Trans. Med. Imaging, 31, 1638, 10.1109/TMI.2012.2201498 · DOI
- Vincent, G., Guillard, G., Bowes, M., 2012. Fully automatic segmentation of the prostate using active appearance models. MICCAI Grand Challenge: Prostate MR Image Segmentation 2012.
- Yuan, J., Qiu, W., Ukwatta, E., Rajchl, M., Sun, Y., Fenster, A., 2012. An efficient convex optimization approach to 3D prostate MRI segmentation with generic star shape prior. MICCAI Grand Challenge: Prostate MR Image Segmentation 2012.
Cited by
Sorted by citations
V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation
Milletari, F., Navab, N., & Ahmadi, S.-A. (2016) · 2016 Fourth International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2016.79 · Cited by 11,931
nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation
Isensee, F., Jaeger, P. F., Kohl, S. A. A., Petersen, J., & Maier-Hein, K. H. (2021) · Nature Methods · doi:10.1038/s41592-020-01008-z · Cited by 10,477
Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool
Taha, A. A., & Hanbury, A. (2015) · BMC Medical Imaging · doi:10.1186/s12880-015-0068-x · Cited by 3,247
Medical image segmentation using deep learning: A survey
Wang, R., Lei, T., Cui, R., Zhang, B., Meng, H., & Nandi, A. K. (2022) · IET Image Processing · doi:10.1049/ipr2.12419 · Cited by 1,012
Yeung, M., Sala, E., Schönlieb, C., & Rundo, L. (2021) · Computerized Medical Imaging and Graphics · doi:10.1016/j.compmedimag.2021.102026 · Cited by 795
Metadata
- Authors
- Litjens, G., Toth, R., van de Ven, W., Hoeks, C., Kerkstra, S., van Ginneken, B., Vincent, G., Guillard, G., Birbeck, N., Zhang, J., Strand, R., Malmberg, F., Ou, Y., Davatzikos, C., Kirschner, M., Jung, F., Yuan, J., Qiu, W., Gao, Q., Edwards, P. “., Maan, B., van der Heijden, F., Ghose, S., Mitra, J., Dowling, J., Barratt, D., Huisman, H., & Madabhushi, A.
- Affiliations
- Radboud University Medical Center (NL)
- Siemens (United States) (US)
- Uppsala University (SE)
- University of Pennsylvania (US)
- Technische Universität Darmstadt (DE)
- Robarts Research Institute (CA)
- Imperial College London (GB)
- University of Twente (NL)
- Commonwealth Scientific and Industrial Research Organisation (AU)
- University College London (GB)
- Case Western Reserve University (US)
- Corresponding author
- Litjens, Geert
- ORCID
- Venue
- Medical Image Analysis
- Publisher
- Elsevier BV
- Published
- 2014-02
- Volume
- 18
- Issue
- 2
- Pages
- 359-373
- ISSN
- 1361-8415 (ISSN-L)
- PMC
- PMC4137968
- OpenAlex
- W2106033751
- Cited by
- 857
- References
- 56
- Citations by year
- 2026: 107 · 2025: 127 · 2024: 95 · 2023: 106 · 2022: 92 · 2021: 87 · 2020: 71 · 2019: 60 · 2018: 35 · 2017: 25 · 2016: 26 · 2015: 16 · 2014: 10
- FWCI
- 19.49
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Medical Image Analysis (publishedVersion)
- PubMed (publishedVersion)
- University of Twente Research Information (publishedVersion)
- PubMed Central (acceptedVersion)
- UCL Discovery (University College London) (submittedVersion)
- Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) (submittedVersion)
- Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion)
- Radboud Repository (Radboud University) (submittedVersion)
- Radboud Repository (Radboud University) (submittedVersion, other-oa)
- University of Twente Research Information (submittedVersion)
- Funders
- Dutch Cancer Society (KUN2007-3971)
- National Cancer Institute of the National Institutes of Health
- National Institute of Biomedical Imaging and Bioengineering of the National Institutes of Health
- National Science Foundation
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Brain Tumor Detection and Classification
- Physical Sciences › Engineering › Biomedical Engineering › Medical Imaging and Analysis
- Keywords
- prostate segmentation, prostate MRI, active appearance models, multi-atlas registration, level sets, segmentation algorithm comparison, prostate cancer, algorithm combination
- MeSH terms
- Algorithms, Humans, Image Enhancement, Image Interpretation, Computer-Assisted, Magnetic Resonance Imaging, Male, Prostatic Neoplasms, Reference Standards, Sensitivity and Specificity, Reproducibility of Results, Artifacts, Imaging, Three-Dimensional
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
