Simultaneous Truth and Performance Level Estimation (STAPLE): An Algorithm for the Validation of Image Segmentation
Open accessAbstract
Characterizing the performance of image segmentation approaches has been a persistent challenge. Performance analysis is important since segmentation algorithms often have limited accuracy and precision. Interactive drawing of the desired segmentation by human raters has often been the only acceptable approach, and yet suffers from intra-rater and inter-rater variability. Automated algorithms have been sought in order to remove the variability introduced by raters, but such algorithms must be assessed to ensure they are suitable for the task. The performance of raters (human or algorithmic) generating segmentations of medical images has been difficult to quantify because of the difficulty of obtaining or estimating a known true segmentation for clinical data. Although physical and digital phantoms can be constructed for which ground truth is known or readily estimated, such phantoms do not fully reflect clinical images due to the difficulty of constructing phantoms which reproduce the full range of imaging characteristics and normal and pathological anatomical variability observed in clinical data. Comparison to a collection of segmentations by raters is an attractive alternative since it can be carried out directly on the relevant clinical imaging data. However, the most appropriate measure or set of measures with which to compare such segmentations has not been clarified and several measures are used in practice. We present here an expectation-maximization algorithm for simultaneous truth and performance level estimation (STAPLE). The algorithm considers a collection of segmentations and computes a probabilistic estimate of the true segmentation and a measure of the performance level represented by each segmentation. The source of each segmentation in the collection may be an appropriately trained human rater or raters, or may be an automated segmentation algorithm. The probabilistic estimate of the true segmentation is formed by estimating an optimal combination of the segmentations, weighting each segmentation depending upon the estimated performance level, and incorporating a prior model for the spatial distribution of structures being segmented as well as spatial homogeneity constraints. STAPLE is straightforward to apply to clinical imaging data, it readily enables assessment of the performance of an automated image segmentation algorithm, and enables direct comparison of human rater and algorithm performance.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Cites
Maximum Likelihood from Incomplete Data Via the <i>EM</i> Algorithm
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1977.tb01600.x · Cited by 50,444 · see citing works
Measures of the Amount of Ecologic Association Between Species
Dice, L. R. (1945) · Ecology · doi:10.2307/1932409 · Cited by 12,153 · see citing works
Zhang, Y., Brady, M., & Smith, S. M. (2001) · IEEE Transactions on Medical Imaging · doi:10.1109/42.906424 · Cited by 7,628 · see citing works
Spatial Interaction and the Statistical Analysis of Lattice Systems
Besag, J. (1974) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1974.tb00999.x · Cited by 6,067 · see citing works
The EM Algorithm and Extensions
Kushary, D., McLachlan, G. J., & Krishnan, T. (1998) · Technometrics · doi:10.2307/1271189 · Cited by 5,419 · see citing works
Adaptive Mixtures of Local Experts
Jacobs, R. A., Jordan, M. I., Nowlan, S. J., & Hinton, G. E. (1991) · Neural Computation · doi:10.1162/neco.1991.3.1.79 · Cited by 5,384 · see citing works
Kittler, J., Hatef, M., Duin, R. P. W., & Matas, J. (1998) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.667881 · Cited by 5,312 · see citing works
THE DISTRIBUTION OF THE FLORA IN THE ALPINE ZONE. <sup>1</sup>
Jaccard, P. (1912) · New Phytologist · doi:10.1111/j.1469-8137.1912.tb05611.x · Cited by 4,561 · see citing works
On the Statistical Analysis of Dirty Pictures
Besag, J. E. (1986) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1986.tb01412.x · Cited by 4,204 · see citing works
Beitrag zur Theorie des Ferromagnetismus
Ising, E. (1925) · Zeitschrift für Physik · doi:10.1007/bf02980577 · Cited by 3,871 · see citing works
Spatial registration and normalization of images
Friston, K. J., Ashburner, J., Frith, C., Poline, J.-B., Heather, J. D., & Frackowiak, R. S. J. (1995) · Human Brain Mapping · doi:10.1002/hbm.460030303 · Cited by 3,744 · see citing works
Theoretical Improvements in Algorithmic Efficiency for Network Flow Problems
Edmonds, J., & Karp, R. M. (1972) · Journal of the ACM · doi:10.1145/321694.321699 · Cited by 2,531 · see citing works
Bilmes, J. A. (1998) · Cited by 2,463 · see citing works
Design and construction of a realistic digital brain phantom
Collins, D. L., Zijdenbos, A. P., Kollokian, V., Sled, J. G., Kabani, N. J., Holmes, C. J., & Evans, A. C. (1998) · IEEE Transactions on Medical Imaging · doi:10.1109/42.712135 · Cited by 1,825 · see citing works
The EM Algorithm and Extensions, 2E
McLachlan, G. J., & Krishnan, T. (2008) · John Wiley & Sons, Inc. eBooks · doi:10.1002/9780470191613 · Cited by 1,536 · see citing works
Decision combination in multiple classifier systems
Ho, T. K., Hull, J. J., & Srihari, S. N. (1994) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.273716 · Cited by 1,515 · see citing works
Applying the right statistics: analyses of measurement studies
Bland, J. M., & Altman, D. G. (2003) · Ultrasound in Obstetrics and Gynecology · doi:10.1002/uog.122 · Cited by 1,305 · see citing works
Morphometric analysis of white matter lesions in MR images: method and validation
Zijdenbos, A. P., Dawant, B. M., Margolin, R. A., & Palmer, A. C. (1994) · IEEE Transactions on Medical Imaging · doi:10.1109/42.363096 · Cited by 1,268 · see citing works
Content-boosted collaborative filtering for improved recommendations
Melville, P., Mooney, R. J., & Nagarajan, R. (2002) · doi:10.5555/777092.777124 · Cited by 1,089 · see citing works
Exact Maximum A Posteriori Estimation for Binary Images
Greig, D. M., Porteous, B. T., & Seheult, A. H. (1989) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1989.tb01764.x · Cited by 1,066 · see citing works
Stochastic relaxation, Gibbs distributions and the Bayesian restoration of images*
Geman, S., & Geman, D. (1993) · Journal of Applied Statistics · doi:10.1080/02664769300000058 · Cited by 984 · see citing works
The Visible Human Male: A Technical Report
Spitzer, V. M., Ackerman, M. J., Scherzinger, A. L., et al. (1996) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.1996.96236280 · Cited by 706 · see citing works
Quantitative magnetic resonance imaging of brain development in premature and mature newborns
Hüppi, P. S., Warfield, S. K., Kikinis, R., Barnes, P. D., Zientara, G. P., Jolesz, F. A., Tsuji, M. K., & Volpe, J. J. (1998) · Annals of Neurology · doi:10.1002/ana.410430213 · Cited by 690 · see citing works
Self-citationAutomated model-based bias field correction of MR images of the brain
Leemput, K. V., Maes, F., Vandermeulen, D., & Suetens, P. (1999) · IEEE Transactions on Medical Imaging · doi:10.1109/42.811268 · Cited by 581 · see citing works
Parametric estimate of intensity inhomogeneities applied to MRI
Styner, M. A., Brechbühler, C., Szckely, G., & Gerig, G. (2000) · IEEE Transactions on Medical Imaging · doi:10.1109/42.845174 · Cited by 523 · see citing works
Exact optimization for markov random fields with convex priors
Ishikawa, H. (2003) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2003.1233908 · Cited by 487 · see citing works
Meta-analytic methods for diagnostic test accuracy
Irwig, L., Macaskill, P., Glasziou, P., & Fahey, M. T. (1995) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(94)00099-c · Cited by 467 · see citing works
Parallel and deterministic algorithms from MRFs: surface reconstruction
Geiger, D., & Girosi, F. (1991) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.134040 · Cited by 452 · see citing works
Automated Segmentation of MR Images of Brain Tumors
Kaus, M. R., Warfield, S. K., Nabavi, A., Black, P. M., Jolesz, F. A., & Kikinis, R. (2001) · Radiology · doi:10.1148/radiology.218.2.r01fe44586 · Cited by 424 · see citing works
Self-citationAdaptive, template moderated, spatially varying statistical classification
Warfield, S. K., Kaus, M. R., Jolesz, F. A., & Kikinis, R. (2000) · Medical Image Analysis · doi:10.1016/s1361-8415(00)00003-7 · Cited by 402 · see citing works
Self-citationMurphy, B. P., Inder, T. E., Hüppi, P. S., Warfield, S. K., Zientara, G. P., Kikinis, R., Jolesz, F. A., & Volpe, J. J. (2001) · PEDIATRICS · doi:10.1542/peds.107.2.217 · Cited by 367 · see citing works
Self-citationValmet: A New Validation Tool for Assessing and Improving 3D Object Segmentation
Gerig, G., Jomier, M., & Chakos, M. (2001) · Lecture notes in computer science · doi:10.1007/3-540-45468-3_62 · Cited by 345 · see citing works
Routine quantitative analysis of brain and cerebrospinal fluid spaces with MR imaging
Kikinis, R., Shenton, M. E., Gerig, G., Martin, J., Anderson, M., Metcalf, D., Guttmann, C. R. G., McCarley, R. W., Jolesz, F. A., Lorensen, W. E., et al. (1992) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.1880020603 · Cited by 237 · see citing works
Bharatha, A., Hirose, M., Hata, N., Warfield, S. K., Ferrant, M., Zou, K. H., Suárez-Santana, E., Ruiz‐Alzola, J., D'Amico, A. V., Cormack, R. A., et al. (2001) · Medical Physics · doi:10.1118/1.1414009 · Cited by 233 · see citing works
Self-citationHierarchical Mixtures of Experts and the EM Algorithm
Jordan, M. I., & Jacobs, R. A. (1994) · Springer eBooks · doi:10.1007/978-1-4471-2097-1_113 · Cited by 183 · see citing works
Warfield, S. K., Dengler, J., Zaers, J., Guttmann, C. R. G., Wells, W. M., Ettinger, G. J., Hiller, J. B., & Kikinis, R. (1995) · Journal of Image Guided Surgery · doi:10.1002/(sici)1522-712x(1995)1:6 · Cited by 172 · see citing works
Self-citationHierarchical mixtures of experts and the EM algorithm
Jordan, M. I., & Jacobs, R. A. (2005) · IEEE International Conference on Neural Networks/IEEE ... International Conference on Neural Networks · doi:10.1109/ijcnn.1993.716791 · Cited by 162 · see citing works
An Automated Registration Algorithm for Measuring MRI Subcortical Brain Structures
Iosifescu, D. V., Shenton, M. E., Warfield, S. K., Kikinis, R., Dengler, J., Jolesz, F. A., & McCarley, R. W. (1997) · NeuroImage · doi:10.1006/nimg.1997.0274 · Cited by 144 · see citing works
Self-citationInteractive, GPU-Based Level Sets for 3D Segmentation
Lefohn, A., Cates, J. E., & Whitaker, R. (2003) · Lecture notes in computer science · doi:10.1007/978-3-540-39899-8_70 · Cited by 137 · see citing works
Validation of Image Segmentation and Expert Quality with an Expectation-Maximization Algorithm
Warfield, S. K., Zou, K. H., & Wells, W. M. (2002) · Lecture notes in computer science · doi:10.1007/3-540-45786-0_37 · Cited by 117 · see citing works
Self-citationHogan, R. E., Mark, K. E., Wang, L., Joshi, S. C., Miller, M. I., & Bucholz, R. D. (2000) · Radiology · doi:10.1148/radiology.216.1.r00jl41291 · Cited by 107 · see citing works
Surface of the brain: three-dimensional MR images created with volume rendering.
Levin, D. N., Hu, X. P., Tan, K. K., et al. (1989) · Radiology · doi:10.1148/radiology.171.1.2928539 · Cited by 103 · see citing works
Three validation metrics for automated probabilistic image segmentation of brain tumours
Zou, K. H., Wells, W. M., Kikinis, R., et al. (2004) · Statistics in Medicine · doi:10.1002/sim.1723 · Cited by 94 · see citing works
Self-citationAutomated Segmentation of MRI of Brain Tumors
Kaus, M. R., Warfield, S. K., Nabavi, A., et al. (2001) · Cited by 87 · see citing works
Self-citationReliability Parameters to Improve Combination Strategies in Multi-Expert Systems
Cordella, L. P., Foggia, P., Sansone, C., et al. (1999) · Pattern Analysis and Applications · doi:10.1007/s100440050029 · Cited by 86 · see citing works
Enhanced spatial priors for segmentation of magnetic resonance imagery
Kapur, T., Grimson, W. E. L., Grimson, L., Kikinis, R., & Wells, W. M. (1998) · Lecture notes in computer science · doi:10.1007/bfb0056231 · Cited by 74 · see citing works
Self-citationThe application of mean field theory to image motion estimation
Zhang, J., & Hanauer, G. G. (1995) · IEEE Transactions on Image Processing · doi:10.1109/83.350816 · Cited by 68 · see citing works
3D model-based segmentation of individual brain structures from magnetic resonance imaging data
Collins, D. L. (1994) · doi:10.82308/26214 · Cited by 68 · see citing works
Expectation Maximization Strategies for Multi-atlas Multi-label Segmentation
Torsten Rohlfing, B., Russakoff, D. B., & Maurer, C. R. (2003) · Lecture notes in computer science · doi:10.1007/978-3-540-45087-0_18 · Cited by 65 · see citing works
Effect of exercise on 23 Na MRI and relaxation characteristics of the human calf muscle
Bansal, N., Szczepaniak, L. S., Ternullo, D., Fleckenstein, J. L., & Malloy, C. R. (2000) · Journal of Magnetic Resonance Imaging · doi:10.1002/(sici)1522-2586(200005)11:5 · Cited by 64 · see citing works
Evaluating Image Segmentation Algorithms Using the Pareto Front
Everingham, M., Muller, H. L., & Thomas, B. (2002) · Lecture notes in computer science · doi:10.1007/3-540-47979-1_3 · Cited by 61 · see citing works
A Novel Nonrigid Registration Algorithm and Applications
Rexilius, J., Warfield, S. K., Guttmann, C. R. G., et al. (2001) · Lecture notes in computer science · doi:10.1007/3-540-45468-3_110 · Cited by 43 · see citing works
Self-citationModel-based three-dimensional medical image segmentation
Kapur, T., Grimson, W. E. L., & Wells, W. M. (1999) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 41 · see citing works
Self-citationAdaptive template moderated spatially varying statistical classification
Warfield, S. K., Kaus, M. R., Jolesz, F. A., et al. (1998) · Lecture notes in computer science · doi:10.1007/bfb0056228 · Cited by 33 · see citing works
Self-citationThe problem of ROC analysis without truth: the EM algorithm and the information matrix
Beiden, S. V., Campbell, G., Meier, K. L., et al. (2000) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.383099 · Cited by 32 · see citing works
Tu, X. M., Kowalski, J., & Jia, G. (1999) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19991130)18:22 · Cited by 28 · see citing works
Measuring global and local spatial correspondence using information theory
Bello, F., & Colchester, A. C. F. (1998) · Lecture notes in computer science · doi:10.1007/bfb0056285 · Cited by 27 · see citing works
Windridge, D., & Kittler, J. (2003) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2003.1182097 · Cited by 26 · see citing works
Toward a Common Validation Methodology for Segmentation and Registration Algorithms
Yoo, T. S., Ackerman, M. J., & Vannier, M. W. (2000) · Lecture notes in computer science · doi:10.1007/978-3-540-40899-4_43 · Cited by 24 · see citing works
Extraction and Application of Expert Priors to Combine Multiple Segmentations of Human Brain Tissue
Rohlfing, T., Russakoff, D. B., & Maurer, C. R. (2003) · Lecture notes in computer science · doi:10.1007/978-3-540-39903-2_71 · Cited by 16 · see citing works
SIMULTANEOUS VALIDATION OF IMAGE SEGMENTATION AND ASSESSMENT OF EXPERT QUALITY.
Warfield, S. K., Zou, K. H., Kaus, M. R., et al. (2002) · Cited by 16 · see citing works
Self-citationWarfield, S. K., Zou, K. H., Kaus, M. R., et al. (2003) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi.2002.1029201 · Cited by 11 · see citing works
Self-citationPhysics-Based Nonrigid Registration for Medical Image Analysis
Rexilius, J., Heinz Handels, A., & Warfield, S. K. (2001) · Cited by 11 · see citing works
Self-citationEvaluating Image Segmentation Algorithms Using the Pareto Front
Everingham, Muller, H. L., & Thomas, B. (2002) · Bristol Research (University of Bristol) · Cited by 5 · see citing works
Fennema‐Notestine, C., Özyurt, İ. B., Brown, G. G., et al. (2003) · Research at the University of Copenhagen (University of Copenhagen) · Cited by 4 · see citing works
Sperfeld (1999) · Pure Amsterdam UMC · doi:10.1002/1531-8249(199911)46:5 · Cited by 2 · see citing works
Meta-analysis: formulating, evaluating, combining, and reporting
Normand, S.-L. T. (1999) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19990215)18:3
Flagged first, then by citations
Research works printed but not linked: 3
- Nicoll, 2001, Basic Principles of Diagnostic Test Use and Interpretation, 1
- 10.2307/1932409.JSTOR1932409 · DOI
- Elfadel, 1993, From fields to networks
Cited by
Sorted by citations
The multimodal brain tumor image segmentation benchmark (BRATS)
Menze, B. H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., Burren, Y., Porz, N., Slotboom, J., Wiest, R., et al. (2015) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2014.2377694 · Cited by 6,781 · see citing works
Bernard, O., Lalande, A., Zotti, C., Cervenansky, F., Yang, X., Heng, P.-A., Cetin, I., Lekadir, K., Camara, O., Gonzalez Ballester, M. A., et al. (2018) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2018.2837502 · Cited by 2,431 · see citing works
Radiomics: the process and the challenges
Kumar, V., Gu, Y., Basu, S., Berglund, A., Eschrich, S. A., Schabath, M. B., Forster, K., Aerts, H. J. W. L., Dekker, A., Fenstermacher, D., et al. (2012) · Magnetic Resonance Imaging · doi:10.1016/j.mri.2012.06.010 · Cited by 2,258 · see citing works
“Many metrics have been used, like volume, center of volume and maximum surface distance, to compare characteristics like robustness and accuracy [57,58].”
MethodUltrasound image segmentation: a survey
Noble, J. A., & Boukerroui, D. (2006) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2006.877092 · Cited by 1,157 · see citing works
Comparison and evaluation of methods for liver segmentation from CT datasets
Heimann, T., van Ginneken, B., Styner, M. A., Arzhaeva, Y., Aurich, V., Bauer, C., Beck, A., Becker, C., Beichel, R., Bekes, G., et al. (2009) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2009.2013851 · Cited by 1,135 · see citing works
Deep learning for brain MRI segmentation: state of the art and future directions
Akkus, Z., Galimzianova, A., Hoogi, A., Rubin, D. L., & Erickson, B. J. (2017) · Journal of Digital Imaging · doi:10.1007/s10278-017-9983-4 · Cited by 1,129 · see citing works
A survey of MRI-based medical image analysis for brain tumor studies
Bauer, S., Wiest, R., Nolte, L.-P., & Reyes, M. (2013) · Physics in Medicine and Biology · doi:10.1088/0031-9155/58/13/r97 · Cited by 1,036 · see citing works
Preparing Medical Imaging Data for Machine Learning
Willemink, M. J., Koszek, W. A., Hardell, C., Wu, J., Fleischmann, D., Harvey, H., Folio, L. R., Summers, R. M., Rubin, D. L., & Lungren, M. P. (2020) · Radiology · doi:10.1148/radiol.2020192224 · Cited by 1,020 · see citing works
Evaluation of prostate segmentation algorithms for mri: The promise12 challenge
Litjens, G., Toth, R., van de Ven, W., Hoeks, C., Kerkstra, S., van Ginneken, B., Vincent, G., Guillard, G., Birbeck, N., Zhang, J., et al. (2014) · Medical Image Analysis · doi:10.1016/j.media.2013.12.002 · Cited by 857 · see citing works
Multi-Atlas Segmentation with Joint Label Fusion
Wang, H., Suh, J. W., Das, S. R., Pluta, J. B., Craige, C., & Yushkevich, P. A. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.143 · Cited by 796 · see citing works
Generalized Overlap Measures for Evaluation and Validation in Medical Image Analysis
Crum, W. R., Camara, O., & Hill, D. L. G. (2006) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2006.880587 · Cited by 776 · see citing works
Multi-atlas segmentation of biomedical images: A survey
Iglesias, J. E., & Sabuncu, M. R. (2015) · Medical Image Analysis · doi:10.1016/j.media.2015.06.012 · Cited by 667 · see citing works
Deep MRI brain extraction: A 3D convolutional neural network for skull stripping
Kleesiek, J., Urban, G., Hubert, A., Schwarz, D., Maier-Hein, K., Bendszus, M., & Biller, A. (2016) · NeuroImage · doi:10.1016/j.neuroimage.2016.01.024 · Cited by 505 · see citing works
Federated learning enables big data for rare cancer boundary detection
Pati, S., Baid, U., Edwards, B., Sheller, M., Wang, S.-H., Reina, G. A., Foley, P., Gruzdev, A., Karkada, D., Davatzikos, C., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-33407-5 · Cited by 394 · see citing works
Doshi, J., Erus, G., Ou, Y., Resnick, S. M., Gur, R. C., Gur, R. E., Satterthwaite, T. D., Furth, S., & Davatzikos, C. (2016) · NeuroImage · doi:10.1016/j.neuroimage.2015.11.073 · Cited by 309 · see citing works
“For each subject of the pooled multi-site dataset ROI segmentation was performed by independently applying three different methods, MUSE, STAPLE (Warfield et al., 2004) and Joint Label Fusion (Wang et al.”
Tumor-Cut: Segmentation of Brain Tumors on Contrast Enhanced MR Images for Radiosurgery Applications
Hamamci, A., Kucuk, N., Karaman, K., Engin, K., & Unal, G. (2012) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2011.2181857 · Cited by 286 · see citing works
Young, J., Modat, M., Cardoso, M. J., Mendelson, A., Cash, D., & Ourselin, S. (2013) · NeuroImage Clinical · doi:10.1016/j.nicl.2013.05.004 · Cited by 249 · see citing works
Ryan, N. S., Keihaninejad, S., Shakespeare, T. J., Lehmann, M., Crutch, S. J., Malone, I. B., Thornton, J. S., Mancini, L., Hyare, H., Yousry, T., et al. (2013) · Brain · doi:10.1093/brain/awt065 · Cited by 205 · see citing works
Wang, H., Das, S. R., Suh, J. W., Altinay, M., Pluta, J., Craige, C., Avants, B., & Yushkevich, P. A. (2011) · NeuroImage · doi:10.1016/j.neuroimage.2011.01.006 · Cited by 177 · see citing works
Tan, L. K., Liew, Y. M., Lim, E., & McLaughlin, R. A. (2017) · Medical Image Analysis · doi:10.1016/j.media.2017.04.002 · Cited by 165 · see citing works
Reverse Classification Accuracy: Predicting Segmentation Performance in the Absence of Ground Truth
Valindria, V. V., Lavdas, I., Bai, W., Kamnitsas, K., Aboagye, E. O., Rockall, A. G., Rueckert, D., & Glocker, B. (2017) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2017.2665165 · Cited by 152 · see citing works
Drukker, K., Chen, W., Gichoya, J. W., Gruszauskas, N. P., Kalpathy-Cramer, J., Koyejo, S., Myers, K. J., Sá, R. C., Sahiner, B., Whitney, H. M., et al. (2023) · Journal of Medical Imaging · doi:10.1117/1.jmi.10.6.061104 · Cited by 149 · see citing works
Mulder, E. R., de Jong, R. A., Knol, D. L., van Schijndel, R. A., Cover, K. S., Visser, P. J., Barkhof, F., & Vrenken, H. (2014) · NeuroImage · doi:10.1016/j.neuroimage.2014.01.058 · Cited by 122 · see citing works
Kainz, W., Neufeld, E., Bolch, W. E., Graff, C. G., Kim, C. H., Kuster, N., Lloyd, B., Morrison, T., Segars, P., Yeom, Y. S., et al. (2019) · IEEE Transactions on Radiation and Plasma Medical Sciences · doi:10.1109/trpms.2018.2883437 · Cited by 109 · see citing works
Pluta, J., Yushkevich, P., Das, S., & Wolk, D. (2012) · Journal of Alzheimer's Disease · doi:10.3233/jad-2012-111931 · Cited by 107 · see citing works
DALSA: domain adaptation for supervised learning from sparsely annotated MR images
Goetz, M., Weber, C., Binczyk, F., Polanska, J., Tarnawski, R., Bobek-Billewicz, B., Koethe, U., Kleesiek, J., Stieltjes, B., & Maier-Hein, K. H. (2016) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2015.2463078 · Cited by 98 · see citing works
Hoyte, L., Jakab, M., Warfield, S. K., Shott, S., Flesh, G., & Fielding, J. R. (2004) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2004.06.067 · Cited by 92 · see citing works
Self-citationCheplygina, V., de Bruijne, M., & Pluim, J. P. W. (2019) · Medical Image Analysis · doi:10.1016/j.media.2019.03.009 · Cited by 93 · see citing works
Lin, D., Lapen, K., Sherer, M. V., Kantor, J., Zhang, Z., Boyce, L. M., Bosch, W., Korenstein, D., & Gillespie, E. F. (2020) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2020.04.011 · Cited by 59 · see citing works
Ts66Yah, a mouse model of down syndrome with improved construct and face validity
Duchon, A., del Mar Muñiz Moreno, M., Chevalier, C., Nalesso, V., Andre, P., Fructuoso-Castellar, M., Mondino, M., Po, C., Noblet, V., Birling, M.-C., et al. (2022) · Disease Models & Mechanisms · doi:10.1242/dmm.049721 · Cited by 58 · see citing works
Pati, S., Baid, U., Edwards, B., Sheller, M. J., Foley, P., Anthony Reina, G., Thakur, S., Sako, C., Bilello, M., Davatzikos, C., et al. (2022) · Physics in Medicine & Biology · doi:10.1088/1361-6560/ac9449 · Cited by 51 · see citing works
Ma, D., Holmes, H. E., Cardoso, M. J., Modat, M., Harrison, I. F., Powell, N. M., O’Callaghan, J. M., Ismail, O., Johnson, R. A., O’Neill, M. J., et al. (2019) · Frontiers in Neuroscience · doi:10.3389/fnins.2019.00011 · Cited by 36 · see citing works
Comparison of algorithms for ultrasound image segmentation without ground truth
Sikka, K., & Deserno, T. M. (2010) · SPIE Proceedings · doi:10.1117/12.844504 · Cited by 4 · see citing works
Lost in the Folds: When Cross-Validation Is Not a Deep Ensemble for Uncertainty Estimation
Kirscher, T., Bujotzek, M., Kirchhoff, Y., Rokuss, M., Isensee, F., Kahl, K.-C., Kovacs, B., & Maier-Hein, K. (2026) · arXiv · doi:10.48550/arxiv.2605.18329
- Authors
- Warfield, S. K., Zou, K. H., & Wells, W. M.
- Affiliations
- Brigham and Women's Hospital (US)
- Venue
- IEEE Transactions on Medical Imaging
- Published
- 2004-07
- Volume
- 23
- Issue
- 7
- Pages
- 903-921
- ISSN
- 0278-0062 (ISSN-L)
- PMC
- PMC1283110
- OpenAlex
- W2148347694
- Semantic Scholar
- d860504ff0a3faaeeccf72e9a6c982b644ed16a2
- Cited by
- 2,072
- References
- 67
- Citations by year
- 2026: 85 · 2025: 119 · 2024: 112 · 2023: 117 · 2022: 128 · 2021: 115 · 2020: 94 · 2019: 123 · 2018: 110 · 2017: 106 · 2016: 128 · 2015: 118 · 2014: 120 · 2013: 115 · 2012: 112
- FWCI
- 33.58
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- IEEE Transactions on Medical Imaging (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (submittedVersion)
- Funders
- National Institutes of Health (1r21ca089449-01, 5p41rr013218-14, 5r01ns035142-07)
- National Institute on Aging (R01 AG 19513, R01AG019513)
- National Institute of Mental Health (R21MH067054, R21MH67054)
- National Cancer Institute (P01CA067165, R01 CA86879, P01 CA-67165, R21 CA89449, R33 CA99015, R01CA086879)
- National Institute of Neurological Disorders and Stroke (R01 NS-35142, R01-NS035142)
- U.S. National Library of Medicine (R01 LM007861-01A1, R01 LM007861, R01 LM007861-02)
- National Center for Research Resources (P41RR-013218, P41 RR13218)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Engineering › Biomedical Engineering › Medical Imaging and Analysis
- Keywords
- staples, probabilistic segmentation, expectation-maximization algorithm, interobserver variability, medical image segmentation
- MeSH terms
- Algorithms, Brain, Decision Making, Computer-Assisted, Humans, Image Enhancement, Image Interpretation, Computer-Assisted, Infant, Newborn, Magnetic Resonance Imaging, Markov Chains, Sensitivity and Specificity, Reproducibility of Results, Models, Statistical, Observer Variation, Phantoms, Imaging
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
