CoChat
Sign Up
PreprintCited by 126Medical Image Segmentation TechniquesCell Image Analysis TechniquesAdvanced Neural Network Applications

Common Limitations of Image Processing Metrics: A Picture Story

Open access
Reinke, Annika, Tizabi, Minu D., Sudre, Carole H., Eisenmann, Matthias, Rädsch, Tim, Baumgartner, Michael, Acion, Laura, Antonelli, Michela +85 more·2021·arXiv
DOI 10.48550/arxiv.2104.05642arXiv 2104.05642
Open this paper in CoChat — free accountOpen PDF

Abstract

While the importance of automatic image analysis is continuously increasing, recent meta-research revealed major flaws with respect to algorithm validation. Performance metrics are particularly key for meaningful, objective, and transparent performance assessment and validation of the used automatic algorithms, but relatively little attention has been given to the practical pitfalls when using specific metrics for a given image analysis task. These are typically related to (1) the disregard of inherent metric properties, such as the behaviour in the presence of class imbalance or small target structures, (2) the disregard of inherent data set properties, such as the non-independence of the test cases, and (3) the disregard of the actual biomedical domain interest that the metrics should reflect. This living dynamically document has the purpose to illustrate important limitations of performance metrics commonly applied in the field of image analysis. In this context, it focuses on biomedical image analysis problems that can be phrased as image-level classification, semantic segmentation, instance segmentation, or object detection task. The current version is based on a Delphi process on metrics conducted by an international consortium of image analysis experts from more than 60 institutions worldwide.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2104.05642

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Current acceptance testing of AI auto contouring algorithms relies on area- and distance-based metrics that measure overall contour agreement but say nothing about where the model is uncertain.

    “Acceptance testing of AI auto contouring algorithms typically involves both area‐based metrics (e.g., Dice Similarity Coefficient, Intersection over Union (IoU)) and distance‐based metrics (e.g., Hausdorff Distance) evaluated against reference standard contours. 11 , 12 , 13 , 14 These metrics are well established for assessing contour agreement, but they say nothing about where the model is confi”

    From Uncertainty quantification of U‐Net based segmentation tool using conformal prediction (2026) · cites this paper as [12]

    The Fundamentals (pp. 23–24) identifies Dice and IoU as overlap-based segmentation metrics and Hausdorff distance as a distance-based metric comparing predicted and reference boundaries. The segmentation discussion (p. 99) distinguishes predicted pixel probabilities from the binarized masks used to calculate metric values, supporting the distinction between contour agreement and model confidence. Our copy does not establish how AI autocontouring acceptance testing is typically conducted or explicitly discuss locating model uncertainty. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 81

  1. [1][1] Shadi AlZu'bi, Mohammed Shehab, Mahmoud Al-Ayyoub, Yaser Jararweh, and Brij Gupta. Parallel implementation for 3d medical volume fuzzy segmentation. Pattern Recognition Letters, 130:312–318, 2020.
  2. [2][2] Saeid Asgari Taghanaki, Kumar Abhishek, Joseph Paul Cohen, Julien Cohen-Adad, and Ghassan Hamarneh. Deep semantic segmentation of natural and medical images: a review. Artificial Intelligence Review, 54(1):137–178, 2021.
  3. [3][3] Min Bai and Raquel Urtasun. Deep watershed transform for instance segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5221–5229, 2017.
  4. [6][6] Miroslav Beneš and Barbara Zitová. Performance evaluation of image segmentation algorithms on microscopic image data. Journal of microscopy, 257(1):65–85, 2015.
  5. [7][7] Glenn W Brier et al. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1–3, 1950.
  6. [9][9] JB Brown. Classifiers and their metrics quantified. Molecular informatics, 37(1-2):1700127, 2018.
  7. [10][10] Neil G Burnet, Simon J Thomas, Kate E Burton, and Sarah J Jefferies. Defining the tumour and target volumes for radiotherapy. Cancer Imaging, 4(2):153, 2004.
  8. [11][11] Chang Cao, Davide Chicco, and Michael M Hoffman. The mcc-f1 curve: a performance evaluation technique for binary classification. arXiv preprint arXiv:2006.11278, 2020. · arXiv
  9. [12][12] Aaron Carass, Snehashis Roy, Adrian Gherman, Jacob C Reinhold, Andrew Jesson, Tal Arbel, Oskar Maier, Heinz Handels, Mohsen Ghafoorian, Bram Platel, et al. Evaluating white matter lesion segmentations with refined sørensen- dice analysis. Scientific reports, 10(1):1–19, 2020.
  10. [14][14] Bowen Cheng, Ross Girshick, Piotr Dollár, Alexander C Berg, and Alexander Kirillov. Boundary iou: Improving object-centric image segmentation evaluation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15334–15342, 2021.
  11. [15][15] Davide Chicco and Giuseppe Jurman. The advantages of the matthews correlation coefficient (mcc) over f1 score and accuracy in binary classification evaluation. BMC genomics, 21(1):1–13, 2020.
  12. [16][16] Davide Chicco, Niklas Tötsch, and Giuseppe Jurman. The matthews correlation coefficient (mcc) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation. BioData mining, 14(1):1–22, 2021.
  13. [17][17] Nancy R Cook. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation, 115(7): 928–935, 2007.
  14. [18][18] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Scharwächter, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset. In CVPR Workshop on The Future of Datasets in Vision, 2015.
  15. [19][19] Paulo Correia and Fernando Pereira. Video object relevance metrics for overall segmentation quality evaluation. EURASIP Journal on Advances in Signal Processing, 2006:1–11, 2006.
  16. [20][20] Jesse Davis and Mark Goadrich. The relationship between precision-recall and roc curves. In Proceedings of the 23rd international conference on Machine learning, pages 233–240, 2006.
  17. [21][21] Bert De Brabandere, Davy Neven, and Luc Van Gool. Semantic instance segmentation with a discriminative loss function. arXiv preprint arXiv:1708.02551, 2017. · arXiv
  18. [22][22] Jeffrey De Fauw, Joseph R Ledsam, Bernardino Romera-Paredes, Stanislav Nikolov, Nenad Tomasev, Sam Blackwell, Harry Askham, Xavier Glorot, Brendan O'Donoghue, Daniel Visentin, et al. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nature medicine, 24(9):1342–1350, 2018.
  19. [23][23] Elizabeth R DeLong, David M DeLong, and Daniel L Clarke-Pearson. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics, pages 837–845, 1988.
  20. [24][24] Lee R Dice. Measures of the amount of ecologic association between species. Ecology, 26(3):297–302, 1945.
  21. [25][25] Bruno Dujardin, Jef Van den Ende, Alfons Van Gompel, Jean-Pierre Unger, and Patrick Van der Stuyft. Likelihood ratios: a real improvement for clinical decision making? European journal of epidemiology, 10(1):29–36, 1994. 134 A. Reinke/M.D. Tizabi, et al.
  22. [26][26] Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 111(1):98–136, 2015.
  23. [27][27] Naiyu Gao, Yanhu Shan, Yupei Wang, Xin Zhao, Yinan Yu, Ming Yang, and Kaiqi Huang. Ssap: Single-shot instance segmentation with affinity pyramid. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 642–651, 2019.
  24. [28][28] Manuel Emilio Gegúndez-Arias, Arturo Aquino, José Manuel Bravo, and Diego Marín. A function for quality evaluation of retinal vessel segmentations. IEEE transactions on medical imaging, 31(2):231–239, 2011.
  25. [29][29] Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477):359–378, 2007.
  26. [32][32] Margherita Grandini, Enrico Bagli, and Giorgio Visani. Metrics for multi-class classification: an overview. arXiv preprint arXiv:2008.05756, 2020. · arXiv
  27. [33][33] Sebastian Gruber and Florian Buettner. Trustworthy deep learning via proper calibration errors: A unifying approach for quantifying the reliability of predictive uncertainty. arXiv preprint arXiv:2203.07835, 2022. · arXiv
  28. [34][34] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pages 1321–1330. PMLR, 2017.
  29. [35][35] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian QWeinberger. On Calibration of Modern Neural Networks. ICML, page 10, 2017.
  30. [36][36] Steven Hicks, Inga Strüke, Vajira Thambawita, Malek Hammou, Pål Halvorsen, Michael Riegler, and Sravanthi Parasa. On evaluation metrics for medical applications of artificial intelligence. medRxiv, 2021.
  31. [37][37] Peter Hirsch, Lisa Mais, and Dagmar Kainmueller. Patchperpix for instance segmentation. arXiv preprint arXiv:2001.07626, 2020. · arXiv
  32. [38][38] Katrin Honauer, Lena Maier-Hein, and Daniel Kondermann. The hci stereo metrics: Geometry-aware performance analysis of stereo algorithms. In Proceedings of the IEEE International Conference on Computer Vision, pages 2120–2128, 2015.
  33. [39][39] Mohammad Hossin and Md Nasir Sulaiman. A review on evaluation metrics for data classification evaluations. International journal of data mining & knowledge management process, 5(2):1, 2015.
  34. [40][40] Daniel P Huttenlocher, Gregory A. Klanderman, and William J Rucklidge. Comparing images using the hausdorff distance. IEEE Transactions on pattern analysis and machine intelligence, 15(9):850–863, 1993.
  35. [41][41] Paul Jaccard. The distribution of the flora in the alpine zone. 1. New phytologist, 11(2):37–50, 1912.
  36. [42][42] Paul F Jaeger, Simon AA Kohl, Sebastian Bickelhaupt, Fabian Isensee, Tristan Anselm Kuder, Heinz-Peter Schlemmer, and Klaus H Maier-Hein. Retina u-net: Embarrassingly simple exploitation of segmentation supervision for medical object detection. In Machine Learning for Health Workshop, pages 171–183. PMLR, 2020.
  37. [43][43] Paul Ferdinand Jäger. Challenges and Opportunities of End-to-End Learning in Medical Image Classification. PhD thesis, Karlsruher Institut für Technologie (KIT), 2020.
  38. [44][44] Leo Joskowicz, D Cohen, N Caplan, and J Sosna. Inter-observer variability of manual contour delineation of structures in ct. European radiology, 29(3):1391–1399, 2019.
  39. [46][46] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9404–9413, 2019.
  40. [47][47] A Kleppe. Area under the curve may hide poor generalisation to external datasets. ESMO open, 7(2), 2022.
  41. [48][48] Andreas Kleppe, Ole-Johan Skrede, Sepp De Raedt, Knut Liestøl, David J Kerr, and Håvard E Danielsen. Designing deep learning studies in cancer diagnostics. Nature Reviews Cancer, 21(3):199–211, 2021.
  42. [49][49] Florian Kofler, Ivan Ezhov, Fabian Isensee, Christoph Berger, Maximilian Korner, Johannes Paetzold, Hongwei Li, Suprosanna Shit, Richard McKinley, Spyridon Bakas, et al. Are we using appropriate segmentation metrics? Identi- fying correlates of human expert perception for CNN training beyond rolling the DICE coefficient. arXiv preprint arXiv:2103.06205v1, 2021. · arXiv
  43. [50][50] Ender Konukoglu, Ben Glocker, DongHye Ye, Antonio Criminisi, and KilianM Pohl. Discriminative segmentation-based evaluation through shape dissimilarity. IEEE transactions on medical imaging, 31(12):2278–2289, 2012. Common Limitations of Image Processing Metrics: A Picture Story 135
  44. [51][51] Jonathan Krause, Varun Gulshan, Ehsan Rahimy, Peter Karth, Kasumi Widner, Greg S Corrado, Lily Peng, and Dale R Webster. Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy. Ophthalmology, 125(8):1264–1272, 2018.
  45. [52][52] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83–97, 1955.
  46. [53][53] Victor Kulikov and Victor Lempitsky. Instance segmentation of biological images using harmonic embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3843–3851, 2020.
  47. [54][54] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014.
  48. [55][55] Hua Ma, Andriy I Bandos, Howard E Rockette, and David Gur. On use of partial area under the roc curve for evaluation of diagnostic performance. Statistics in medicine, 32(20):3449–3458, 2013.
  49. [56][56] Lena Maier-Hein, Matthias Eisenmann, Annika Reinke, Sinan Onogur, Marko Stankovic, Patrick Scholz, Tal Arbel, Hrvoje Bogunovic, Andrew P Bradley, Aaron Carass, et al. Why rankings of biomedical image analysis competitions should be interpreted with care. Nature communications, 9(1):1–13, 2018.
  50. [57][57] Lena Maier-Hein, Annika Reinke, Evangelia Christodoulou, Ben Glocker, Patrick Godau, Fabian Isensee, Jens Kleesiek, Michal Kozubek, Mauricio Reyes, Michael A Riegler, et al. Metrics reloaded: Pitfalls and recommendations for image analysis validation. arXiv preprint arXiv:2206.01653, 2022. · arXiv
  51. [58][58] Ran Margolin, Lihi Zelnik-Manor, and Ayellet Tal. How to evaluate foreground maps? In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 248–255, 2014.
  52. [59][59] Pavel Matula, Martin Maška, Dmitry V Sorokin, Petr Matula, Carlos Ortiz-de Solórzano, and Michal Kozubek. Cell tracking accuracy measurement based on comparison of acyclic oriented graphs. PloS one, 10(12):e0144959, 2015.
  53. [60][60] Bjoern HMenze, Andras Jakab, Stefan Bauer, Jayashree Kalpathy-Cramer, Keyvan Farahani, Justin Kirby, Yuliya Burren, Nicole Porz, Johannes Slotboom, Roland Wiest, et al. The multimodal brain tumor image segmentation benchmark (brats). IEEE transactions on medical imaging, 34(10):1993–2024, 2014.
  54. [61][61] Pawel Mlynarski, Hervé Delingette, Hamza Alghamdi, Pierre-Yves Bondiau, and Nicholas Ayache. Anatomically consistent cnn-based segmentation of organs-at-risk in cranial radiotherapy. Journal of Medical Imaging, 7(1):014502, 2020.
  55. [62][62] Annette M Molinaro. Diagnostic tests: how to estimate the positive predictive value. Neuro-Oncology Practice, 2(4): 162–166, 2015.
  56. [63][63] Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
  57. [65][65] Tanya Nair. Exploring uncertainty measures in deep networks for multiple sclerosis lesion detection and segmentation. PhD thesis, McGill University, 2018.
  58. [66][66] Nudrat Nida, Aun Irtaza, Ali Javed, Muhammad Haroon Yousaf, and Muhammad Tariq Mahmood. Melanoma lesion detection and segmentation using deep region based convolutional neural network and fuzzy c-means clustering. International journal of medical informatics, 124:37–48, 2019.
  59. [67][67] Stanislav Nikolov, Sam Blackwell, Alexei Zverovitch, Ruheena Mendes, Michelle Livne, Jeffrey De Fauw, Yojan Patel, Clemens Meyer, Harry Askham, Bernadino Romera-Paredes, et al. Clinically applicable segmentation of head and neck anatomy for radiotherapy: deep learning algorithm development and validation study. Journal of Medical Internet Research, 23(7):e26151, 2021.
  60. [68][68] Kemal Oksuz, Baris Can Cam, Emre Akbas, and Sinan Kalkan. Localization recall precision (lrp): A new performance metric for object detection. In Proceedings of the European Conference on Computer Vision (ECCV), pages 504–519, 2018.
  61. [69][69] Hung Viet Pham, Shangshu Qian, Jiannan Wang, Thibaud Lutellier, Jonathan Rosenthal, Lin Tan, Yaoliang Yu, and Nachiappan Nagappan. Problems and opportunities in training deep learning software systems: An analysis of variance. In Proceedings of the 35th IEEE/ACM international conference on automated software engineering, pages 771–783, 2020.
  62. [70][70] David MW Powers. Evaluation: from precision, recall and f-measure to roc, informedness, markedness and correlation. arXiv preprint arXiv:2010.16061, 2020. · arXiv
  63. [71][71] Annika Reinke, Matthias Eisenmann, Sinan Onogur, Marko Stankovic, Patrick Scholz, Peter M Full, Hrvoje Bogunovic, Bennett A Landman, Oskar Maier, Bjoern Menze, et al. How to exploit weaknesses in biomedical challenge design and organization. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 388–395. Springer, 2018.
  64. [72][72] Azriel Rosenfeld and John L Pfaltz. Sequential operations in digital picture processing. Journal of the ACM (JACM), 13 (4):471–494, 1966. 136 A. Reinke/M.D. Tizabi, et al.
  65. [74][74] Anindo Saha, Joeran Bosma, Jasper Linmans, Matin Hosseinzadeh, and Henkjan Huisman. Anatomical and diagnostic bayesian segmentation in prostate mri − should different clinical objectives mandate different loss functions? arXiv preprint arXiv:2110.12889, 2021. · arXiv
  66. [75][75] Suprosanna Shit, Johannes C Paetzold, Anjany Sekuboyina, Ivan Ezhov, Alexander Unger, Andrey Zhylka, Josien PW Pluim, Ulrich Bauer, and Bjoern H Menze. cldice-a novel topology-preserving loss function for tubular structure segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16560–16569, 2021.
  67. [77][77] Ana-Maria Šimundić. Measures of diagnostic accuracy: basic definitions. ejifcc, 19(4):203, 2009.
  68. [78][78] Ewout W Steyerberg, Andrew J Vickers, Nancy R Cook, Thomas Gerds, Mithat Gonen, Nancy Obuchowski, Michael J Pencina, and Michael W Kattan. Assessing the performance of prediction models: a framework for some traditional and novel measures. Epidemiology (Cambridge, Mass.), 21(1):128, 2010.
  69. [79][79] Carole H Sudre, Wenqi Li, Tom Vercauteren, Sebastien Ourselin, and M Jorge Cardoso. Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations. In Deep learning in medical image analysis and multimodal learning for clinical decision support, pages 240–248. Springer, 2017.
  70. [80][80] Yanmin Sun, Andrew K. C. Wong, and Mohamed S. Kamel. Classification of imbalanced data: a review. Int. J. Pattern Recognit. Artif. Intell., 23:687–719, 2009.
  71. [81][81] Abdel Aziz Taha and Allan Hanbury. Metrics for evaluating 3d medical image segmentation: analysis, selection, and tool. BMC medical imaging, 15(1):1–28, 2015.
  72. [82][82] Abdel Aziz Taha, Allan Hanbury, and Oscar A Jimenez del Toro. A formal method for selecting evaluation metrics for image segmentation. In 2014 IEEE international conference on image processing (ICIP), pages 932–936. IEEE, 2014.
  73. [83][83] Uchila N Umesh, Robert A Peterson, and Matthew H Sauber. Interjudge agreement and the maximum value of kappa. Educational and Psychological Measurement, 49(4):835–850, 1989.
  74. [85][85] Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas Schön. Evaluating model calibration in classification. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 3459–3467. PMLR, 2019.
  75. [86][86] Bram Van Ginneken, Tobias Heimann, and Martin Styner. 3d segmentation in the clinic: A grand challenge. InMICCAI Workshop on 3D Segmentation in the Clinic: A Grand Challenge, volume 1, pages 7–15, 2007.
  76. [87][87] Bram Van Ginneken, Samuel G Armato III, Bartjan de Hoop, Saskia van Amelsvoort-van de Vorst, Thomas Duindam, Meindert Niemeijer, Keelin Murphy, Arnold Schilham, Alessandra Retico, Maria Evelina Fantacci, et al. Comparing and combining algorithms for computer-aided detection of pulmonary nodules in computed tomography scans: the anode09 study. Medical image analysis, 14(6):707–722, 2010.
  77. [89][89] Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, et al. Scipy 1.0: fundamental algorithms for scientific computing in python. Nature methods, 17(3):261–272, 2020.
  78. [90][90] NJ Wald and JP Bestwick. Is the area under an roc curve a valid measure of the performance of a screening or diagnostic test? Journal of medical screening, 21(1):51–56, 2014.
  79. [91][91] Manuel Wiesenfarth, Annika Reinke, Bennett A Landman, Matthias Eisenmann, Laura Aguilera Saiz, M Jorge Cardoso, Lena Maier-Hein, and Annette Kopp-Schneider. Methods and open-source toolkit for analyzing and visualizing challenge results. Scientific Reports, 11(1):1–15, 2021.
  80. [92][92] Varduhi Yeghiazaryan and Irina D Voiculescu. Family of boundary overlap metrics for the evaluation of medical image segmentation. Journal of Medical Imaging, 5(1):015006, 2018.
  81. [93][93] Aston Zhang, Zachary C Lipton, Mu Li, and Alexander J Smola. Dive into deep learning. arXiv preprint arXiv:2106.11342, 2021. Common Limitations of Image Processing Metrics: A Picture Story 137 · arXiv

Cited by

Sorted by citations

Authors
Reinke, A., Tizabi, M. D., Sudre, C. H., Eisenmann, M., Rädsch, T., Baumgartner, M., Acion, L., Antonelli, M., Arbel, T., Bakas, S., Bankhead, P., Benis, A., Blaschko, M., Buettner, F., Cardoso, M. J., Chen, J., Cheplygina, V., Christodoulou, E., Cimini, B., Collins, G. S., Engelhardt, S., Farahani, K., Ferrer, L., Galdran, A., van Ginneken, B., Glocker, B., Godau, P., Haase, R., Hamprecht, F., Hashimoto, D. A., Heckmann-Nötzel, D., Hirsch, P., Hoffman, M. M., Huisman, M., Isensee, F., Jannin, P., Kahn, C. E., Kainmueller, D., Kainz, B., Karargyris, A., Karthikesalingam, A., Kavur, A. E., Kenngott, H., Kleesiek, J., Kleppe, A., Kohler, S., Kofler, F., Kopp-Schneider, A., Kooi, T., Kozubek, M., Kreshuk, A., Kurc, T., Landman, B. A., Litjens, G., Madani, A., Maier-Hein, K., Martel, A. L., Mattson, P., Meijering, E., Menze, B., Moher, D., Moons, K. G. M., Müller, H., Nichyporuk, B., Nickel, F., Noyan, M. A., Petersen, J., Polat, G., Rafelski, S. M., Rajpoot, N., Reyes, M., Rieke, N., Riegler, M., Rivaz, H., Saez-Rodriguez, J., Sánchez, C. I., Schroeter, J., Saha, A., Selver, M. A., Sharan, L., Shetty, S., van Smeden, M., Stieltjes, B., Summers, R. M., Taha, A. A., Tiulpin, A., Tsaftaris, S. A., Van Calster, B., Varoquaux, G., Wiesenfarth, M., Yaniv, Z. R., Jäger, P., & Maier-Hein, L.
Affiliations
  • University of Copenhagen (DK)
  • Vanderbilt University (US)
  • Radboud University Medical Center (NL)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
Volume
2021
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2104.05642
arXiv
2104.05642
OpenAlex
W3153770398
Cited by
126
References
1
Citations by year
2026: 7 · 2025: 16 · 2024: 34 · 2023: 37 · 2022: 23 · 2021: 9
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • Spiral (Imperial College London) (submittedVersion)
  • University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
  • Edinburgh Research Explorer (University of Edinburgh) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Institutes of Health
  • National Cancer Institute (U01CA242871)
  • National Institute of Neurological Disorders and Stroke (R01NS042645)
  • NIH Clinical Center
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Biophysics › Cell Image Analysis Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
algorithm validation, class imbalance, medical image analysis, image-level classification, semantic segmentation, instance segmentation, object detection, Delphi consensus

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review