CoChat
Sign Up
Cited by 241Cell Image Analysis TechniquesMedical Image Segmentation TechniquesArtificial Intelligence in Healthcare and Education

The DOI matches its Crossref record; 36 of the 37 references it cites were found in a registry or an index, and 1 are not checked yet.

Understanding metric-related pitfalls in image analysis validation

Open access
Reinke, Annika, Tizabi, Minu D., Baumgartner, Michael, Eisenmann, Matthias, Heckmann-Nötzel, Doreen, Kavur, A. Emre, Rädsch, Tim, Sudre, Carole H., Acion, Laura, Antonelli, Michela, Arbel, Tal, Bakas, Spyridon, Benis, Arriel, Buettner, Florian, Cardoso, M. Jorge, Cheplygina, Veronika, Chen, Jianxu, Christodoulou, Evangelia, Cimini, Beth A., Farahani, Keyvan, Ferrer, Luciana, Galdran, Adrian, van Ginneken, Bram, Glocker, Ben, Godau, Patrick, Hashimoto, Daniel A., Hoffman, Michael M., Huisman, Merel, Isensee, Fabian, Jannin, Pierre, Kahn, Charles E., Kainmueller, Dagmar, Kainz, Bernhard, Karargyris, Alexandros, Kleesiek, Jens, Kofler, Florian, Kooi, Thijs, Kopp-Schneider, Annette, Kozubek, Michal, Kreshuk, Anna, Kurc, Tahsin, Landman, Bennett A., Litjens, Geert, Madani, Amin, Maier-Hein, Klaus, Martel, Anne L., Meijering, Erik, Menze, Bjoern, Moons, Karel G. M., Müller, Henning, Nichyporuk, Brennan, Nickel, Felix, Petersen, Jens, Rafelski, Susanne M., Rajpoot, Nasir, Reyes, Mauricio, Riegler, Michael A., Rieke, Nicola, Saez-Rodriguez, Julio, Sánchez, Clara I., Shetty, Shravya, Summers, Ronald M., Taha, Abdel A., Tiulpin, Aleksei, Tsaftaris, Sotirios A., Van Calster, Ben, Varoquaux, Gaël, Yaniv, Ziv R., Jäger, Paul F., Maier-Hein, Lena·2024·Nature Methods
DOI 10.1038/s41592-023-02150-0PMC PMC11181963
Open this paper in CoChat — free accountOpen PDF

Abstract

Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41592-023-02150-0

Cites

36 of 37 references verified0 citation errors · 0 not checkable

Statistics

Figures

Extended Data Fig. 1 [P2.2] Disregard of the properties of the target structures. (a) Small structure sizes. The predictions of two algorithms (Prediction 1/2) differ in only a single pixel. In the case of the small structure (bottom row), this has a substantial effect on the corresponding Dice Similarity Coefficient (DSC) metric value (similar for the Intersection over Union (IoU)). This pitfall is also relevant for other overlap-based metrics such as the centerline Dice Similarity Coefficient (clDice), and localization criteria such as Box/Approx/Mask IoU and Intersection over Reference (IoR). (b) Complex structure shapes. Common overlap-based metrics (here: DSC) are unaware of complex structure shapes and treat Predictions 1 and 2 equally. The clDice uncovers the fact that Prediction 1 misses the fine-granular branches of the reference and favors Prediction 2, which focuses on the center line of the object. This pitfall is also relevant for other overlap-based such as metrics IoU and pixel-level Fβ Score as well as localization criteria such as Box/Approx/Mask IoU, Center Distance, Mask IoU > 0, Point inside Mask/Box/Approx, and IoR.
Extended Data Fig. 1 [P2.2] Disregard of the properties of the target structures. (a) Small structure sizes. The predictions of two algorithms (Prediction 1/2) differ in only a single pixel. In the case of the small structure (bottom row), this has a substantial effect on the corresponding Dice Similarity Coefficient (DSC) metric value (similar for the Intersection over Union (IoU)). This pitfall is also relevant for other overlap-based metrics such as the centerline Dice Similarity Coefficient (clDice), and localization criteria such as Box/Approx/Mask IoU and Intersection over Reference (IoR). (b) Complex structure shapes. Common overlap-based metrics (here: DSC) are unaware of complex structure shapes and treat Predictions 1 and 2 equally. The clDice uncovers the fact that Prediction 1 misses the fine-granular branches of the reference and favors Prediction 2, which focuses on the center line of the object. This pitfall is also relevant for other overlap-based such as metrics IoU and pixel-level Fβ Score as well as localization criteria such as Box/Approx/Mask IoU, Center Distance, Mask IoU > 0, Point inside Mask/Box/Approx, and IoR.CC BY · PubMed Central
Extended Data Fig. 2 [P2.4] Disregard of the properties of the algorithm output. (a) Possibility of overlapping predictions. If multiple structures of the same type can be seen within the same image (here: reference objects R1 and R2), it is generally advisable to phrase the problem as instance segmentation (InS; right) rather than semantic segmentation (SemS; left). This way, issues with boundary-based metrics resulting from comparing a given structure boundary to the boundary of the wrong instance in the reference can be avoided. In the provided example, the distance of the red boundary pixel to the reference, as measured by a boundary-based metric in SemS problems, would be zero, because different instances of the same structure cannot be distinguished. This problem is overcome by phrasing the problem as InS. In this case, (only) the boundary of the matched instance (here: R2) is considered for distance computation. (b) Possibility of empty prediction or reference. Each column represents a potential scenario for per-image validation of objects, categorized by whether True Positives (TPs), False Negatives (FNs), and False Positives (FPs) are present (n > 0) or not (n = 0) after matching/assignment. The sketches on the top showcase each scenario when setting “n > 0” to “n = 1”. For each scenario, Sensitivity, Positive Predictive Value (PPV), and the F1 Score are calculated. Some scenarios yield undefined values (Not a Number (NaN)).
Extended Data Fig. 2 [P2.4] Disregard of the properties of the algorithm output. (a) Possibility of overlapping predictions. If multiple structures of the same type can be seen within the same image (here: reference objects R1 and R2), it is generally advisable to phrase the problem as instance segmentation (InS; right) rather than semantic segmentation (SemS; left). This way, issues with boundary-based metrics resulting from comparing a given structure boundary to the boundary of the wrong instance in the reference can be avoided. In the provided example, the distance of the red boundary pixel to the reference, as measured by a boundary-based metric in SemS problems, would be zero, because different instances of the same structure cannot be distinguished. This problem is overcome by phrasing the problem as InS. In this case, (only) the boundary of the matched instance (here: R2) is considered for distance computation. (b) Possibility of empty prediction or reference. Each column represents a potential scenario for per-image validation of objects, categorized by whether True Positives (TPs), False Negatives (FNs), and False Positives (FPs) are present (n > 0) or not (n = 0) after matching/assignment. The sketches on the top showcase each scenario when setting “n > 0” to “n = 1”. For each scenario, Sensitivity, Positive Predictive Value (PPV), and the F1 Score are calculated. Some scenarios yield undefined values (Not a Number (NaN)).CC BY · PubMed Central
Fig. 1 Examples of metric-related pitfalls in image analysis validation. (A) Medical image analysis example: Voxel-based metrics are not appropriate for detection problems. Measuring the voxel-level performance of a prediction yields a near-perfect Sensitivity. However, the Sensitivity at the instance level reveals that lesions are actually missed by the algorithm. (B) Biological image analysis example: The task of predicting fibrillarin in the dense fibrillary component of the nucleolus should be phrased as a segmentation task, for which segmentation metrics reveal the low quality of the prediction. Phrasing the task as image reconstruction instead and validating it using metrics such as the Person Correlation Coefficient yields misleadingly high metric scores [12, 68, 74, 88, 88].
Fig. 1 Examples of metric-related pitfalls in image analysis validation. (A) Medical image analysis example: Voxel-based metrics are not appropriate for detection problems. Measuring the voxel-level performance of a prediction yields a near-perfect Sensitivity. However, the Sensitivity at the instance level reveals that lesions are actually missed by the algorithm. (B) Biological image analysis example: The task of predicting fibrillarin in the dense fibrillary component of the nucleolus should be phrased as a segmentation task, for which segmentation metrics reveal the low quality of the prediction. Phrasing the task as image reconstruction instead and validating it using metrics such as the Person Correlation Coefficient yields misleadingly high metric scores [12, 68, 74, 88, 88].CC BY · PubMed Central
Fig. 2 Overview of the taxonomy for metric-related pitfalls. Pitfalls can be grouped into three main categories: [P1] Pitfalls related to the inadequate choice of the problem category, [P2] pitfalls related to poor metric selection, and [P3] pitfalls related to poor metric application. [P2] and [P3] are further split into subcategories. For all categories, pitfall sources are presented (green), with references to corresponding illustrations of representative examples. Note that the order in which the pitfall sources are presented does not correlate with importance.
Fig. 2 Overview of the taxonomy for metric-related pitfalls. Pitfalls can be grouped into three main categories: [P1] Pitfalls related to the inadequate choice of the problem category, [P2] pitfalls related to poor metric selection, and [P3] pitfalls related to poor metric application. [P2] and [P3] are further split into subcategories. For all categories, pitfall sources are presented (green), with references to corresponding illustrations of representative examples. Note that the order in which the pitfall sources are presented does not correlate with importance.CC BY · PubMed Central
Fig. 3 [P1] Pitfalls related to the inadequate choice of the problem category. Wrong choice of problem category. Effect of using segmentation metrics for object detection problems. The pixel-level Dice Similarity Coefficient (DSC) of a prediction recognizing every structure (Prediction 2) is lower than that of a prediction that only recognizes one of the three structures (Prediction 1).
Fig. 3 [P1] Pitfalls related to the inadequate choice of the problem category. Wrong choice of problem category. Effect of using segmentation metrics for object detection problems. The pixel-level Dice Similarity Coefficient (DSC) of a prediction recognizing every structure (Prediction 2) is lower than that of a prediction that only recognizes one of the three structures (Prediction 1).CC BY · PubMed Central
Fig. 4 [P2.1] Disregard of the domain interest. (a) Importance of structure boundaries. The predictions of two algorithms (Prediction 1/2) capture the boundary of the given structure substantially differently, but lead to the exact same Dice Similarity Coefficient (DSC), due to its boundary unawareness. This pitfall is also relevant for other overlap-based metrics such as centerline Dice Similarity Coefficient (clDice), pixel-level Fβ Score, and Intersection over Union (IoU), as well as localization criteria such as Box/Approx/Mask IoU, Center Distance, Mask IoU > 0, Point inside Mask/Box/Approx, and Intersection over Reference (IoR). (b) Unequal severity of class confusions. When predicting the severity of a disease for three patients in an ordinal classification problem, Prediction 1 assumes a much lower severity for Patient 3 than actually observed. This critical issue is overlooked by common metrics (here: Accuracy), which measure no difference to Prediction 2, which assesses the severity much better. Metrics with pre-defined weights (here: Expected Cost (EC)) correctly penalize Prediction 1 much more than Prediction 2. This pitfall is also relevant for other counting metrics, such as Balanced Accuracy (BA), Fβ Score, Positive Likelihood Ratio (LR+), Matthews Correlation Coefficient (MCC), Net Benefit (NB), Negative Predictive Value (NPV), Positive Predictive Value (PPV), Sensitivity, and Specificity.
Fig. 4 [P2.1] Disregard of the domain interest. (a) Importance of structure boundaries. The predictions of two algorithms (Prediction 1/2) capture the boundary of the given structure substantially differently, but lead to the exact same Dice Similarity Coefficient (DSC), due to its boundary unawareness. This pitfall is also relevant for other overlap-based metrics such as centerline Dice Similarity Coefficient (clDice), pixel-level Fβ Score, and Intersection over Union (IoU), as well as localization criteria such as Box/Approx/Mask IoU, Center Distance, Mask IoU > 0, Point inside Mask/Box/Approx, and Intersection over Reference (IoR). (b) Unequal severity of class confusions. When predicting the severity of a disease for three patients in an ordinal classification problem, Prediction 1 assumes a much lower severity for Patient 3 than actually observed. This critical issue is overlooked by common metrics (here: Accuracy), which measure no difference to Prediction 2, which assesses the severity much better. Metrics with pre-defined weights (here: Expected Cost (EC)) correctly penalize Prediction 1 much more than Prediction 2. This pitfall is also relevant for other counting metrics, such as Balanced Accuracy (BA), Fβ Score, Positive Likelihood Ratio (LR+), Matthews Correlation Coefficient (MCC), Net Benefit (NB), Negative Predictive Value (NPV), Positive Predictive Value (PPV), Sensitivity, and Specificity.CC BY · PubMed Central
Fig. 5 [P2.3] Disregard of the properties of the data set. (a) High class imbalance. In the case of underrepresented classes, common metrics may yield misleading values. In the given example, Accuracy and Balanced Accuracy (BA) have a high score despite the high amount of False Positive (FP) samples. The class imbalance is only uncovered by metrics considering predictive values (here: Matthews Correlation Coefficient (MCC)). This pitfall is also relevant for other counting and multi-threshold metrics such as Area under the Receiver Operating Characteristic Curve (AUROC), Expected Cost (EC) (depending on the chosen costs), Positive Likelihood Ratio (LR+), Net Benefit (NB), Sensitivity, Specificity, and Weighted Cohen’s Kappa (WCK). (b) Small test set size. The values of the Expected Calibration Error (ECE) depend on the sample size. Even for a simulated perfectly calibrated model, the ECE will be substantially greater than zero for small sample sizes [37]. (c) Imperfect reference standard. A single erroneously annotated pixel may lead to a large decrease in performance, especially in the case of the Hausdorff Distance (HD) when applied to small structures. The Hausdorff Distance 95th Percentile (HD95), on the other hand, was designed to deal with spatial outliers. This pitfall is also relevant for localization criteria such as Box/Approx Intersection over Union (IoU) and Point inside Box/Approx. Further abbreviations: True Positive (TP), False Negative (FN), True Negative (TN).
Fig. 5 [P2.3] Disregard of the properties of the data set. (a) High class imbalance. In the case of underrepresented classes, common metrics may yield misleading values. In the given example, Accuracy and Balanced Accuracy (BA) have a high score despite the high amount of False Positive (FP) samples. The class imbalance is only uncovered by metrics considering predictive values (here: Matthews Correlation Coefficient (MCC)). This pitfall is also relevant for other counting and multi-threshold metrics such as Area under the Receiver Operating Characteristic Curve (AUROC), Expected Cost (EC) (depending on the chosen costs), Positive Likelihood Ratio (LR+), Net Benefit (NB), Sensitivity, Specificity, and Weighted Cohen’s Kappa (WCK). (b) Small test set size. The values of the Expected Calibration Error (ECE) depend on the sample size. Even for a simulated perfectly calibrated model, the ECE will be substantially greater than zero for small sample sizes [37]. (c) Imperfect reference standard. A single erroneously annotated pixel may lead to a large decrease in performance, especially in the case of the Hausdorff Distance (HD) when applied to small structures. The Hausdorff Distance 95th Percentile (HD95), on the other hand, was designed to deal with spatial outliers. This pitfall is also relevant for localization criteria such as Box/Approx Intersection over Union (IoU) and Point inside Box/Approx. Further abbreviations: True Positive (TP), False Negative (FN), True Negative (TN).CC BY · PubMed Central
Fig. 6 [P3] Pitfalls related to poor metric application. (a) Non-standardized metric implementation. In the case of the Average Precision (AP) metric and the construction of the Precision-Recall (PR)-curve, the strategy of how identical scores (here: confidence score of 0.80 is present twice) are treated has a substantial impact on the metric scores. Microsoft Common Objects in Context (COCO) [57] and CityScapes [18] are used as examples. (b) Non-independence of test cases. The number of images taken from Patient 1 is much higher compared to that acquired from Patients 2-5. Averaging over all Dice Similarity Coefficient (DSC) values, denoted by ∅, results in a high averaged score. Aggregating metric values per patient reveals much higher scores for Patient 1 compared to the others, which would have been hidden by simple aggregation. (c) Uninformative visualization. A single box plot (left) does not give sufficient information about the raw metric value distribution. Adding the raw metric values as jittered dots on top (right) adds important information (here: on clusters). In the case of non-independent validation data, color/shape-coding helps reveal data clusters.
Fig. 6 [P3] Pitfalls related to poor metric application. (a) Non-standardized metric implementation. In the case of the Average Precision (AP) metric and the construction of the Precision-Recall (PR)-curve, the strategy of how identical scores (here: confidence score of 0.80 is present twice) are treated has a substantial impact on the metric scores. Microsoft Common Objects in Context (COCO) [57] and CityScapes [18] are used as examples. (b) Non-independence of test cases. The number of images taken from Patient 1 is much higher compared to that acquired from Patients 2-5. Averaging over all Dice Similarity Coefficient (DSC) values, denoted by ∅, results in a high averaged score. Aggregating metric values per patient reveals much higher scores for Patient 1 compared to the others, which would have been hidden by simple aggregation. (c) Uninformative visualization. A single box plot (left) does not give sufficient information about the raw metric value distribution. Adding the raw metric values as jittered dots on top (right) adds important information (here: on clusters). In the case of non-independent validation data, color/shape-coding helps reveal data clusters.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 1

  1. Lennerz, J. K., Green, U., Williamson, D. F. K. & Mahmood, F. A unifying force for the realization of medical ai. NPJ Digit. Med. 5, 172 (2022).

Cited by

Sorted by citations

See all 241

Related

See all

Metadata

Authors
Reinke, A., Tizabi, M. D., Baumgartner, M., Eisenmann, M., Heckmann-Nötzel, D., Kavur, A. E., Rädsch, T., Sudre, C. H., Acion, L., Antonelli, M., Arbel, T., Bakas, S., Benis, A., Buettner, F., Cardoso, M. J., Cheplygina, V., Chen, J., Christodoulou, E., Cimini, B. A., Farahani, K., Ferrer, L., Galdran, A., van Ginneken, B., Glocker, B., Godau, P., Hashimoto, D. A., Hoffman, M. M., Huisman, M., Isensee, F., Jannin, P., Kahn, C. E., Kainmueller, D., Kainz, B., Karargyris, A., Kleesiek, J., Kofler, F., Kooi, T., Kopp-Schneider, A., Kozubek, M., Kreshuk, A., Kurc, T., Landman, B. A., Litjens, G., Madani, A., Maier-Hein, K., Martel, A. L., Meijering, E., Menze, B., Moons, K. G. M., Müller, H., Nichyporuk, B., Nickel, F., Petersen, J., Rafelski, S. M., Rajpoot, N., Reyes, M., Riegler, M. A., Rieke, N., Saez-Rodriguez, J., Sánchez, C. I., Shetty, S., Summers, R. M., Taha, A. A., Tiulpin, A., Tsaftaris, S. A., Van Calster, B., Varoquaux, G., Yaniv, Z. R., Jäger, P. F., & Maier-Hein, L.
Affiliations
  • German Cancer Research Center (DE)
  • King's College London (GB)
  • Mila - Quebec Artificial Intelligence Institute (CA)
  • Indiana University Indianapolis (US)
  • Holon Institute of Technology (IL)
  • Goethe University Frankfurt (DE)
  • IT University of Copenhagen (DK)
  • Leibniz Institute for Analytical Sciences - ISAS (DE)
  • Broad Institute (US)
  • National Cancer Institute (US)
  • Consejo Nacional de Investigaciones Científicas y Técnicas (AR)
  • Radboud University Medical Center (NL)
  • Imperial College London (GB)
  • University of Pennsylvania (US)
  • University Health Network (CA)
  • Inserm (FR)
  • University of Potsdam (DE)
  • Friedrich-Alexander-Universität Erlangen-Nürnberg (DE)
  • Institut de Chirurgie Guidée par l'Image (FR)
  • Essen University Hospital (DE)
  • Lunit (South Korea) (KR)
  • Masaryk University (CZ)
  • European Molecular Biology Laboratory (DE)
  • Vanderbilt University (US)
  • University of Toronto (CA)
  • UNSW Sydney (AU)
  • University of Zurich (CH)
  • Utrecht University (NL)
  • University of Geneva (CH)
  • University Medical Center Hamburg-Eppendorf (DE)
  • Allen Institute for Cell Science (US)
  • University of Warwick (GB)
  • University of Bern (CH)
  • Simula Metropolitan Center for Digital Engineering (NO)
  • Google (United States) (US)
  • National Institutes of Health (US)
  • TU Wien (AT)
  • Oulu University Hospital (FI)
  • University of Edinburgh (GB)
  • National Institute of Allergy and Infectious Diseases (US)
Corresponding author
  • Reinke, Annika
  • Tizabi, Minu D.
  • Maier-Hein, Klaus
  • Jäger, Paul F.
ORCID
Venue
Nature Methods
Publisher
Springer Science and Business Media LLC
Published
2024-02
Volume
21
Issue
2
Pages
182-194
ISSN
  • 1548-7091 (ISSN-L)
  • 1548-7105
DOI
10.1038/s41592-023-02150-0
PMC
PMC11181963
OpenAlex
W4353015179
Cited by
241
References
36
Citations by year
2026: 108 · 2025: 82 · 2024: 48
FWCI
12.21
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Methods (publishedVersion)
  • PubMed (publishedVersion)
  • UvA-DARE (University of Amsterdam) (acceptedVersion, other-oa)
  • PubMed Central (acceptedVersion)
  • Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
  • Zurich Open Repository and Archive (University of Zurich) (acceptedVersion, cc-by-nc)
  • Radboud Repository (Radboud University) (submittedVersion)
  • PubMed Central (submittedVersion, cc-by)
  • HAL (submittedVersion)
  • Universitätsbibliographie, Universität Duisburg-Essen (submittedVersion)
  • TECNALIA Publications (Fundación TECNALIA Research & Innovation) (submittedVersion, public-domain)
  • ArODES (HES-SO (https://www.hes-so.ch/)) (submittedVersion)
  • IRIS eCampus Telematic University (Università degli Studi eCampus) (submittedVersion)
  • Lirias (KU Leuven) (submittedVersion)
  • Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion)
  • Pure Amsterdam UMC (submittedVersion, cc-by)
  • Amsterdam UMC (VU Amsterdam) - Institutional Repository (submittedVersion, other-oa)
  • University of Birmingham Research Portal (University of Birmingham) (submittedVersion)
  • Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
  • PuSH - Publication Server of Helmholtz Zentrum München (submittedVersion)
  • Zurich Open Repository and Archive (University of Zurich) (submittedVersion, cc-by-nc)
  • Zurich Open Repository and Archive (University of Zurich)
Funders
  • National Science Foundation (GM135019)
  • Silicon Valley Community Foundation (2020-225720)
  • Foundation for Cardiovascular Research
  • Canadian Institute for Advanced Research
  • Wellcome Trust (WT 203148/Z/16/Z), 203148, WT213038/Z/18/Z, 213038)
  • Alzheimer's Society (AS‐JF‐17‐011)
  • Cancer Research UK (C49297/A27294)
  • Royal Academy of Engineering (RCSRF1819\8\25)
  • European Commission (101002198, HORIZON2020, 101088594)
  • Agence Nationale de la Recherche (ANR-10-IAHU-0002, ANR-10–IAHU-02, ANR-10)
  • Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung (31274, 27294, 225720, 212939, 205320)
  • Bundesministerium für Bildung und Forschung (161L0272)
  • Nederlandse Organisatie voor Wetenschappelijk Onderzoek
  • Sydäntutkimussäätiö
  • Oulun Yliopisto (336449, Profi6 336449)
  • Novo Nordisk Fonden (NNF21OC0068816)
  • Innosuisse - Schweizerische Agentur für Innovationsförderung
  • National Institutes of Health (LM2018129, 5u24ca215109-05, 2020-225720, GM135019)
  • Natural Sciences and Engineering Research Council of Canada (rgpin-2022-05134)
  • Engineering and Physical Sciences Research Council ([WT203148/Z/16/Z], WT213038/Z/18/Z)
  • Centre For Medical Engineering, King’s College London (WT 203148/Z/16/Z)
  • National Cancer Institute (U24CA180924, UH3CA225021, UG3CA225021, U24CA215109,, 3U24CA215109-02, U01CA242871, 4UH3-CA225021-03, U24CA279629)
  • National Institute of General Medical Sciences (P41GM135019)
  • National Institute of Neurological Disorders and Stroke (R01NS042645)
  • NIH Clinical Center
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Biophysics › Cell Image Analysis Techniques
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
validation metrics, medical image analysis, Delphi consensus, expert consensus, reproducibility

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review