Student or educator? Sign up with your school email and get 6 months of Pro free.
Cited by 302

The DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.

METhodological RadiomICs Score (METRICS): a quality scoring tool for radiomics research endorsed by EuSoMII

Open access
Kocak, Burak, Akinci D’Antonoli, Tugba, Mercaldo, Nathaniel, Alberich-Bayarri, Angel, Baessler, Bettina, Ambrosini, Ilaria, Andreychenko, Anna E., Bakas, Spyridon, Beets-Tan, Regina G. H., Bressem, Keno, Buvat, Irene, Cannella, Roberto, Cappellini, Luca Alessandro, Cavallo, Armando Ugo, Chepelev, Leonid L., Chu, Linda Chi Hang, Demircioglu, Aydin, deSouza, Nandita M., Dietzel, Matthias, Fanni, Salvatore Claudio, Fedorov, Andrey, Fournier, Laure S., Giannini, Valentina, Girometti, Rossano, Groot Lipman, Kevin B. W., Kalarakis, Georgios, Kelly, Brendan S., Klontzas, Michail E., Koh, Dow-Mu, Kotter, Elmar, Lee, Ho Yun, Maas, Mario, Marti-Bonmati, Luis, Müller, Henning, Obuchowski, Nancy, Orlhac, Fanny, Papanikolaou, Nikolaos, Petrash, Ekaterina, Pfaehler, Elisabeth, Pinto dos Santos, Daniel, Ponsiglione, Andrea, Sabater, Sebastià, Sardanelli, Francesco, Seeböck, Philipp, Sijtsema, Nanna M., Stanzione, Arnaldo, Traverso, Alberto, Ugga, Lorenzo, Vallières, Martin, van Dijk, Lisanne V., van Griethuysen, Joost J. M., van Hamersvelt, Robbert W., van Ooijen, Peter, Vernuccio, Federica, Wang, Alan, Williams, Stuart, Witowski, Jan, Zhang, Zhongyi, Zwanenburg, Alex, Cuocolo, Renato·2024·Insights into Imaging

Abstract

Purpose: To propose a new quality scoring tool, METhodological RadiomICs Score (METRICS), to assess and improve research quality of radiomics studies. Methods: We conducted an online modified Delphi study with a group of international experts. It was performed in three consecutive stages: Stage#1, item preparation; Stage#2, panel discussion among EuSoMII Auditing Group members to identify the items to be voted; and Stage#3, four rounds of the modified Delphi exercise by panelists to determine the items eligible for the METRICS and their weights. The consensus threshold was 75%. Based on the median ranks derived from expert panel opinion and their rank-sum based conversion to importance scores, the category and item weights were calculated. Result: In total, 59 panelists from 19 countries participated in selection and ranking of the items and categories. Final METRICS tool included 30 items within 9 categories. According to their weights, the categories were in descending order of importance: study design, imaging data, image processing and feature extraction, metrics and comparison, testing, feature processing, preparation for modeling, segmentation, and open science. A web application and a repository were developed to streamline the calculation of the METRICS score and to collect feedback from the radiomics community. Conclusion: In this work, we developed a scoring tool for assessing the methodological quality of the radiomics research, with a large international panel and a modified Delphi protocol. With its conditional format to cover methodological variations, it provides a well-constructed framework for the key methodological concepts to assess the quality of radiomic research papers. Critical relevance statement: A quality assessment tool, METhodological RadiomICs Score (METRICS), is made available by a large group of international domain experts, with transparent methodology, aiming at evaluating and improving research quality in radiomics and machine learning. Key points: • A methodological scoring tool, METRICS, was developed for assessing the quality of radiomics research, with a large international expert panel and a modified Delphi protocol. • The proposed scoring tool presents expert opinion-based importance weights of categories and items with a transparent methodology for the first time. • METRICS accounts for varying use cases, from handcrafted radiomics to entirely deep learning-based pipelines. • A web application has been developed to help with the calculation of the METRICS score (https://metricsscore.github.io/metrics/METRICS.html) and a repository created to collect feedback from the radiomics community (https://github.com/metricsscore/metrics). Graphical Abstract:

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s13244-023-01572-w

Cites

43 of 43 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 43 references

Cited by

See all 302

Metadata

Authors
Kocak, B., Akinci D’Antonoli, T., Mercaldo, N., Alberich-Bayarri, A., Baessler, B., Ambrosini, I., Andreychenko, A. E., Bakas, S., Beets-Tan, R. G. H., Bressem, K., Buvat, I., Cannella, R., Cappellini, L. A., Cavallo, A. U., Chepelev, L. L., Chu, L. C. H., Demircioglu, A., deSouza, N. M., Dietzel, M., Fanni, S. C., Fedorov, A., Fournier, L. S., Giannini, V., Girometti, R., Groot Lipman, K. B. W., Kalarakis, G., Kelly, B. S., Klontzas, M. E., Koh, D.-M., Kotter, E., Lee, H. Y., Maas, M., Marti-Bonmati, L., Müller, H., Obuchowski, N., Orlhac, F., Papanikolaou, N., Petrash, E., Pfaehler, E., Pinto dos Santos, D., Ponsiglione, A., Sabater, S., Sardanelli, F., Seeböck, P., Sijtsema, N. M., Stanzione, A., Traverso, A., Ugga, L., Vallières, M., van Dijk, L. V., van Griethuysen, J. J. M., van Hamersvelt, R. W., van Ooijen, P., Vernuccio, F., Wang, A., Williams, S., Witowski, J., Zhang, Z., Zwanenburg, A., & Cuocolo, R.
Venue
Insights into Imaging
Published
2024-01-17
Volume
15
Issue
1
Pages
8-8
ISSN
1869-4101 (ISSN-L)
OpenAlex
W4390928581
Cited by
302
References
43
FWCI
69.22
Type
journal article
Language
en
Open access
gold

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: METhodological RadiomICs Score (METRICS): a quality scoring tool for radiomics research endorsed by EuSoMII. https://app.cochat.ai/papers/10.1186/s13244-023-01572-w
The paper
Kocak, B., Akinci D’Antonoli, T., Mercaldo, N., Alberich-Bayarri, A., Baessler, B., Ambrosini, I., Andreychenko, A. E., Bakas, S., Beets-Tan, R. G. H., Bressem, K., Buvat, I., Cannella, R., Cappellini, L. A., Cavallo, A. U., Chepelev, L. L., Chu, L. C. H., Demircioglu, A., deSouza, N. M., Dietzel, M., . . . Cuocolo, R. (2024). METhodological RadiomICs Score (METRICS): a quality scoring tool for radiomics research endorsed by EuSoMII. Insights into Imaging. https://doi.org/10.1186/s13244-023-01572-w

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.