Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 208 of the 209 references it cites were found in a registry or an index; 44 of the 77 sources its claims cite were checked for support.

The METRIC-framework for assessing data quality for trustworthy AI in medicine: a systematic review

Open access

Abstract

The adoption of machine learning (ML) and, more specifically, deep learning (DL) applications into all major areas of our lives is underway. The development of trustworthy AI is especially important in medicine due to the large implications for patients’ lives. While trustworthiness concerns various aspects including ethical, transparency and safety requirements, we focus on the importance of data quality (training/test) in DL. Since data quality dictates the behaviour of ML products, evaluating data quality will play a key part in the regulatory approval of medical ML products. We perform a systematic review following PRISMA guidelines using the databases Web of Science, PubMed and ACM Digital Library. We identify 5408 studies, out of which 120 records fulfil our eligibility criteria. From this literature, we synthesise the existing knowledge on data quality frameworks and combine it with the perspective of ML applications in medicine. As a result, we propose the METRIC-framework, a specialised data quality framework for medical training data comprising 15 awareness dimensions, along which developers of medical ML applications should investigate the content of a dataset. This knowledge helps to reduce biases as a major source of unfairness, increase robustness, facilitate interpretability and thus lays the foundation for trustworthy AI in medicine. The METRIC-framework may serve as a base for systematically assessing training datasets, establishing reference datasets, and designing test datasets which has the potential to accelerate the approval of medical ML products.

Overview

CoChat Summary

TL;DR

This systematic review synthesizes data quality literature and proposes the METRIC-framework, a set of 15 awareness dimensions for assessing medical training data quality for trustworthy AI. It is intended to help evaluate whether a dataset’s content is appropriate for a specific medical ML use case.

Key findings

  • From 5408 identified studies (4633 from database search, 775 from snowballing), 120 records met eligibility criteria; the corpus comprised 35 general data, 8 big data, and 77 ML data entries, with 76 non-life science and 44 life science studies.
  • The authors extracted 461 different data quality terms with 991 mentions and hierarchically clustered them into 38 relevant dimensions/subdimensions, organized into 15 awareness dimensions across five clusters: measurement process, timeliness, representativeness, informativeness, and consistency.
  • The METRIC-framework separates data management (documentation, security, privacy, FAIR principles) as a prerequisite rather than part of content quality; the dimensions are not yet ranked or equipped with measures.
  • The review recommends prioritizing six dimensions for assessment/improvement: the representativeness cluster plus feature importance, distribution consistency, and human-induced error; ML papers were skewed toward robustness, amount of data, missing data, and label noise, with fairness and explainability underrepresented.

Methods

  • Unregistered systematic review following PRISMA guidelines; searched Web of Science, PubMed, and ACM Digital Library on 12 April 2024 using a predefined string combining data quality terms with framework/dimensions or ML/AI/DL terms.
  • Title/abstract screening was done by two authors independently, with consensus discussion and a third author if needed; snowballing added 775 records; full-text assessment yielded 120 included records.
  • Data extraction: 120 records were read by two authors, relevant terms extracted, and related concepts clustered into subdimensions requiring at least three references; definitions were adopted from Black and van Nederpelt, Wang and Strong, or derived from the corpus.

Limitations

  • The paper states the review was unregistered; database choice may skew interpretations somewhat, mitigated by snowballing; results are limited to ML approaches despite using “AI” terminology; the framework does not yet include assessment/measurement guidelines, dimensions are not ranked, and literature may be skewed toward dimensions with larger reported effects.
  • Evident from the text: the framework is conceptual and not empirically validated, and the authors note that quantification of data quality dimension effects on ML applications is future work.
  • Evident from the text: no inter-rater reliability statistics are reported for screening or extraction, although the process used independent reviewers and consensus.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41746-024-01196-4

Cites

208 of 209 references verified0 citation errors · 1 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 PRISMA flow diagram. The flow diagram shows the number of records identified, included and excluded at the different stages of the systematic review. The eligibility criteria for inclusion and exclusion are presented in the bottom right hand side. From a total of 5408 identified studies (4633 from database search, 775 from snowballing), the resulting literature corpus on data quality for trustworthy AI in medicine includes 120 studies.
Fig. 1 PRISMA flow diagram. The flow diagram shows the number of records identified, included and excluded at the different stages of the systematic review. The eligibility criteria for inclusion and exclusion are presented in the bottom right hand side. From a total of 5408 identified studies (4633 from database search, 775 from snowballing), the resulting literature corpus on data quality for trustworthy AI in medicine includes 120 studies.CC BY · PubMed Central
Fig. 2 Studies included in the literature corpus sorted by publication date. The 120 studies are divided into the three categories general data (35), big data (8) and ML data (77), which represent major changes in the perception of data quality. The studies' affiliation to either non-life science (76) or life science (44) related topics is indicated as well.
Fig. 2 Studies included in the literature corpus sorted by publication date. The 120 studies are divided into the three categories general data (35), big data (8) and ML data (77), which represent major changes in the perception of data quality. The studies' affiliation to either non-life science (76) or life science (44) related topics is indicated as well.CC BY · PubMed Central
Fig. 3 The METRIC-framework. This specialised framework for evaluating data quality of the content of medical training data includes a comprehensive set of awareness dimensions. The inner circle divides data quality into five clusters. These clusters contain a total of 15 data quality dimensions, which are shown on the outer circle. The subdimensions presented in grey on the border of the figure contribute to the superordinate dimension. Due to the shape of the graphic, we refer to it as wheel of data quality.
Fig. 3 The METRIC-framework. This specialised framework for evaluating data quality of the content of medical training data includes a comprehensive set of awareness dimensions. The inner circle divides data quality into five clusters. These clusters contain a total of 15 data quality dimensions, which are shown on the outer circle. The subdimensions presented in grey on the border of the figure contribute to the superordinate dimension. Due to the shape of the graphic, we refer to it as wheel of data quality.CC BY · PubMed Central
Fig. 4 METRIC-Framework in relation to data management. The cluster data management is concerned with the effective usage of the dataset. It includes basic requirements for the dataset but does not address data quality issues regarding its content. Therefore, it can be seen as a prerequisite for assessment using the METRIC-framework. Figuratively speaking, the data management cluster serves as a stable foundation for the wheel of data quality.
Fig. 4 METRIC-Framework in relation to data management. The cluster data management is concerned with the effective usage of the dataset. It includes basic requirements for the dataset but does not address data quality issues regarding its content. Therefore, it can be seen as a prerequisite for assessment using the METRIC-framework. Figuratively speaking, the data management cluster serves as a stable foundation for the wheel of data quality.CC BY · PubMed Central
Fig. 5 Categorisation of the METRIC-framework. Categorisation of dimensions along the properties quantitative vs. qualitative measure (left) and use case dependence for evaluating data quality (right). The affiliation to a category is colour-coded. The colour scale is presented in the inner circle.
Fig. 5 Categorisation of the METRIC-framework. Categorisation of dimensions along the properties quantitative vs. qualitative measure (left) and use case dependence for evaluating data quality (right). The affiliation to a category is colour-coded. The colour scale is presented in the inner circle.CC BY · PubMed Central
Fig. 6 Search string visualisation. Visualisation of the keywords and logical connections that formed the search string. Each box can be translated to parantheses in the search string. Keywords inside each box are connected with each other by a logical OR.
Fig. 6 Search string visualisation. Visualisation of the keywords and logical connections that formed the search string. Each box can be translated to parantheses in the search string. Keywords inside each box are connected with each other by a logical OR.CC BY · PubMed Central

Verification

Claims the paper makes

23 Valid · 2 Not valid · 7 Not checked · 70 Own results and interpretations

22 more claims · 4 with their cited sources checked

Create a free account to see all 102 claims and to verify the 1 reference still open.

See all claims — free account
  • It may state two separate claims as one. 31
  • Probably not a substantial claim. 61

Create a free account to see these claims.

Citations

Cites

Show all 194 references

Research works printed but not linked: 8

  1. Zemel, R., Wu, Y., Swersky, K., Pitassi, T. & Dwork, C. Learning fair representations. In Proc. 30th International Conference on Machine Learning, vol. 28, 325–333 (PMLR, Atlanta, Georgia, USA, 2013).
  2. Long, J., Richards, J. & Seko, C. The Canadian Institute for Health Information Data Quality Framework, version 1: a meta-evaluation and future directions. In Proc. Sixth International Conference on Information Quality, 370–383 (2001).
  3. Almutiry, O., Wills, G., Alwabel, A., Crowder, R. & Walters, R. Toward a framework for data quality in cloud-based health information system. In Proc. International Conference on Information Society (i-Society 2013), 153–157 (IEEE, 2013).
  4. Gudivada, V., Apon, A. & Ding, J. Data quality considerations for big data and machine learning: Going beyond data cleaning and transformations. Int. J. Adv. Softw. 10, 1–20 (2017).
  5. Zhang, H., Singh, H., Ghassemi, M. & Joshi, S. ‘Why did the model fail?’ Attributing model performance changes to distribution shifts. In Proc. 40th International Conference on Machine Learning, Vol. 202, 41550–41578 (2023).
  6. Hong, S. & Shen, J. Impact of training size on deep learning performance in in vivo 1H MRS. In Proc. ISMRM & SMRT Annual Meeting & Exhibition (2021).
  7. Thaler, R. H. & Sunstein, C. R. Nudge: Improving Decisions About Health, Wealth, and Happiness (Yale University Press, 2009).
  8. He, H., Bai, Y., Garcia, E. A. & Li, S. Adasyn: adaptive synthetic sampling approach for imbalanced learning. In Proc. IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence) (IEEE, Hong Kong, 2008).

Cited by

See all 185

Metadata

Authors
Schwabe, D., Becker, K., Seyferth, M., Klaß, A., & Schaeffter, T.
Affiliations
Physikalisch-Technische Bundesanstalt (DE)
Corresponding author
Schwabe, Daniel
Venue
npj Digital Medicine
Published
2024-08-03
Volume
7
Issue
1
Pages
203-203
ISSN
2398-6352 (ISSN-L)
OpenAlex
W4401283978
Cited by
169
References
194
Citations by year
2026: 92 · 2025: 69 · 2024: 8
FWCI
3.85
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Informationszentrum Lebenswissenschaften (ZB MED) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
European Commission (101100700)
Topics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
Keywords
data quality assessment, trustworthy AI, machine learning in medicine, deep learning, training data bias, robustness, interpretability, reference dataset

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: The METRIC-framework for assessing data quality for trustworthy AI in medicine: a systematic review. https://app.cochat.ai/papers/10.1038/s41746-024-01196-4
The paper
Schwabe, D., Becker, K., Seyferth, M., Klaß, A., & Schaeffter, T. (2024). The METRIC-framework for assessing data quality for trustworthy AI in medicine: a systematic review. npj Digital Medicine. https://doi.org/10.1038/s41746-024-01196-4

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.