The DOI matches its Crossref record; 208 of the 209 references it cites were found in a registry or an index; 44 of the 77 sources its claims cite were checked for support.
The METRIC-framework for assessing data quality for trustworthy AI in medicine: a systematic review
Open accessAbstract
The adoption of machine learning (ML) and, more specifically, deep learning (DL) applications into all major areas of our lives is underway. The development of trustworthy AI is especially important in medicine due to the large implications for patients’ lives. While trustworthiness concerns various aspects including ethical, transparency and safety requirements, we focus on the importance of data quality (training/test) in DL. Since data quality dictates the behaviour of ML products, evaluating data quality will play a key part in the regulatory approval of medical ML products. We perform a systematic review following PRISMA guidelines using the databases Web of Science, PubMed and ACM Digital Library. We identify 5408 studies, out of which 120 records fulfil our eligibility criteria. From this literature, we synthesise the existing knowledge on data quality frameworks and combine it with the perspective of ML applications in medicine. As a result, we propose the METRIC-framework, a specialised data quality framework for medical training data comprising 15 awareness dimensions, along which developers of medical ML applications should investigate the content of a dataset. This knowledge helps to reduce biases as a major source of unfairness, increase robustness, facilitate interpretability and thus lays the foundation for trustworthy AI in medicine. The METRIC-framework may serve as a base for systematically assessing training datasets, establishing reference datasets, and designing test datasets which has the potential to accelerate the approval of medical ML products.
Overview
CoChat Summary
TL;DR
This systematic review synthesizes data quality literature and proposes the METRIC-framework, a set of 15 awareness dimensions for assessing medical training data quality for trustworthy AI. It is intended to help evaluate whether a dataset’s content is appropriate for a specific medical ML use case.
Key findings
- From 5408 identified studies (4633 from database search, 775 from snowballing), 120 records met eligibility criteria; the corpus comprised 35 general data, 8 big data, and 77 ML data entries, with 76 non-life science and 44 life science studies.
- The authors extracted 461 different data quality terms with 991 mentions and hierarchically clustered them into 38 relevant dimensions/subdimensions, organized into 15 awareness dimensions across five clusters: measurement process, timeliness, representativeness, informativeness, and consistency.
- The METRIC-framework separates data management (documentation, security, privacy, FAIR principles) as a prerequisite rather than part of content quality; the dimensions are not yet ranked or equipped with measures.
- The review recommends prioritizing six dimensions for assessment/improvement: the representativeness cluster plus feature importance, distribution consistency, and human-induced error; ML papers were skewed toward robustness, amount of data, missing data, and label noise, with fairness and explainability underrepresented.
Methods
- Unregistered systematic review following PRISMA guidelines; searched Web of Science, PubMed, and ACM Digital Library on 12 April 2024 using a predefined string combining data quality terms with framework/dimensions or ML/AI/DL terms.
- Title/abstract screening was done by two authors independently, with consensus discussion and a third author if needed; snowballing added 775 records; full-text assessment yielded 120 included records.
- Data extraction: 120 records were read by two authors, relevant terms extracted, and related concepts clustered into subdimensions requiring at least three references; definitions were adopted from Black and van Nederpelt, Wang and Strong, or derived from the corpus.
Limitations
- The paper states the review was unregistered; database choice may skew interpretations somewhat, mitigated by snowballing; results are limited to ML approaches despite using “AI” terminology; the framework does not yet include assessment/measurement guidelines, dimensions are not ranked, and literature may be skewed toward dimensions with larger reported effects.
- Evident from the text: the framework is conceptual and not empirically validated, and the authors note that quantification of data quality dimension effects on ML applications is future work.
- Evident from the text: no inter-rater reliability statistics are reported for screening or extraction, although the process used independent reviewers and consensus.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
23 Valid · 2 Not valid · 7 Not checked · 70 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Another dominant topic is completeness to which the ML community almost exclusively refers to as missing data119,125–128,133–135,182.”
§ The METRIC-framework for assessing data quality for trustworthy AI in medicine: a systematic review > Results > ML data quality
What the sources say
- Partly supportedcites [119] Jouseau, R., Salva, S. & Samir, C. On studying the effect of data quality on classification performances. Intelligent Data Engineering and Automated Learning – IDEAL. 82–93 (Springer Cham, 2022). sourceExperimental — accuracy not yet measured
The Introduction (p. 3) identifies “missing values” as a data-quality error studied in machine-learning classification, so this paper exemplifies that terminology. It does not call the topic “completeness” or establish that the ML community uses “missing data” almost exclusively; nor does it establish the topic’s dominance in DL papers. (LLM, conf=0.83)
- Not checked — no full text of the sourcecites [125] Michel, E., Zernikow, B. & Wichert, S. A. Use of an artificial neural network (ANN) for classifying nursing care needed, using incomplete input data. Med. Inform. Internet Med. 25, 147–158 (2000). source
- Supportedcites [126] Barakat, M. S. et al. The effect of imputing missing clinical attribute values on training lung cancer survival prediction model performance. Health Inf. Sci. Syst. 5, 16 (2017). source
The Introduction discusses missing data in machine-learning datasets and contrasts records with missing data against complete sets. The Conclusion likewise contrasts a complete dataset with levels of missing data. This supports the terminology for which this paper is cited; whether completeness is a dominant topic is the citing authors’ own framing. (LLM, conf=0.86)
- Supportedcites [127] Radliński, Ł. The impact of data quality on software testing effort prediction. Electronics 12, 1656 (2023). source
The Introduction (page unavailable) frames missing values as a data-quality problem for machine-learning models and discusses ways to handle them. The paper therefore supports its use as an example of ML work on missing data; the broader characterization of the literature is the citing authors’ synthesis. (LLM, conf=0.87)
- Not checked — no full text of the sourcecites [128] Ghotra, B., McIntosh, S. & Hassan, A. E. Revisiting the impact of classification techniques on the performance of defect prediction models. In Proc. IEEE/ACM 37th IEEE International Conference on Software Engineering (2015). source
- Not checked — no full text of the sourcecites [133] Blake, R. & Mangiameli, P. The effects and interactions of data quality and problem complexity on classification. J. Data Inf. Qual. 2, 1–28 (2011). source
- Partly supportedcites [134] Benedick, P.-L., Robert, J. & Traon, Y. L. A systematic approach for evaluating artificial intelligence models in industrial settings. Sensors 21, 6195 (2021). sourceExperimental — accuracy not yet measured
Section 3 describes missing time-series data points as a “dropping perturbation” and discusses filling missing values. The paper does not say that completeness is a dominant topic in deep-learning papers or that the ML community almost exclusively calls it “missing data”; its literature review in Section 2 instead focuses on robustness under perturbations. (LLM, conf=0.82)
- Supportedcites [135] Che, Z., Purushotham, S., Cho, K., Sontag, D. & Liu, Y. Recurrent neural networks for multivariate time series with missing values. Sci. Rep. 8, 6085 (2018). source
The Introduction describes a deep-learning framework for time series with missing data, and the Summary describes handling missing values with recurrent neural networks. This paper therefore serves as an example of the deep-learning literature using “missing data” and “missing values” for the issue the citing sentence calls completeness. (LLM, conf=0.85)
- Partly supportedcites [182] Whang, S. E., Roh, Y., Song, H. & Lee, J.-G. Data collection and quality challenges in deep learning: a data-centric AI perspective. VLDB J. 32, 791–813 (2023). sourceExperimental — accuracy not yet measured
Section 4.2 (p. 10) says missing-data imputation is an active research topic in statistics and machine learning, supporting the claim that missing data is a significant topic. The paper does not establish that completeness is a dominant topic in deep-learning papers or that the ML community almost exclusively uses “missing data” to refer to completeness. (LLM, conf=0.78)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“If the data used for training a model is bad, the resulting AI will be bad as well (‘garbage in, garbage out’49).”
§ The METRIC-framework for assessing data quality for trustworthy AI in medicine: a systematic review > Introduction
What the sources say
- Supportedcites [49] Geiger, R. S. et al. Garbage in, garbage out? Do machine learning application papers in social computing report where human-labeled training data comes from? In Proc. Conference on Fairness, Accountability, and Transparency, 325–336 (2020). source
The Introduction (p. 1) says supervised machine learning requires high-quality training data to produce high-quality classifiers. It also explicitly explains ‘Garbage In, Garbage Out’ as the principle that flawed inputs produce flawed outputs, supporting the citing sentence’s paraphrase. (LLM, conf=0.94)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Neural networks are prone to learning biases from training data and amplifying them at test time50, giving rise to a much discussed aspect of AI behaviour: fairness51.”
§ The METRIC-framework for assessing data quality for trustworthy AI in medicine: a systematic review > Introduction
What the sources say
- Partly supportedcites [50] Zhao, J., Wang, T., Yatskar, M., Ordonez, V. & Chang, K.-W. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In Proc. Conference on Empirical Methods in Natural Language Processing, 2979–2989 (Association for Computational Linguistics, Copenhagen, Denmark, 2017). sourceExperimental — accuracy not yet measured
Section 6.3 reports that models trained on two gender-biased image datasets amplified those biases on development data. The Model and §5.2 passages show that these were structured prediction models using convolutional neural-network features. This supports the claimed phenomenon in those systems, but not the broader assertion that neural networks generally are prone to it. (LLM, conf=0.85)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Yet, one of the main causes of undesirable learned patterns lies in biased training data55,56.”
§ The METRIC-framework for assessing data quality for trustworthy AI in medicine: a systematic review > Introduction
What the sources say
- Partly supportedcites [55] Suresh, H. & Guttag, J. A framework for understanding sources of harm throughout the machine learning life cycle. In Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO ’21), October 5–9, 2021, NY, USA. ACM, New York, NY, USA. https://doi.org/10.1145/3465416.3483305 (2021). sourceExperimental — accuracy not yet measured
The Introduction (page unavailable) acknowledges that biased data can contribute to undesirable ML behavior, and §3.2 describes how an unrepresentative development sample can impair a model. The paper does not rank biased training data as a main cause; it also identifies sources of harm in modeling, evaluation, and deployment. (LLM, conf=0.85)
- Supportedcites [56] Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K. & Galstyan, A. A survey on bias and fairness in machine learning. ACM Comput. Surv. 54, 1–35 (2021). source
Section 3, “Bias in Data, Algorithms, and User Experiences,” says that algorithms trained on biased data learn those biases and reflect them in their predictions. It also recognizes other causes of bias, which is consistent with describing biased training data as one cause rather than the only cause. (LLM, conf=0.91)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“In particular, representativeness is often not a factor86,87 while it is a crucial property for secondary use of data in clinical studies88 or when reusing medical data as training data for ML applications.”
§ The METRIC-framework for assessing data quality for trustworthy AI in medicine: a systematic review > Results > General data quality
What the sources say
- Not checked — no full text of the sourcecites [86] Chan, K. S., Fowles, J. B. & Weiner, J. P. Electronic health records and the reliability and validity of quality measures: a review of the literature. Med. Care Res. Rev. 67, 503–527 (2010). source
- Supportedcites [87] Weiskopf, N. G. & Weng, C. Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research. J. Am. Med. Inform. Assoc. 20, 144–151 (2013). source
The Results section identifies five dimensions of EHR data quality derived from the reviewed literature: completeness, correctness, concordance, plausibility, and currency. Representativeness is not among them, supporting the claim that it is often absent as a data-quality factor. The indexed copy provides no page number for this passage. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
22 more claims · 4 with their cited sources checked
Create a free account to see all 102 claims and to verify the 1 reference still open.
- It may state two separate claims as one. 31
- Probably not a substantial claim. 61
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
You Only Look Once: Unified, Real-Time Object Detection
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.91 · Cited by 53,803
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
SMOTE synthetic minority over-sampling technique
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302
The FAIR Guiding Principles for scientific data management and stewardship
Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., et al. (2016) · Scientific Data · doi:10.1038/sdata.2016.18 · Cited by 18,423
Dermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362
A survey on Image Data Augmentation for Deep Learning
Shorten, C., & Khoshgoftaar, T. M. (2019) · Journal of Big Data · doi:10.1186/s40537-019-0197-0 · Cited by 13,203
Research works printed but not linked: 8
- Zemel, R., Wu, Y., Swersky, K., Pitassi, T. & Dwork, C. Learning fair representations. In Proc. 30th International Conference on Machine Learning, vol. 28, 325–333 (PMLR, Atlanta, Georgia, USA, 2013).
- Long, J., Richards, J. & Seko, C. The Canadian Institute for Health Information Data Quality Framework, version 1: a meta-evaluation and future directions. In Proc. Sixth International Conference on Information Quality, 370–383 (2001).
- Almutiry, O., Wills, G., Alwabel, A., Crowder, R. & Walters, R. Toward a framework for data quality in cloud-based health information system. In Proc. International Conference on Information Society (i-Society 2013), 153–157 (IEEE, 2013).
- Gudivada, V., Apon, A. & Ding, J. Data quality considerations for big data and machine learning: Going beyond data cleaning and transformations. Int. J. Adv. Softw. 10, 1–20 (2017).
- Zhang, H., Singh, H., Ghassemi, M. & Joshi, S. ‘Why did the model fail?’ Attributing model performance changes to distribution shifts. In Proc. 40th International Conference on Machine Learning, Vol. 202, 41550–41578 (2023).
- Hong, S. & Shen, J. Impact of training size on deep learning performance in in vivo 1H MRS. In Proc. ISMRM & SMRT Annual Meeting & Exhibition (2021).
- Thaler, R. H. & Sunstein, C. R. Nudge: Improving Decisions About Health, Wealth, and Happiness (Yale University Press, 2009).
- He, H., Bai, Y., Garcia, E. A. & Li, S. Adasyn: adaptive synthetic sampling approach for imbalanced learning. In Proc. IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence) (IEEE, Hong Kong, 2008).
Cited by
Evaluating transparency in AI/ML model characteristics for FDA-reviewed medical devices
Mehta, V., Komanduri, A., Bhadouriya, R. S., Mehta, V., Johnson, M. D., Shrestha, P., Nikolov, M. C., Jain, B., Shah, N. H., & Schulman, K. A. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02052-9 · Cited by 52
Gebler, R., Reinecke, I., Sedlmayr, M., & Goldammer, M. (2025) · Journal of Medical Internet Research · doi:10.2196/74976 · Cited by 29
Yang, L., Allen, G., Zhang, Z., & Zhao, Y. (2024) · Buildings · doi:10.3390/buildings15010021 · Cited by 29
A review of recent artificial intelligence for traditional medicine
Hou, C., Gao, Y., Lin, X., Wu, J., Li, N., Lv, H., & Chu, W. C. (2025) · Journal of Traditional and Complementary Medicine · doi:10.1016/j.jtcme.2025.02.009 · Cited by 29
Dwyer, B., Flathers, M., Sano, A., Dempsey, A. G., Cipriani, A., Gazi, A. H., Hill, B., Gorban, C., Rodríguez, C. I., Stromeyer, C., et al. (2025) · NPP—Digital Psychiatry and Neuroscience · doi:10.1038/s44277-025-00049-6 · Cited by 32
Metadata
- Authors
- Schwabe, D., Becker, K., Seyferth, M., Klaß, A., & Schaeffter, T.
- Affiliations
- Physikalisch-Technische Bundesanstalt (DE)
- Corresponding author
- Schwabe, Daniel
- Venue
- npj Digital Medicine
- Published
- 2024-08-03
- Volume
- 7
- Issue
- 1
- Pages
- 203-203
- ISSN
- 2398-6352 (ISSN-L)
- PMC
- PMC11297942
- OpenAlex
- W4401283978
- Cited by
- 169
- References
- 194
- Citations by year
- 2026: 92 · 2025: 69 · 2024: 8
- FWCI
- 3.85
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- npj Digital Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Informationszentrum Lebenswissenschaften (ZB MED) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- European Commission (101100700)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- data quality assessment, trustworthy AI, machine learning in medicine, deep learning, training data bias, robustness, interpretability, reference dataset
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The METRIC-framework for assessing data quality for trustworthy AI in medicine: a systematic review. https://app.cochat.ai/papers/10.1038/s41746-024-01196-4
- The paper
- Schwabe, D., Becker, K., Seyferth, M., Klaß, A., & Schaeffter, T. (2024). The METRIC-framework for assessing data quality for trustworthy AI in medicine: a systematic review. npj Digital Medicine. https://doi.org/10.1038/s41746-024-01196-4
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.